Class: Lemans::Verifier

Inherits:
Object
  • Object
show all
Defined in:
lib/lemans/verifier.rb

Overview

Verifies a trial in the sandbox the agent worked in, after Trial has closed its network. The tests are uploaded fresh at verification time, never before.

Constant Summary collapse

REWARD_RANGE = 
(0.0..1.0)
TESTS_DIR =

Where the task's tests land at verification time

"/tests"
ASSETS =

Harness-owned files used in verification tests

Pathname(File.expand_path("verifier/assets", __dir__))
VERIFY_BIN = 
"verify"
TAMPERED =

The message a person finds where the suite output would have been.

"The graded surfaces could not be restored from the sealed baseline: the sandbox no " \
"longer holds the tree sealed before the agent's first turn. Removing or rewriting " \
"it is a failed check, so this run scores 0.\n"

Instance Method Summary collapse

Constructor Details

#initialize(bench:, task:, dir:, snapshot: nil) ⇒ Verifier

Returns a new instance of Verifier.



27
28
29
30
31
32
 
# File 'lib/lemans/verifier.rb', line 27

def initialize(bench:, task:, dir:, snapshot: nil)
  @bench = bench
  @task = task
  @dir = dir
  @snapshot = snapshot
end

Instance Method Details

#call(environment) ⇒ Object 



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/lemans/verifier.rb', line 34

def call(environment)
  upload_tests(environment)
  prepare(environment)

  # A baseline the agent made unrestorable is a verdict, not an error.
  unless restore_baseline(environment)
    dir.mkpath
    dir.join("verifier.log").write(TAMPERED)
    return 0.0
  end

  reward = verify(environment)
  download_evidence(environment)
  reward
rescue InfrastructureError => e
  salvage_evidence(environment)
  # Everything under verification is the verifier's failure, never the
  # model's
  raise if e.is_a?(VerifierError)

  raise VerifierError, e.message
rescue StandardError
  salvage_evidence(environment)
  raise
end