Class: Lemans::Trial::Verifier

Inherits:
Object
  • Object
show all
Defined in:
lib/lemans/trial/verifier.rb

Overview

Verifies a trial in the sandbox the agent worked in, after Trial has closed its network. The tests are uploaded fresh at verification time, never before.

Defined Under Namespace

Classes: Verification

Constant Summary collapse

REWARD_RANGE = 
(0.0..1.0)
TESTS_DIR =

Where the task's tests land at verification time

"/tests"
ASSETS =

Harness-owned files used in verification tests

Pathname(File.expand_path("verifier/assets", __dir__))
VERIFY_BIN = 
"verify"
TAMPERED =

The message a person finds where the suite output would have been.

"The graded surfaces could not be restored from the sealed baseline: the sandbox no " \
"longer holds the tree sealed before the agent's first turn. Removing or rewriting " \
"it is a failed check, so this run scores 0.\n"

Instance Method Summary collapse

Constructor Details

#initialize(task, environment, snapshot) ⇒ Verifier

Returns a new instance of Verifier.



31
32
33
34
35
36
 
# File 'lib/lemans/trial/verifier.rb', line 31

def initialize(task, environment, snapshot)
  @task = task
  @environment = environment
  @snapshot = snapshot
  @timeout = task.verifier.timeout
end

Instance Method Details

#verify!(&evidence_collector) ⇒ Object 



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
 
# File 'lib/lemans/trial/verifier.rb', line 38

def verify!(&evidence_collector)
  upload_tests!
  prepare_env!

  # A baseline the agent made unrestorable is a verdict, not an error.
  return Verification.new(reward: 0.0, logs: TAMPERED) unless snapshot.restore!

  verification = run_tests!

  download_evidence(evidence_collector)

  verification
rescue InfrastructureError => e
  salvage_evidence(evidence_collector)
  # Everything under verification is the verifier's failure, never the
  # model's
  raise if e.is_a?(VerifierError)

  raise VerifierError, e.message
rescue StandardError
  salvage_evidence(evidence_collector)
  raise
end