Class: RobotLab::To::Evals::Code

Inherits:
Base
  • Object
show all
Defined in:
lib/robot_lab/to/evals/code.rb

Overview

Measured / deterministic eval for software products.

verify  - floor gate; a command that must exit 0 (correctness)
measure - optional command printing a number to stdout; higher = better
target  - optional; met_target fires when measured value >= target

Composes Verifier for command execution. With no measure, it behaves exactly like the original verify-command gate: improved == gate, and met_target never fires (stop falls back to --stop-when).

Instance Method Summary collapse

Methods inherited from Base

#protected_paths

Constructor Details

#initialize(verify: nil, measure: nil, target: nil, timeout: 600, work_dir: Dir.pwd) ⇒ Code

Returns a new instance of Code.



16
17
18
19
20
21
22
23
 
# File 'lib/robot_lab/to/evals/code.rb', line 16

def initialize(verify: nil, measure: nil, target: nil, timeout: 600, work_dir: Dir.pwd)
  super()
  @verify   = verify
  @measure  = measure
  @target   = target
  @timeout  = timeout
  @work_dir = work_dir
end

Instance Method Details

#score(context) ⇒ Object 



25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/robot_lab/to/evals/code.rb', line 25

def score(context)
  gate_ok, output = run_gate
  value           = measure_value
  Score.new(
    gate_ok:    gate_ok,
    improved:   improved?(gate_ok, value, context.previous_value),
    met_target: target_met?(value),
    value:      value,
    detail:     detail_for(gate_ok, value),
    output:     output
  )
end