Class: RobotLab::To::Evals::Prose

Inherits:
Base
  • Object
show all
Defined in:
lib/robot_lab/to/evals/prose.rb

Overview

Judged / pairwise eval for prose products (docs, opinion, a book).

spec        - optional spec/outline artifact the judge measures against
judge_model - model for the pairwise judge (defaults to the doer's model)
floor       - optional command for mechanizable checks (links, coverage)

The descent signal is a pairwise verdict: the judge is asked whether the working tree is BETTER than the parent commit -- absolute scores from an LLM are too noisy to descend. There is no measurable target, so met_target is always false; runs end on --stop-on-plateau or a human.

Constant Summary collapse

JUDGE_SYSTEM = 
<<~PROMPT
  You are a meticulous editor. You will see two versions of a document
  (A = the previously accepted version, B = a new draft) plus the objective
  and, if given, the spec they serve. Decide whether B is BETTER, WORSE, or
  the SAME as A at satisfying them. Judge substance -- accuracy,
  completeness, clarity, structure -- not length. Reply with exactly one
  word: better, worse, or same.
PROMPT

Instance Method Summary collapse

Constructor Details

#initialize(objective_model:, work_dir: Dir.pwd, git: CommitManager.new(work_dir: work_dir), judge_model: objective_model, spec: nil, floor: nil) ⇒ Prose

Returns a new instance of Prose.



26
27
28
29
30
31
32
33
34
35
 
# File 'lib/robot_lab/to/evals/prose.rb', line 26

def initialize(objective_model:, work_dir: Dir.pwd,
               git: CommitManager.new(work_dir: work_dir),
               judge_model: objective_model, spec: nil, floor: nil)
  super()
  @spec        = spec
  @judge_model = judge_model
  @floor       = floor
  @work_dir    = work_dir
  @git         = git
end

Instance Method Details

#parse_verdict(reply) ⇒ Object

Map a judge reply onto a verdict; defaults to :same when unclear so an ambiguous judgment does not count as improvement.



49
50
51
52
53
54
55
 
# File 'lib/robot_lab/to/evals/prose.rb', line 49

def parse_verdict(reply)
  text = reply.to_s.downcase
  return :better if text.match?(/\bbetter\b/)
  return :worse  if text.match?(/\bworse\b/)

  :same
end

#protected_paths ⇒ Object

The spec is the eval's own criteria and must be locked from the robot.



38
 
# File 'lib/robot_lab/to/evals/prose.rb', line 38

def protected_paths = Array(@spec)

#score(context) ⇒ Object 



40
41
42
43
44
45
 
# File 'lib/robot_lab/to/evals/prose.rb', line 40

def score(context)
  gate, output = floor_result
  verdict      = gate ? pairwise(context) : :worse
  Score.new(gate_ok: gate, improved: verdict == :better, met_target: false,
            value: nil, detail: "floor=#{gate} judge=#{verdict}", output: output)
end