Class: Insika::Evals::Judge

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/evals/judge.rb

Overview

The LLM-judge (RFC-0008 §3.3, Fase B). Scores a golden's rubric against the actual assistant reply — the subjective layer on top of the deterministic asserts. Pure over an injected ask callable (prompt -> raw model text), so it's unit-testable without an LLM; the real ask (RubyLLM on the utility_model, temp 0) is built by the CLI.

Conservative by construction: an unparseable judge reply scores 0 (fails) rather than silently passing.

A PANEL, not a single voice (RFC-0013 §3.9). quorum: N samples ONE model N times, which measures that model's variance and little else — at temperature 0 it mostly returns the same answer, including the same blind spot. Two DIFFERENT models disagreeing about a rubric is the signal worth having, so asks: takes one callable per model: each judge is scored independently (its own samples, its own median, its own pass/fail against the case's min_score), then • aggregate combines the scores into the one the report and the baseline read (:median | :mean | :min — :min is the strict panel), • min_agreement decides the verdict: the FRACTION of judges that must pass on their own. 0.5 = a majority; 1.0 = unanimous. quorum still applies, per judge, so a panel can also be sampled.

Defined Under Namespace

Classes: Verdict

Constant Summary collapse

DEFAULT_MIN_SCORE = 
0.7
AGGREGATES = 
%i[median mean min].freeze
POLICY_INSTRUCTIONS =

The store's operating policy, stated to the judge in the same words the deterministic layer checks. Empty when the store has no opinion — then the rubric alone decides, and inventing a default here would be inventing an opinion for someone else's store.

{
  "ask_once" => "This store allows AT MOST ONE question per reply. Two questions in " \
                "one message is a failure even if the content is otherwise good.",
  "investigate_first" => "This store wants the objective established BEFORE acting: on a " \
                         "vague request the assistant should ask (once or twice, not a " \
                         "form), not search immediately.",
  "act_fast" => "This store wants the assistant to ACT on the first plausible reading and " \
                "refine after — asking something it could have answered by searching is a " \
                "failure."
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(ask: nil, asks: nil, quorum: 1, aggregate: :median, min_agreement: 0.5) ⇒ Judge

ask: ->(prompt) { "" } — one judge (kept: the common case). asks: [callable, …] — a panel, one entry per model.

Raises:

  • (ArgumentError) 


52
53
54
55
56
57
58
59
60
61
 
# File 'lib/insika/evals/judge.rb', line 52

def initialize(ask: nil, asks: nil, quorum: 1, aggregate: :median, min_agreement: 0.5)
  @asks = Array(asks || ask).compact
  raise ArgumentError, "a judge needs at least one `ask`" if @asks.empty?

  @quorum = [quorum.to_i, 1].max
  @aggregate = aggregate.to_s.to_sym
  raise ArgumentError, "unknown aggregate: #{aggregate}" unless AGGREGATES.include?(@aggregate)

  @min_agreement = min_agreement.to_f.clamp(0.0, 1.0)
end

Instance Method Details

#score(golden:, result:) ⇒ Object

Golden + the LAST TurnResult -> Verdict, or nil when there's nothing to judge (no rubric). min_score comes from the golden (default 0.7).



65
66
67
68
69
70
71
72
73
74
75
76
77
78
 
# File 'lib/insika/evals/judge.rb', line 65

def score(golden:, result:)
  rubric = golden.rubric.to_s.strip
  return nil if rubric.empty?

  prompt = build_prompt(rubric, golden.user_turns, result.output_text.to_s, golden.policy)
  min = golden.min_score || DEFAULT_MIN_SCORE
  panel = @asks.map { |ask| judge_once(ask, prompt, min) }

  agreed = panel.count { |j| j[:pass] }
  Verdict.new(score: combine(panel.map { |j| j[:score] }).round(3),
              pass: (agreed.to_f / panel.length) >= @min_agreement,
              reason: panel.map { |j| j[:reason] }.reject(&:empty?).first.to_s,
              judges: panel.map { |j| j[:score] })
end