Class: Insika::Commands::JudgeShadowPairs

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/commands/judge_shadow_pairs.rb

Overview

C7 — turn complete shadow pairs into judged pairs, using the judges the operator ALREADY configured (settings, through JudgePanel — the one builder the eval CLI and the refinement gate use too). Never automatic: a Studio button or the CLI dispatches it, and it states its cost (2 provider calls per judge per pair, both orders — position-bias rule) before it runs.

Refusals are visible, a fake judge is not: no configured judges, or fewer models than the criterion demands, is a ValidationError — an unjudged window reads :invalid, where a window judged by judges nobody chose reads as evidence.

Constant Summary collapse

DEFAULT_LIMIT = 
50
DEFAULT_EXPIRE_HOURS = 
24

Instance Method Summary collapse

Constructor Details

#initialize(shadow_pairs:, settings_store:, criterion:, llm: nil, event_stream: nil) ⇒ JudgeShadowPairs

Returns a new instance of JudgeShadowPairs.



22
23
24
25
26
27
28
 
# File 'lib/insika/commands/judge_shadow_pairs.rb', line 22

def initialize(shadow_pairs:, settings_store:, criterion:, llm: nil, event_stream: nil)
  @shadow_pairs = shadow_pairs
  @settings_store = settings_store
  @criterion = criterion
  @llm = llm
  @event_stream = event_stream
end

Instance Method Details

#call(command) ⇒ Object

payload: { limit: Integer = 50, agent: String|nil, expire_after_hours: Integer|nil } -> { judged:, skipped:, expired:, failed:, models: [..] }

Raises:



32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/insika/commands/judge_shadow_pairs.rb', line 32

def call(command)
  payload = command.payload
  limit = begin
    Integer(payload[:limit] || payload["limit"] || DEFAULT_LIMIT)
  rescue ArgumentError
    raise ValidationError, "limit must be an integer"
  end
  agent = Coercion.presence(payload[:agent] || payload["agent"])
  expire_hours = begin
    Integer(payload[:expire_after_hours] || payload["expire_after_hours"] || DEFAULT_EXPIRE_HOURS)
  rescue ArgumentError
    raise ValidationError, "expire_after_hours must be an integer"
  end

  # Judges BEFORE the expire: a refusal (no judges, too few models) must
  # leave every pair untouched — expiring first would stamp :incomplete on
  # pairs and that is terminal evidence against a click that judged nothing.
  panel = Insika::Evals::JudgePanel.pairwise(settings, llm: @llm)
  raise ValidationError, "no judges configured in settings['evals'] — refusing to judge without judges" if panel.nil?

  pairwise, models = panel
  minimum = @criterion.rule.min_judge_models
  distinct = models.uniq
  if distinct.length < minimum
    raise ValidationError, "the criterion requires at least #{minimum} distinct judge models; " \
                           "#{distinct.length} configured (#{distinct.join(', ')})"
  end

  # A mirror that dropped a half stops looking like pending work.
  expired = @shadow_pairs.expire(older_than: Time.now.utc - (expire_hours * 3600))

  judged = 0
  failed = 0
  @shadow_pairs.unjudged(limit: limit, agent: agent).each do |pair|
    if judge_pair(pair, pairwise, models)
      judged += 1
    else
      failed += 1
    end
  end
  skipped = @shadow_pairs.unjudged(agent: agent).length

  emit_summary(judged: judged, skipped: skipped, expired: expired, failed: failed, models: models)
  { judged: judged, skipped: skipped, expired: expired, failed: failed, models: models }
end