Module: Insika::Evals::JudgePanel

Defined in:
lib/insika/evals/judge.rb

Overview

Builds the configured judge panel from settings["evals"].

It lives HERE and not in evals/run.rb because the CLI is no longer the only caller: the refinement gate scores a candidate with the SAME judges the operator configured, and is explicit that a second copy of the judge would be the worst possible outcome — the gate would then be grading against a rubric nobody tuned. One builder, two callers.

Class Method Summary collapse

Class Method Details

.build(settings, overrides: {}, chat_factory: nil, llm: nil) ⇒ Object

settings: the evals hash (judges/quorum/aggregate/min_agreement). overrides: the CLI's flags, which win over the stored config. -> [Judge, [model names]] | nil when nobody is configured to ask. NIL AND NOT a no-op judge: a rubric'd case with no judge reads as judge_pending, which is visible, where a judge that always passes would be silent. llm: the graph's own RubyLLM context; nil = the global constant. Only the deployment root and the CLI build judges today, and both are one graph per process — the seam keeps the default honest for the day an embedded graph gates a candidate on its own credentials.



171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'lib/insika/evals/judge.rb', line 171

def build(settings, overrides: {}, chat_factory: nil, llm: nil)
  settings = Coercion.deep_stringify(settings || {})
  overrides = overrides.transform_keys(&:to_s)
  models = resolve_models(settings, overrides)
  return nil if models.empty?

  factory = chat_factory || ->(model, provider) { ruby_llm_ask(model, provider, llm: llm) }
  judge = Judge.new(
    asks: models.map { |m| factory.call(m["model"], m["provider"]) },
    quorum: overrides["quorum"] || settings["quorum"] || 1,
    aggregate: overrides["aggregate"] || settings["aggregate"] || "median",
    min_agreement: overrides["min_agreement"] || settings["min_agreement"] || 0.5
  )
  [judge, models.map { |m| m["model"] }]
end

.judge(settings, **kw) ⇒ Object

Sugar for the callers that only want the judge (the gate).



188
 
# File 'lib/insika/evals/judge.rb', line 188

def judge(settings, **kw) = build(settings, **kw)&.first

.pairwise(settings, overrides: {}, chat_factory: nil, llm: nil) ⇒ Object

The SAME configured models, asked a different question. One builder because "who judges here" is one operator decision: a pairwise panel configured apart from the rubric panel would let a run be graded by judges nobody chose. -> [Pairwise, [model names]] | nil when nobody is configured.



194
195
196
197
198
199
200
201
202
 
# File 'lib/insika/evals/judge.rb', line 194

def pairwise(settings, overrides: {}, chat_factory: nil, llm: nil)
  settings = Coercion.deep_stringify(settings || {})
  models = resolve_models(settings, overrides.transform_keys(&:to_s))
  return nil if models.empty?

  factory = chat_factory || ->(model, provider) { ruby_llm_ask(model, provider, llm: llm) }
  [Pairwise.new(asks: models.map { |m| factory.call(m["model"], m["provider"]) }),
   models.map { |m| m["model"] }]
end

.resolve_models(settings, overrides) ⇒ Object 



204
205
206
207
208
209
210
211
 
# File 'lib/insika/evals/judge.rb', line 204

def resolve_models(settings, overrides)
  models = if Coercion.present?(overrides["judge_model"])
             [{ "model" => overrides["judge_model"], "provider" => overrides["judge_provider"] }]
           else
             Array(settings["judges"])
           end
  models.map { |m| Coercion.deep_stringify(m) }.reject { |m| m["model"].to_s.strip.empty? }
end

.ruby_llm_ask(model, provider, llm: nil) ⇒ Object

The default way to reach a model: RubyLLM, temperature 0, required lazily so nothing here loads a provider gem until a judge is actually configured.



215
216
217
218
219
220
221
222
 
# File 'lib/insika/evals/judge.rb', line 215

def ruby_llm_ask(model, provider, llm: nil)
  require "ruby_llm"
  llm ||= RubyLLM
  lambda do |prompt|
    llm.chat(model: model, provider: provider, assume_model_exists: true)
       .with_temperature(0).ask(prompt).content
  end
end