Module: Insika::Evals::JudgePanel

Defined in:
lib/insika/evals/judge.rb

Overview

Builds the configured judge panel from settings["evals"].

It lives HERE and not in evals/run.rb because the CLI is no longer the only caller: the refinement gate scores a candidate with the SAME judges the operator configured, and is explicit that a second copy of the judge would be the worst possible outcome — the gate would then be grading against a rubric nobody tuned. One builder, two callers.

Class Method Summary collapse

Class Method Details

.build(settings, overrides: {}, chat_factory: nil, llm: nil) ⇒ Object

settings: the evals hash (judges/quorum/aggregate/min_agreement). overrides: the CLI's flags, which win over the stored config. -> [Judge, [model names]] | nil when nobody is configured to ask. NIL AND NOT a no-op judge: a rubric'd case with no judge reads as judge_pending, which is visible, where a judge that always passes would be silent. llm: the graph's own RubyLLM context; nil = the global constant. Only the deployment root and the CLI build judges today, and both are one graph per process — the seam keeps the default honest for the day an embedded graph gates a candidate on its own credentials.



216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
 
# File 'lib/insika/evals/judge.rb', line 216

def build(settings, overrides: {}, chat_factory: nil, llm: nil)
  settings = Coercion.deep_stringify(settings || {})
  overrides = overrides.transform_keys(&:to_s)
  models = resolve_models(settings, overrides)
  return nil if models.empty?

  factory = chat_factory || ->(model, provider) { ruby_llm_ask(model, provider, llm: llm) }
  judge = Judge.new(
    asks: models.map { |m| factory.call(m["model"], m["provider"]) },
    quorum: overrides["quorum"] || settings["quorum"] || 1,
    aggregate: overrides["aggregate"] || settings["aggregate"] || "median",
    min_agreement: overrides["min_agreement"] || settings["min_agreement"] || 0.5
  )
  [judge, models.map { |m| m["model"] }]
end

.judge(settings, **kw) ⇒ Object

Sugar for the callers that only want the judge (the gate).



233
 
# File 'lib/insika/evals/judge.rb', line 233

def judge(settings, **kw) = build(settings, **kw)&.first

.pairwise(settings, overrides: {}, chat_factory: nil, llm: nil) ⇒ Object

The SAME configured models, asked a different question. One builder because "who judges here" is one operator decision: a pairwise panel configured apart from the rubric panel would let a run be graded by judges nobody chose. -> [Pairwise, [model names]] | nil when nobody is configured.



239
240
241
242
243
244
245
246
247
 
# File 'lib/insika/evals/judge.rb', line 239

def pairwise(settings, overrides: {}, chat_factory: nil, llm: nil)
  settings = Coercion.deep_stringify(settings || {})
  models = resolve_models(settings, overrides.transform_keys(&:to_s))
  return nil if models.empty?

  factory = chat_factory || ->(model, provider) { ruby_llm_ask(model, provider, llm: llm) }
  [Pairwise.new(asks: models.map { |m| factory.call(m["model"], m["provider"]) }),
   models.map { |m| m["model"] }]
end

.resolve_models(settings, overrides) ⇒ Object 



249
250
251
252
253
254
255
256
 
# File 'lib/insika/evals/judge.rb', line 249

def resolve_models(settings, overrides)
  models = if Coercion.present?(overrides["judge_model"])
             [{ "model" => overrides["judge_model"], "provider" => overrides["judge_provider"] }]
           else
             Array(settings["judges"])
           end
  models.map { |m| Coercion.deep_stringify(m) }.reject { |m| m["model"].to_s.strip.empty? }
end

.ruby_llm_ask(model, provider, llm: nil) ⇒ Object

The default way to reach a model: RubyLLM, temperature 0, required lazily so nothing here loads a provider gem until a judge is actually configured.



260
261
262
263
264
265
266
267
 
# File 'lib/insika/evals/judge.rb', line 260

def ruby_llm_ask(model, provider, llm: nil)
  require "ruby_llm"
  llm ||= RubyLLM
  lambda do |prompt|
    llm.chat(model: model, provider: provider, assume_model_exists: true)
       .with_temperature(0).ask(prompt).content
  end
end