Module: Insika::Evals::Baseline

Defined in:
lib/insika/evals/baseline.rb

Overview

gating. A baseline is the accepted state of the golden set — { cases: { id => { pass, score } } }. A gated run compares against it and blocks only on a REGRESSION, so known-failing cases don't wedge the gate while a real drop (a passing case that now fails, or a judge score that fell past the tolerance) does. That's the pre-merge gate for prompt/tool/model changes.

Defined Under Namespace

Classes: Regression

Class Method Summary collapse

Class Method Details

.compare(results, baseline, tolerance:) ⇒ Object

Compares a run against a loaded baseline. -> [Regression]. Only cases present in BOTH are compared: a new case (no baseline entry) never blocks the gate (it shows in the report as ❌ but is not a "regression"); document this in README. • pass→fail : baseline pass, now failing (hard regression). • pass→skipped: baseline pass, now unrunnable HERE. says the gate never blocks ON a skip, and it does not: a case that was already skipped or unknown stays silent. But a case that used to run on this deployment and no longer can means the agent lost a tool or a declaration — a suite shrinking in silence is exactly what this outcome was added to prevent. Re-baseline if the shrink is intended. • score-drop : baseline score - current judge score > tolerance (quality drift, even if the case still technically passes).



50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/insika/evals/baseline.rb', line 50

def compare(results, baseline, tolerance:)
  base = baseline["cases"] || {}
  results.filter_map do |r|
    b = base[r.id]
    next unless b

    if b["pass"] && r.skipped?
      Regression.new(id: r.id, kind: "pass→skipped",
                     detail: "was passing, now unrunnable here: #{r.skipped}")
    elsif b["pass"] && !r.pass?
      Regression.new(id: r.id, kind: "pass→fail", detail: "was passing, now failing")
    elsif b["score"] && r.judge && (b["score"] - r.judge.score) > tolerance
      Regression.new(id: r.id, kind: "score-drop",
                     detail: "judge #{b['score']} -> #{r.judge.score} (> #{tolerance})")
    end
  end
end

.load(path) ⇒ Object 



31
 
# File 'lib/insika/evals/baseline.rb', line 31

def load(path) = JSON.parse(File.read(path))

.snapshot(results, at:) ⇒ Object

[CaseResult] -> baseline hash. at is stamped by the caller (kept out of here so the module stays deterministic/testable). A SKIPPED case is left out entirely: writing it as pass: false would accept "this deployment cannot run it" as the accepted state, and the case would never block anywhere again.



22
23
24
25
26
27
28
29
 
# File 'lib/insika/evals/baseline.rb', line 22

def snapshot(results, at:)
  {
    "at" => at,
    "cases" => results.reject(&:skipped?).each_with_object({}) do |r, h|
      h[r.id] = { "pass" => r.pass?, "score" => (r.judge&.score) }
    end
  }
end

.write(path, results, at:) ⇒ Object 



33
34
35
 
# File 'lib/insika/evals/baseline.rb', line 33

def write(path, results, at:)
  File.write(path, JSON.pretty_generate(snapshot(results, at: at)))
end