Class: Insika::Evals::Runner

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/evals/runner.rb

Overview

Replays golden cases through a Transport and evaluates each. Pure over the Transport (injected) — the fake makes the orchestration unit-testable offline; the HttpTransport makes a real run.

Multi-turn: a case's turns replay IN ORDER under one conversation id (eval-<id>); earlier turns build context, the assertion runs on the LAST turn's result. A turn that errors aborts the rest of that conversation (there's nothing to continue from) and fails the case.

Defined Under Namespace

Classes: RunCase

Instance Method Summary collapse

Constructor Details

#initialize(transport:, judge: nil, conv_map: {}, capabilities: nil, pairwise: nil) ⇒ Runner

judge: an Evals::Judge (optional). When set, a case with a rubric whose turn ran cleanly gets a subjective verdict attached on top of the deterministic pass.

capabilities: what the DEPLOYMENT has, per agent — anything answering #for(agent_id) with { "tools" =>, "capabilities" => } or nil. Used to skip a case the deployment cannot satisfy, BEFORE spending a turn on it. nil (or an unknown agent) = no resolution, and then a case with requires RUNS and says so in the report: "could not rule it out" is not a reason to stop testing something, and a suite that shrinks in silence is the failure this feature exists to avoid.

pairwise: an Evals::Pairwise (optional). Only cases carrying a reference: are compared, and the verdict never touches pass/fail — it is the answer to "can we replace it", reported beside the suite's own verdict.



38
39
40
41
42
43
44
 
# File 'lib/insika/evals/runner.rb', line 38

def initialize(transport:, judge: nil, conv_map: {}, capabilities: nil, pairwise: nil)
  @transport = transport
  @judge = judge
  @conv_map = conv_map || {}
  @capabilities = capabilities
  @pairwise = pairwise
end

Instance Method Details

#run(goldens) ⇒ Object

[Golden] -> [RunCase]. Each RunCase carries the CaseResult (for the report) + per-turn timings (for --mode perf).



48
49
50
 
# File 'lib/insika/evals/runner.rb', line 48

def run(goldens)
  goldens.map { |g| run_case(g) }
end

#run_case(golden) ⇒ Object 



52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/insika/evals/runner.rb', line 52

def run_case(golden)
  skip = skip_reason(golden)
  return RunCase.new(result: Assertions.skip(golden, skip), timings: []) if skip

  # A backend that resolves state from a pre-existing conversation (e.g. a
  # consumer needing a real Chat UUID as X-Chat-Id) supplies it via conv_map; otherwise the
  # synthetic "eval-<id>" keeps the adapter's own multi-turn continuation.
  conv = @conv_map[golden.id] || "eval-#{golden.id}"
  turns = []
  timings = []
  spent = []
  cached = []
  golden.user_turns.each do |message|
    outcome = @transport.turn(agent: golden.agent, conv: conv, message: message)
    timings << { ttfb: outcome.ttfb, total: outcome.total }
    spent << billed_tokens(outcome.usage)
    cached << cached_tokens(outcome.usage)
    turns << outcome.result
    break if outcome.result.error
  end
  last = turns.last
  # Tool/content assertions read the last turn (unchanged); the policy checks
  # read every turn — "one question per reply" is a rule about each of them.
  result = Assertions.evaluate(golden, last, turns: turns)
  # Subjective layer: only when a judge is configured, the case has a rubric, and
  # the turn ran cleanly (nothing to judge on an errored turn).
  result.judge = @judge.score(golden: golden, result: last) if @judge && result.rubric && result.error.nil?
  # Against the incumbent. Same rule as the judge: nothing to
  # compare on a turn that errored — half a conversation would lose the
  # comparison for a reason that has nothing to do with the agent.
  result.pairwise = @pairwise.compare(golden: golden, turns: turns) if @pairwise && result.error.nil?
  RunCase.new(result: result, timings: timings, tokens: sum_tokens(spent),
              cached: sum_tokens(cached))
end