Class: Insika::Refinement::Gate

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/refinement/gate.rb

Overview

Scores a candidate by RUNNING it. Not by asking a model whether the edit looks good — that measures nothing, and says so in one line.

1. clone the agent into a throwaway id (`<agent>-cand-<run8>`)
2. copy its instruction files, apply the candidate's edits to the COPY
3. replay the agent's golden set against the clone over the ordinary public
 surface (`POST /v1/responses`) — real turns, real tools, real guardrails
4. compare to the accepted baseline; ANY regression disqualifies
5. delete the clone, keep the report

Step 3 is what makes this expensive and what makes it worth anything. The gate is the entire safety story of refinement: everything upstream can be wrong — a hallucinated rationale, a model that misread the evidence — and the worst outcome is still a candidate that fails to improve a score and never lands.

The clone is deleted in an ensure, including when the replay raises. A leftover -cand- agent is servable at /v1/responses by anyone who knows the id, so leaking one is a real (if obscure) exposure, not just clutter.

Defined Under Namespace

Classes: Report

Constant Summary collapse

DEFAULT_TOLERANCE = 
0.05

Instance Method Summary collapse

Constructor Details

#initialize(profiles:, agent_files:, goldens:, baselines:, transport_factory:, capabilities_factory: nil, judge_factory: nil, tolerance: DEFAULT_TOLERANCE) ⇒ Gate

transport_factory: -> an Evals transport for the replay. A LAMBDA and not a transport, because the gate is constructed at boot and the deployment's own URL/token are what it has to talk to; passing the built object would freeze a credential the operator can rotate. capabilities_factory: -> an Evals::HttpCapabilities for the clone, or nil. Without it a case whose requires the agent cannot satisfy RUNS and fails (says it must skip) — and then the gate and evals/run.rb, the two callers of the one evaluator, disagree about what the corpus even measures. exists to prevent exactly that.



54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/insika/refinement/gate.rb', line 54

def initialize(profiles:, agent_files:, goldens:, baselines:, transport_factory:,
               capabilities_factory: nil, judge_factory: nil, tolerance: DEFAULT_TOLERANCE)
  @profiles = profiles
  @agent_files = agent_files
  @goldens = goldens
  @baselines = baselines
  @transport_factory = transport_factory
  @capabilities_factory = capabilities_factory
  @judge_factory = judge_factory
  @tolerance = tolerance
end

Instance Method Details

#clone_id_for(agent_id, run_id) ⇒ Object

<agent>-cand-<run8>: recognizable at a glance in the Studio's agent list and in a provider bill, and scoped to the run so two gates cannot collide.



138
 
# File 'lib/insika/refinement/gate.rb', line 138

def clone_id_for(agent_id, run_id) = "#{agent_id}-cand-#{run_id.to_s.delete('-')[0, 8]}"

#judged?(baseline) ⇒ Boolean

Was this baseline recorded with a judge? A single scored case is enough: it proves the accepted state was measured by a rubric the replay has to match. A baseline with no scores at all was recorded blind too, so both sides are equally deterministic and the comparison, while weak, is not inverted.

Returns:

  • (Boolean) 


150
151
152
 
# File 'lib/insika/refinement/gate.rb', line 150

def judged?(baseline)
  (baseline["cases"] || {}).any? { |_id, entry| entry.is_a?(Hash) && !entry["score"].nil? }
end

#passing_cases(baseline) ⇒ Object

How many accepted cases could actually regress. This is the gate's real strength, and it is worth being able to say out loud.



142
143
144
 
# File 'lib/insika/refinement/gate.rb', line 142

def passing_cases(baseline)
  (baseline["cases"] || {}).count { |_id, entry| entry.is_a?(Hash) && entry["pass"] }
end

#score(agent_id:, candidate:, run_id:, tolerance: nil) ⇒ Object

-> Report. Never raises for an ordinary refusal (no cases, no baseline, a replay that blew up): those are verdicts, and a run that recorded WHY it could not gate is more useful than an exception in a log.



69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
# File 'lib/insika/refinement/gate.rb', line 69

def score(agent_id:, candidate:, run_id:, tolerance: nil)
  cases = @goldens.for_agent(agent_id)
  return refusal(candidate, "the agent has no golden cases — nothing to gate against") if cases.empty?

  baseline = @baselines.get(agent_id)
  # Without an accepted state, `Baseline.compare` compares nothing and reports
  # zero regressions — a green light meaning "we did not look". Refusing is the
  # only honest reading, and the fix is one command.
  if baseline.nil?
    return refusal(candidate, "no recorded baseline for '#{agent_id}' — " \
                              "run `insika evals:baseline import` or record one before gating")
  end

  # And an ALL-RED baseline is the same hole with a record in front of it.
  # `compare` only reports a regression against a case the baseline had
  # PASSING, so a baseline where nothing passes cannot produce one: every
  # candidate sails through, including a harmful one.
  #
  # Found by running this against a real agent: a replay that 401'd recorded a
  # baseline of two failures, and from then on the gate accepted everything —
  # including an edit written to be harmful. "Known-failing cases do not wedge
  # the gate" is the right rule for the pre-merge check (a red case is work in
  # progress, not a blocker); here it degrades into "nothing can ever fail",
  # and a gate that cannot fail is not a gate.
  if passing_cases(baseline).zero?
    return refusal(candidate, "the recorded baseline for '#{agent_id}' has no PASSING case " \
                              "(#{baseline_size(baseline)} recorded, all failing) — nothing could " \
                              "regress, so every candidate would pass. Fix the agent or the cases, " \
                              "then re-record the baseline from a green run")
  end

  # And a baseline JUDGED by a rubric, replayed with no judge, is the third
  # shape of the same hole — the one this gate actually shipped with.
  #
  # `CaseResult#pass?` reads a missing judge verdict as a pass (a rubric'd case
  # is `judge_pending?`, which nothing consults), so a replay with no judge
  # scores every rubric case as passing. Compared against a baseline recorded
  # WITH a judge, that is not a weaker measurement, it is an inverted one:
  # every candidate reads as an improvement.
  #
  # Measured, not reasoned: gating the real pilot agent with `settings["evals"]`
  # unset reported **6/6, no regression** against a baseline the same corpus had
  # just scored **2/6** — `produto-sem-cep` was judged 0.0 and "passed". Both
  # candidates on the panel cleared. That is's failure exactly: the CLI and
  # the gate, the two callers of the one evaluator, disagreeing about what the
  # corpus measures.
  judge = @judge_factory&.call
  if judge.nil? && judged?(baseline)
    return refusal(candidate, "the recorded baseline for '#{agent_id}' carries judge scores but no " \
                              "judge is configured — a rubric'd case with no verdict counts as a " \
                              "PASS, so every candidate would beat it. Configure the judge panel " \
                              "(Studio → Settings → Evals, or `settings[\"evals\"][\"judges\"]`) or " \
                              "re-record the baseline without one")
  end

  clone_id = clone_id_for(agent_id, run_id)
  begin
    build_clone(agent_id, clone_id, candidate)
    ran = replay(cases, clone_id, judge)
    verdict(candidate, ran, baseline, tolerance || @tolerance)
  rescue StandardError => e
    refusal(candidate, "gate failed to run: #{e.class}: #{e.message}")
  ensure
    destroy_clone(clone_id)
  end
end