Class: Insika::Harvest::Gate

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/harvest/gate.rb

Overview

C7 — the eval half of the double gate (D7). Scores ONE candidate by RUNNING it — the Refinement::Gate mechanism, with the skill's apply:

1. clone the agent into `<agent>-harvest-<run8>`,
2. copy its instruction files, write the candidate skill into the
 clone's AGENT-scoped SkillStore and enable it on the clone's
 allowlist (the clone's catalog then serves it — the skill is gated
 by being *usable*, not by prose),
3. replay the golden set against the clone over the ordinary public
 surface, compare to the accepted baseline,
4. ANY regression disqualifies,
5. destroy the clone in an `ensure`.

Judges are MANDATORY in exactly the three refusal shapes the refined gate already encodes (no baseline / all-red baseline / judged baseline without a judge) — the P18 stamp. A skill that regresses ANY golden case is rejected — this gate is a veto, never a score to argue with.

Defined Under Namespace

Classes: Report

Constant Summary collapse

DEFAULT_TOLERANCE = 
0.05

Instance Method Summary collapse

Constructor Details

#initialize(profiles:, agent_files:, goldens:, baselines:, skill_store:, skill_catalog:, transport_factory:, capabilities_factory: nil, judge_factory: nil, tolerance: DEFAULT_TOLERANCE) ⇒ Gate

skill_catalog: the graph's catalog (overlays the SkillStore) — the apply writes into the store then RELOADS it, so the clone's next dispatch serves the skill. judge_factory: the settings panel (the same one the refinement gate receives).



41
42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/insika/harvest/gate.rb', line 41

def initialize(profiles:, agent_files:, goldens:, baselines:, skill_store:,
               skill_catalog:, transport_factory:, capabilities_factory: nil,
               judge_factory: nil, tolerance: DEFAULT_TOLERANCE)
  @profiles = profiles
  @agent_files = agent_files
  @goldens = goldens
  @baselines = baselines
  @skill_store = skill_store
  @skill_catalog = skill_catalog
  @transport_factory = transport_factory
  @capabilities_factory = capabilities_factory
  @judge_factory = judge_factory
  @tolerance = tolerance
end

Instance Method Details

#clone_id_for(agent_id, run_id) ⇒ Object 



97
 
# File 'lib/insika/harvest/gate.rb', line 97

def clone_id_for(agent_id, run_id) = "#{agent_id}-harvest-#{run_id.to_s.delete('-')[0, 8]}"

#judged?(baseline) ⇒ Boolean

Returns:

  • (Boolean) 


103
104
105
 
# File 'lib/insika/harvest/gate.rb', line 103

def judged?(baseline)
  (baseline["cases"] || {}).any? { |_id, entry| entry.is_a?(Hash) && !entry["score"].nil? }
end

#passing_cases(baseline) ⇒ Object 



99
100
101
 
# File 'lib/insika/harvest/gate.rb', line 99

def passing_cases(baseline)
  (baseline["cases"] || {}).count { |_id, entry| entry.is_a?(Hash) && entry["pass"] }
end

#score(agent_id:, skill:, run_id:) ⇒ Object

skill: the candidate's { name:, description:, body:, triggers: }. -> Report. Never raises for an ordinary refusal (no cases, no baseline, a replay that blew up).



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
 
# File 'lib/insika/harvest/gate.rb', line 59

def score(agent_id:, skill:, run_id:)
  cases = @goldens.for_agent(agent_id)
  return refusal(skill, "the agent has no golden cases — nothing to gate against") if cases.empty?

  baseline = @baselines.get(agent_id)
  if baseline.nil?
    return refusal(skill, "no recorded baseline for '#{agent_id}' — " \
                          "run `insika evals:baseline import` or record one before gating")
  end

  if passing_cases(baseline).zero?
    return refusal(skill, "the recorded baseline for '#{agent_id}' has no PASSING case " \
                          "(#{baseline_size(baseline)} recorded, all failing) — nothing could " \
                          "regress, so every candidate would pass. Fix the agent or the cases, " \
                          "then re-record the baseline from a green run")
  end

  judge = @judge_factory&.call
  if judge.nil? && judged?(baseline)
    return refusal(skill, "the recorded baseline for '#{agent_id}' carries judge scores but no " \
                          "judge is configured — a rubric'd case with no verdict counts as a " \
                          "PASS, so every candidate would beat it. Configure the judge panel " \
                          "(Studio → Settings → Evals, or `settings[\"evals\"][\"judges\"]`) or " \
                          "re-record the baseline without one")
  end

  clone_id = clone_id_for(agent_id, run_id)
  begin
    build_clone(agent_id, clone_id, skill)
    ran = replay(cases, clone_id, judge)
    verdict(skill, ran, baseline, @tolerance)
  rescue StandardError => e
    refusal(skill, "gate failed to run: #{e.class}: #{e.message}")
  ensure
    destroy_clone(clone_id)
  end
end