Module: Insika::Parity::Verdict

Defined in:
lib/insika/parity/verdict.rb

Overview

C6 — the mechanical fold. Given the window's pairs and the frozen criterion, produce ONE verdict and show its arithmetic. Pure: no store, no clock beyond the injected now, no LLM. Every number the Studio prints comes from here, so two people reading the same pairs get the same answer (E3).

The order of the steps IS the rule:

window -> pre-registration -> buckets -> volume -> sanity -> primary -> guards

Defined Under Namespace

Classes: Check, Report

Constant Summary collapse

DECIDED = 
%w[better comparable worse].freeze

Class Method Summary collapse

Class Method Details

.fold(pairs:, criterion:, now: Time.now.utc) ⇒ Object 



40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
 
# File 'lib/insika/parity/verdict.rb', line 40

def fold(pairs:, criterion:, now: Time.now.utc)
  rule = criterion.rule
  from = now - (rule.window_days * 86_400)
  windowed = Array(pairs).select { |p| in_window?(p, from, now) }

  checks = []

  # 2. Pre-registration: a window whose pairs disagree about which file was
  # frozen is :invalid — a post-hoc edit produces NO verdict, not a worse one.
  sha = sha_check(windowed, criterion.sha)
  if sha && !sha.met
    checks << sha
    return build(verdict: :invalid, criterion: criterion, from: from, to: now,
                 rule: rule, counts: nil, daily: [], per_agent: {},
                 checks: checks, reason: sha.note)
  end
  checks << sha if sha

  # 3. Buckets: decided is better+comparable+worse against the incumbent's
  # model (`vs: agent`). human-assisted/silent/incomplete/open/split/unknown
  # are counted and reported, never in the denominator.
  counts = count(windowed)
  daily = daily_of(windowed, from, now, rule)
  per_agent = per_agent_of(windowed)

  # 4. Volume: every day at the floor, and enough decided pairs in total —
  # short volume is :insufficient (an honest "keep running"), never a fail.
  volume = volume_check(daily, rule)
  checks << volume
  decided_check = Check.new(id: :min_decided, met: counts[:decided] >= rule.min_decided,
                            actual: counts[:decided], required: rule.min_decided,
                            note: "decided #{counts[:decided]} < #{rule.min_decided}")
  checks << decided_check
  unless volume.met && decided_check.met
    notes = [volume.met ? nil : volume.note, decided_check.met ? nil : decided_check.note].compact
    return build(verdict: :insufficient, criterion: criterion, from: from, to: now,
                 rule: rule, counts: counts, daily: daily, per_agent: per_agent,
                 checks: checks, reason: notes.join("; "))
  end

  # 5. Sanity: a panel that cannot decide, or a mirror that cannot pair,
  # is not measuring anything -> :invalid. The denominator is the panel's
  # own work: judged pairs against a model — human-assisted pairs are
  # counted and reported, never in a denominator.
  judged = counts[:judged] - counts[:human_assisted]
  undecided_rate = judged.positive? ? (counts[:split] + counts[:unknown]).fdiv(judged) : 0.0
  undecided = Check.new(id: :undecided, met: undecided_rate <= rule.undecided_rate_ceiling,
                        actual: undecided_rate.round(4), required: "<= #{rule.undecided_rate_ceiling}",
                        note: "undecided rate #{undecided_rate.round(3)} > #{rule.undecided_rate_ceiling} " \
                              "(#{counts[:split]} split, #{counts[:unknown]} unknown of #{judged} judged)")
  checks << undecided

  non_open = windowed.length - counts[:open]
  incomplete_rate = non_open.positive? ? counts[:incomplete].fdiv(non_open) : 0.0
  incomplete = Check.new(id: :incomplete, met: incomplete_rate <= rule.incomplete_rate_ceiling,
                         actual: incomplete_rate.round(4), required: "<= #{rule.incomplete_rate_ceiling}",
                         note: "incomplete rate #{incomplete_rate.round(3)} > #{rule.incomplete_rate_ceiling} " \
                               "(#{counts[:incomplete]} of #{non_open} non-open pairs never got both halves)")
  checks << incomplete
  unless undecided.met && incomplete.met
    notes = [undecided.met ? nil : undecided.note, incomplete.met ? nil : incomplete.note].compact
    return build(verdict: :invalid, criterion: criterion, from: from, to: now,
                 rule: rule, counts: counts, daily: daily, per_agent: per_agent,
                 rates: { undecided: undecided_rate, incomplete: incomplete_rate },
                 checks: checks, reason: notes.join("; "))
  end

  # 6. Primary: the lower bound of the 95% Wilson interval on win-or-tie.
  # The estimator is NAMED in the criterion, so a point estimate can never be
  # quietly substituted.
  win_or_tie = counts[:decided].positive? ? (counts[:better] + counts[:comparable]).fdiv(counts[:decided]) : 0.0
  lower = wilson_lower(counts[:better] + counts[:comparable], counts[:decided])
  primary = Check.new(id: :primary, met: lower >= rule.win_or_tie_floor,
                      actual: { win_or_tie: win_or_tie.round(4), wilson_lower_95: lower.round(4) },
                      required: "#{rule.estimator} >= #{rule.win_or_tie_floor}",
                      note: "win-or-tie #{win_or_tie.round(3)} (wilson lower #{lower.round(3)}) " \
                            "< floor #{rule.win_or_tie_floor}")
  checks << primary

  # 7. Guards: the worse tail on its own, and any store blocking its own cut.
  worse_rate = counts[:decided].positive? ? counts[:worse].fdiv(counts[:decided]) : 0.0
  worse = Check.new(id: :worse_rate, met: worse_rate <= rule.worse_rate_ceiling,
                    actual: worse_rate.round(4), required: "<= #{rule.worse_rate_ceiling}",
                    note: "worse rate #{worse_rate.round(3)} > #{rule.worse_rate_ceiling} " \
                          "(#{counts[:worse]} worse of #{counts[:decided]} decided)")
  checks << worse
  agent = per_agent_check(per_agent, rule)
  checks << agent if agent

  failed = [primary, worse, agent].compact.reject(&:met)
  unless failed.empty?
    return build(verdict: :fail, criterion: criterion, from: from, to: now,
                 rule: rule, counts: counts, daily: daily, per_agent: per_agent,
                 rates: { win_or_tie: win_or_tie, lower: lower, worse: worse_rate,
                          undecided: undecided_rate, incomplete: incomplete_rate },
                 checks: checks, reason: failed.map(&:note).join("; "))
  end

  build(verdict: :pass, criterion: criterion, from: from, to: now,
        rule: rule, counts: counts, daily: daily, per_agent: per_agent,
        rates: { win_or_tie: win_or_tie, lower: lower, worse: worse_rate,
                 undecided: undecided_rate, incomplete: incomplete_rate },
        checks: checks, reason: "the cut cleared: every check in the window is green")
end

.wilson_lower(successes, n, z: 1.96) ⇒ Object

Wilson score interval, lower bound. Pure arithmetic, unit-tested against published values (n=210, k=183 -> ~0.819).



147
148
149
150
151
152
153
154
155
156
 
# File 'lib/insika/parity/verdict.rb', line 147

def wilson_lower(successes, n, z: 1.96)
  return 0.0 if n.to_i <= 0

  p_hat = successes.to_f / n
  z2 = z * z
  denominator = 1 + (z2 / n)
  centre = p_hat + (z2 / (2 * n))
  spread = Math.sqrt((p_hat * (1 - p_hat) + (z2 / (4 * n))) / n)
  (centre - (z * spread)) / denominator
end