Module: Vicary::Conformance

Defined in:
lib/vicary/conformance.rb

Overview

Read the shared spec and score this implementation against it.

The spec lives in the repository's conformance/ directory, is generated from the Python implementation, and is what all three front doors run against. See conformance/README.md for the bar; the short version is that every frame's masked output must be byte-identical including placeholder numbering.

Why the scoreboard reports two denominators. 16 of the 51 frames expect nothing to be masked — they exist to catch over-redaction. An implementation that returns its input unchanged therefore scores 16 of 51 and looks a third of the way done while detecting nothing. So the number that leads is matched of frames_requiring_masking, with the 51-frame total beside it rather than instead of it. A ratio whose numerator a null implementation can inflate is not a measure of progress.

Defined Under Namespace

Classes: Frame, Gate, GateSpec, Golden, Identity, Outcome, Scoreboard, Span, Spec, SpecError

Constant Summary collapse

DOCUMENT_VERSION = 
1

Class Method Summary collapse

Class Method Details

.directory ⇒ Object

Locate the repository's conformance/ directory, or raise naming the search.

Raises:



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/vicary/conformance.rb', line 60

def directory
  tried = []
  current = Pathname.new(__dir__).expand_path
  8.times do
    candidate = current.join("conformance")
    tried << candidate
    return candidate if candidate.join("frames.json").file?

    parent = current.parent
    break if parent == current

    current = parent
  end
  raise SpecError,
        "no conformance/frames.json found. Looked in: #{tried.join(', ')}. " \
        "The spec lives in the repository, not in an installed gem — a " \
        "packaged copy would imply the installed one is authoritative."
end

.load_gates(dir = nil) ⇒ Object 



115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
 
# File 'lib/vicary/conformance.rb', line 115

def load_gates(dir = nil)
  dir = Pathname.new(dir || directory)
  raw = JSON.parse(dir.join("gates.json").read)
  require_version(raw["document_version"], "gates.json")
  GateSpec.new(
    reference_arm: raw.fetch("reference_arm"),
    requirements: raw.fetch("requirements"),
    gates: raw.fetch("gates").map do |g|
      Gate.new(id: g.fetch("id"), label: g.fetch("label"),
               unit: g.fetch("unit"), op: g.fetch("op"),
               bar: g.fetch("bar"), bars_by_corpus: g["bars_by_corpus"],
               requires: g.fetch("requires"), why: g.fetch("why"))
    end,
  )
end

.load_primitives(dir = nil) ⇒ Object

The primitives spec — the layer underneath the frames.

frames.json scores finished output, which is the right final bar and a poor first one: a port with nothing implemented scores 0 of 38 and learns nothing about which of the forty-odd primitives underneath is wrong. primitives.json is that missing layer, generated from the Python functions and byte-compared against a fresh export by python/tests/test_conformance.py.

Returned as the parsed document rather than as structs: it is a table of forty-odd differently-shaped sections, and a struct per section would be forty transcriptions of the thing the file exists to stop anyone transcribing.



144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/vicary/conformance.rb', line 144

def load_primitives(dir = nil)
  dir = Pathname.new(dir || directory)
  path = dir.join("primitives.json")
  unless path.file?
    raise SpecError,
          "no primitives.json at #{path}. The ports would check their " \
          "tokenisation against nothing."
  end

  raw = JSON.parse(path.read)
  require_version(raw["document_version"], "primitives.json")
  raw
end

.load_spec(dir = nil) ⇒ Object 



79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
 
# File 'lib/vicary/conformance.rb', line 79

def load_spec(dir = nil)
  dir = Pathname.new(dir || directory)
  raw = JSON.parse(dir.join("frames.json").read)
  require_version(raw["document_version"], "frames.json")

  frames = raw.fetch("frames").map do |f|
    Frame.new(
      frame_id: f.fetch("frame_id"),
      group: f.fetch("group"),
      sentence: f.fetch("sentence"),
      held_out: f.fetch("held_out", false),
      prompt_context: f.fetch("prompt_context", ""),
      note: f.fetch("note", ""),
      spans: f.fetch("spans").map { |s| span_from(s) },
    )
  end

  golden = raw.fetch("golden").transform_values do |g|
    Golden.new(masked: g.fetch("masked"),
               placeholders: g.fetch("placeholders"),
               mapping: g.fetch("mapping"),
               aligns: g.fetch("aligns"))
  end

  identity = raw.fetch("identity")
  Spec.new(
    fixture_version: raw.fetch("fixture_version"),
    reference_arm: raw.fetch("reference_arm"),
    identity: Identity.new(first_name: identity.fetch("first_name"),
                           last_name: identity.fetch("last_name"),
                           school_name: identity.fetch("school_name")),
    frames: frames,
    golden: golden,
  )
end

.report(board, gates, gate_block = nil) ⇒ Object

Render the scoreboard.

Leads with the masking-required ratio, the one a null implementation cannot inflate. Gates print NOT MEASURED per gate rather than being reduced out of the denominator — eight of nine held is a different statement from nine of nine, and a badge cannot tell them apart.

gate_block is a rendered gate block from gates.rb. Passed in rather than computed here because measuring a gate needs the detector and the asset, and this module is the spec loader — requiring them would make the loader depend on the thing it exists to score. Absent, the unmeasured block below is printed, which is the honest output for a caller that measured nothing.



213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
 
# File 'lib/vicary/conformance.rb', line 213

def report(board, gates, gate_block = nil)
  lines = []
  lines << "conformance — fixture #{board.fixture_version}, arm #{board.reference_arm}"
  lines << ("-" * 58)
  lines << format("  frames requiring masking   %3d / %d",
                  board.matched_requiring_masking, board.requiring_masking)
  lines << format("  all frames                 %3d / %d   (%d expect no " \
                  "masking, so an identity function scores that many)",
                  board.matched, board.total,
                  board.total - board.requiring_masking)
  lines << ("-" * 58)
  if gate_block
    lines << gate_block
    return lines.join("\n")
  end

  lines << "  gates:"
  gates.gates.each do |gate|
    needs = gate.requires.empty? ? "" : "  NEEDS #{gate.requires.join('+')}"
    lines << format("    NOT MEASURED  %-28s %s %s %s%s",
                    gate.label, gate.op, gate.bar, gate.unit, needs)
  end
  lines << "  -> the caller measured no gate. A green run here means the " \
           "spec loads,"
  lines << "     never that the gate set is clear."
  lines.join("\n")
end

.score(spec) ⇒ Object

Score an implementation against every frame.

The block receives (sentence, identity) — the same input every Python arm receives. Omitting the identity measures a different system and misses the easiest spans in the fixture.



163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
 
# File 'lib/vicary/conformance.rb', line 163

def score(spec)
  outcomes = spec.frames.map do |frame|
    golden = spec.golden[frame.frame_id]
    if golden.nil?
      raise SpecError,
            "frame #{frame.frame_id} has no golden output in the spec; the " \
            "file is internally inconsistent and scoring against it would " \
            "be meaningless"
    end

    produced = nil
    error = nil
    begin
      produced = yield(frame.sentence, spec.identity)
    rescue StandardError => e
      produced = ""
      error = e.message
    end

    Outcome.new(frame_id: frame.frame_id,
                requires_masking: !golden.placeholders.empty?,
                matched: error.nil? && produced == golden.masked,
                expected: golden.masked, produced: produced, error: error)
  end

  requiring = outcomes.select(&:requires_masking)
  Scoreboard.new(
    fixture_version: spec.fixture_version,
    reference_arm: spec.reference_arm,
    total: outcomes.size,
    matched: outcomes.count(&:matched),
    requiring_masking: requiring.size,
    matched_requiring_masking: requiring.count(&:matched),
    outcomes: outcomes,
  )
end