Module: Vicary

Defined in:
lib/vicary.rb,
lib/vicary/asset.rb,
lib/vicary/gates.rb,
lib/vicary/census.rb,
lib/vicary/corpus.rb,
lib/vicary/minter.rb,
lib/vicary/redact.rb,
lib/vicary/lexicon.rb,
lib/vicary/version.rb,
lib/vicary/gazetteer.rb,
lib/vicary/candidates.rb,
lib/vicary/structured.rb,
lib/vicary/conformance.rb,
lib/vicary/latency_baseline.rb

Overview

vicary — offline redaction of personal names in student compositions.

The RubyGems front door. Vicary.redact is the one call most hosts want: hand it a composition and the student's own identity, get the masked text back. Vicary.redact_with_report returns the same bytes plus the map Vicary.restore needs to put the originals back.

What the surface is claiming. redact does the deciding now, and it reproduces all 52 fixture frames byte-for-byte against the Python reference, placeholder numbering included — the arm being local-gazetteer-lowercase. It raised NotPortedError before that rather than returning the text unchanged, because a partially ported redactor is a reasonable thing to measure and an unreasonable thing to hand a host: it would mask a phone number, miss every name in the essay, and give the caller no way to tell.

Three layers check that claim, and each catches what the one above it cannot:

  • rake conformance scores the 54 frames — the final bar, and a coarse first one;
  • rake test runs test/primitives_test.rb, forty-odd primitives over the shared primitives.json corpus, which says which brick is crooked;
  • rake redaction_parity runs both implementations over prose neither corpus contains and diffs the bytes, because several rules only diverge across a newline and both corpora are single-line.

The scoreboard prints the real count on every run precisely so readiness is never somebody's recollection.

Defined Under Namespace

Modules: Asset, Candidates, Census, Conformance, Corpus, Gates, Gazetteer, LatencyBaseline, Lexicon, Structured Classes: NotPortedError, PlaceholderMinter

Constant Summary collapse

DETECTS_NAMES =

Whether this build detects third-party names.

Exported so a host can assert on it rather than infer it from a version number. True since candidate generation landed; it is still meaningful, because NAMES_IDENTITY turns the whole route back off at runtime.

true
NAMES_IDENTITY =

Only the identity the caller handed over, plus the structured entities. No gazetteer is loaded and no candidate is generated — 0% recall on third-party names, which is a defensible choice only when it is a chosen one.

"identity"
NAMES_GAZETTEER =

Generation plus the offline notability oracle: the shippable arm.

"gazetteer"
NAMES_LOWERCASE =

...and the lowercase route, the only one that reaches a student who writes without capitals.

"gazetteer-lowercase"
DEFAULT_NAME_DETECTION =

The default, matching the reference. Recall is what to buy inbound.

NAMES_LOWERCASE
NAME_DETECTION_ENV_VAR = 
"VICARY_NAME_DETECTION"
IDENTITY_ALIASES = 
Set.new(%w[identity off none 0 false no]).freeze
GAZETTEER_ALIASES = 
Set.new(%w[gazetteer on 1 true yes names]).freeze
LOWERCASE_ALIASES = 
Set.new(%w[gazetteer-lowercase gazetteer_lowercase lowercase full max]).freeze
VERSION =

Single source of this package's version.

Shared across all three front doors on purpose: one detector, one number. A gem 0.3.0 that corresponds to nothing on PyPI cannot be reasoned about, and the parity claim is between versions, not between package names.

"0.2.4"

Class Method Summary collapse

Class Method Details

.gazetteer_oracles(level) ⇒ Object

Wire the bundled gazetteer into a detection level.

Generation and the oracle are ONE decision, not two: generation alone masks every public figure a student writes about, and the oracle alone has nothing to judge. There is deliberately no supported way to ask for half of it.

At NAMES_IDENTITY this returns nothing and the 2.1 MB asset is never touched. At the other two levels the first lookup pays the decompression; call Vicary::Gazetteer.load at process start to move that off the first request.



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/vicary/redact.rb', line 97

def gazetteer_oracles(level)
  return { candidates: false } if level == NAMES_IDENTITY

  oracles = {
    candidates: true,
    notable: ->(name) { Gazetteer.notable?(name) },
    notability_tier: ->(name) { Gazetteer.notability(name) },
    title: ->(name) { Gazetteer.title?(name) },
    title_prefix: ->(key) { Gazetteer.title_prefix?(key) },
    # Wired at BOTH gazetteer levels, unlike `given_name` below. This one
    # decides a placeholder's type, not a verdict, so it has nothing to do
    # with which candidate routes are on.
    settlement: ->(name) { Gazetteer.settlement?(name) },
  }
  # The one difference between the two gazetteer levels. Absent rather than
  # nil, so `gazetteer` and `gazetteer-lowercase` differ by the presence of a
  # key rather than by a value the merge would have to strip.
  oracles[:given_name] = ->(token) { Gazetteer.common_given_name?(token) } if level == NAMES_LOWERCASE
  oracles
end

.name_detection(value = nil) ⇒ Object

Resolve how hard the detector looks for names it was not handed.

Explicit argument, then VICARY_NAME_DETECTION, then the code default.

An unrecognized non-empty value resolves to the default, not to identity. Dropping silently to identity would leave redaction on and reporting spans while finding none of the names a reader would call PII — a failure that looks exactly like success from every log line and metric.



77
78
79
80
81
82
83
84
 
# File 'lib/vicary/redact.rb', line 77

def name_detection(value = nil)
  raw = (value || ENV[NAME_DETECTION_ENV_VAR] || "").strip.downcase
  return NAMES_IDENTITY if !raw.empty? && IDENTITY_ALIASES.include?(raw)
  return NAMES_GAZETTEER if GAZETTEER_ALIASES.include?(raw)
  return NAMES_LOWERCASE if LOWERCASE_ALIASES.include?(raw)

  DEFAULT_NAME_DETECTION
end

.redact(text, identity, options = {}) ⇒ Object

Redact personal names and structured PII from text.

Parameters:

  • text (String) —

    the composition to redact.

  • identity (Object) —

    the student the detector is told about — anything answering first_name, last_name and school_name. Every reference arm interpolates these strings, so a caller that omits them is measuring a different system and misses the easiest spans in the fixture.

  • options (Hash) (defaults to: {}) —

    the defaults are the reference arm; every flag exists so its arm stays separately measurable.



168
169
170
 
# File 'lib/vicary/redact.rb', line 168

def redact(text, identity, options = {})
  redact_with_report(text, identity, options)[0]
end

.redact_with_report(text, identity, options = {}) ⇒ Object

Redact text, returning the masked bytes and everything needed to undo it.

One minter for the whole document, because placeholder indices follow mint order across every pass.

Returns [masked_text, n_masked, restore_map].



124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/vicary/redact.rb', line 124

def redact_with_report(text, identity, options = {})
  names = options[:names]
  keep = options[:keep] || Set.new
  number_placeholders = options.fetch(:number_placeholders, true)
  headings_are_orthographic = options.fetch(:headings_are_orthographic, true)
  corroborate = options.fetch(:corroborate, true)
  relation_refusal = options.fetch(:relation_refusal, true)
  title_relation_refusal = options.fetch(:title_relation_refusal, true)

  minter = PlaceholderMinter.new(number: number_placeholders)
  return [text, 0, minter.assigned] if text.nil? || text.empty?

  masked, n = Structured.mask(text, identity, minter)

  # Candidate generation runs LAST, so every exact pattern has already
  # claimed its span.
  oracles = gazetteer_oracles(name_detection(names))
  if oracles.delete(:candidates)
    masked, count = Candidates.mask_candidates(
      masked,
      oracles.merge(
        keep: keep,
        corroborate: corroborate,
        minter: minter,
        headings_are_orthographic: headings_are_orthographic,
        relation_refusal: relation_refusal,
        title_relation_refusal: title_relation_refusal,
      ),
    )
    n += count
  end

  [masked, n, minter.assigned]
end

.restore(text, map) ⇒ Object

Put the originals back.

Longest placeholder first, so {NAME_1} cannot be partially consumed while {NAME_11} is still pending.



87
88
89
90
91
92
93
94
 
# File 'lib/vicary/minter.rb', line 87

def self.restore(text, map)
  map.keys.sort_by { |k| -k.length }.reduce(text) do |out, placeholder|
    # Block form, not the two-argument one: a replacement *string* interprets
    # `\1`, `\&` and `\\`, so a restored name containing a backslash would come
    # back altered. The block returns the original bytes untouched.
    out.gsub(placeholder) { map[placeholder] }
  end
end