Module: Vicary::Gazetteer

Defined in:
lib/vicary/gazetteer.rb

Overview

Offline notability lookup: is this name a public figure, or somebody's cousin?

The Ruby port of python/src/vicary/gazetteer.py. Same tiers, same verdicts, same asymmetry — notable => KEEP, everything else => REDACT — so a miss here costs precision (a public figure masked) while a false positive costs recall, which is the gap the library exists to close.

Candidate generation over capitalised token sequences proposes Terrence Okonkwo and Vincent van Gogh with equal confidence, because in English prose they are the same thing: two capitalised words. No syntactic feature separates them, so the filter is a set-membership lookup, and this module is the lookup.

A candidate is never split into tokens and tested piecewise. If it were, Priya Raghunathan-Bell would resolve notable off Bell and a real student's name would leak. Whole-string matching is what makes the multi-token tier safe to populate broadly, and it is why honorifics are not stripped before lookup: Coach Bramwell matches no label and therefore redacts, where stripping the title would demote it to a bare surname — the shape most likely to collide with a public figure. The accepted cost is that President Lincoln over-redacts.

Two tiers are deliberately invisible to Index#notability: given points the other way (a common first name is evidence of a person, so on the inbound path it means redact), and settlement types a mask rather than granting one. Wiring either into notability would readmit the exact PII the tiers exist to remove, which is why each has its own guard test.

Defined Under Namespace

Classes: AssetError, Index

Constant Summary collapse

NOT_NOTABLE =

Lookup verdicts. Strings rather than symbols so they survive a JSON round trip into and out of the conformance spec unchanged.

"not_notable"
TITLE = 
"title"
FULL_NAME = 
"full_name"
ICONIC_SHORT = 
"iconic_short"
PLACE = 
"place"
DEMONYM =

A nationality or regional adjective — Cuban, Nigerian, Bostonian.

Its own verdict rather than folded into PLACE because it is not a place: it is a word derived from one, it is the only keep tier with no notability evidence behind it, and eval attribution needs to see it separately to tell whether this tier is where a leak came from.

"demonym"
TIER_NAMES =

Every tier this reader knows. An asset carrying a tier absent from this list is refused rather than ignored — a tier added to the builder and forgotten here would read back as an empty set, and an empty KEEP tier redacts everything it was built to protect while presenting as over-aggressive tuning.

%w[full short place given title demonym settlement].freeze
PARTICLES =

Name particles that may lead a two- or three-token partial surname.

Kept in sync with the Python runtime's list by a unit test rather than by import; the asset itself carries no copy.

Set.new(%w[
  van von de del della di da du la le
  les der den ten ter dos das al bin ibn
  mac mc st saint san abu ben op vander
]).freeze
ROLE_TITLES =

Honorifics and role titles. NOT stripped before lookup — exposed because a leading title is a positive signal that a candidate is a real person in the student's life, which is a candidate-generator concern.

Set.new(%w[
  mr mrs ms miss mx dr doctor prof professor
  coach principal officer sgt sergeant capt captain
  rev reverend father sister brother pastor rabbi
  imam nurse sen senator rep gov governor mayor
  sir dame lady lord aunt uncle grandma grandpa
]).freeze
SMART_QUOTES =

Curly quotes and dashes that NFKD leaves alone.

Student prose is full of them — a word processor turns every apostrophe curly — and without this mapping Lincoln’s folds to lincoln s and misses every tier, silently over-masking a notable name on the most ordinary punctuation there is. Identical to the Python runtime's _SMART_QUOTES; a unit test pins them together.

{
  "‘" => "'", "’" => "'", "ʼ" => "'", "′" => "'",
  "“" => '"', "”" => '"',
  "‐" => "-", "‑" => "-", "‒" => "-", "–" => "-",
  "—" => "-", "−" => "-"
}.freeze
SMART_QUOTE_PATTERN = 
Regexp.union(SMART_QUOTES.keys).freeze
ALNUM =

Letters and digits, matching Python's Unicode-aware str.isalnum().

/[[:alpha:][:digit:]]/.freeze
EMPTY = 
Set.new.freeze
NORMALIZE_CACHE_MAX =

Fold a name to its lookup key.

Accent-stripped, lower-cased, punctuation reduced to spaces. The apostrophe and internal hyphen survive because they belong to the name (O'Keeffe, Raghunathan-Bell) rather than surrounding it. A trailing possessive is dropped, because Terrence's older brother presents the name as Terrence's and a lookup that misses on the clitic is a leak.

Must fold identically to the Python runtime's normalize, because the asset is keyed by one fold and probed by the other. If they drift, every lookup silently misses and the gazetteer answers "nothing is notable" while looking perfectly healthy.

One documented divergence from Python, unreachable on this asset, shared with the TypeScript port for the same reason. Python drops characters whose canonical combining class is non-zero; this drops \p{M} — every mark. The two sets differ only for marks with a combining class of zero (some Thai and Indic vowel signs), which Python turns into a space and this drops outright. That changes a key only when such a mark sits between two alphanumerics, which cannot happen in a gazetteer whose keys are Latin-folded, nor in the English prose the conformance frames carry. How many folded keys to remember. Sized against the measurement that motivated the cache: 25 carrier essays, redacted twice each, made 29,361 calls over 11,888 distinct inputs — 59.5% repeats. A document's vocabulary is the working set, so this holds several essays' worth and is cleared wholesale rather than evicted entry by entry.

20_000

Class Method Summary collapse

Class Method Details

.common_given_name?(token) ⇒ Boolean

True when token is a common given name — a REDACT signal, not a KEEP.

Returns:

  • (Boolean) 


390
391
392
 
# File 'lib/vicary/gazetteer.rb', line 390

def common_given_name?(token)
  load.common_given_name?(token)
end

.load(directory: nil) ⇒ Object

Load (and memoize) the notability index.

Lazy: requiring this file reads nothing. Call it at process init to move the decompression off the first request's latency; otherwise the first lookup pays it.



366
367
368
369
370
371
372
 
# File 'lib/vicary/gazetteer.rb', line 366

def load(directory: nil)
  return @cached if @cached && directory.nil?

  index = Index.new(Asset.load(directory: directory))
  @cached = index if directory.nil?
  index
end

.max_title_tokens ⇒ Object

Longest title in tokens — how far a title scanner must look ahead.



422
423
424
 
# File 'lib/vicary/gazetteer.rb', line 422

def max_title_tokens
  load.max_title_tokens
end

.normalize(name) ⇒ Object

Memoized normalize. Pure function of its argument, so the cache cannot change an answer — it removes an NFKD decomposition and five intermediate strings per repeated token.

Worth doing because of what it does to GC, not only to CPU: after the identity-pattern fix, sweeping and marking were ~29% of this port's time on the longest essays, and this path allocates on every call.



142
143
144
145
146
147
148
149
 
# File 'lib/vicary/gazetteer.rb', line 142

def self.normalize(name)
  cache = (@normalize_cache ||= {})
  hit = cache[name]
  return hit if hit

  cache.clear if cache.size >= NORMALIZE_CACHE_MAX
  cache[name] = normalize_uncached(name)
end

.normalize_uncached(name) ⇒ Object 



156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
 
# File 'lib/vicary/gazetteer.rb', line 156

def self.normalize_uncached(name)
  folded = name.gsub(SMART_QUOTE_PATTERN) { |char| SMART_QUOTES.fetch(char, char) }
  folded = folded.unicode_normalize(:nfkd)
  folded = folded.gsub(/\p{M}/, "")
  folded = folded.downcase

  key = folded.each_char.map { |char|
    ALNUM.match?(char) || char == "'" || char == "-" ? char : " "
  }.join.split(" ").reject(&:empty?).join(" ")

  ["'s", "s'"].each do |clitic|
    next unless key.end_with?(clitic) && key.length > clitic.length + 1

    key = key[0...-clitic.length].sub(/'+\z/, "").strip
    break
  end

  key
end

.notability(name) ⇒ Object

Which tier matched name, for telemetry and for eval attribution.



385
386
387
 
# File 'lib/vicary/gazetteer.rb', line 385

def notability(name)
  load.notability(name)
end

.notable?(name) ⇒ Boolean

True when name is a public figure or a public place. notable => KEEP.

Returns:

  • (Boolean) 


380
381
382
 
# File 'lib/vicary/gazetteer.rb', line 380

def notable?(name)
  load.notable?(name)
end

.reset_cache ⇒ Object

Drop the memoized index. For tests that swap in a fixture asset.



375
376
377
 
# File 'lib/vicary/gazetteer.rb', line 375

def reset_cache
  @cached = nil
end

.reset_normalize_cache ⇒ Object

Drop the fold cache. For tests that swap the asset underneath.



152
153
154
 
# File 'lib/vicary/gazetteer.rb', line 152

def self.reset_normalize_cache
  @normalize_cache = nil
end

.settlement?(name) ⇒ Boolean

True when name is a town or city — a TYPING signal, not a keep.

Returns:

  • (Boolean) 


395
396
397
 
# File 'lib/vicary/gazetteer.rb', line 395

def settlement?(name)
  load.settlement?(name)
end

.title?(name) ⇒ Boolean

True when name is a published work or a fictional character.

Returns:

  • (Boolean) 


400
401
402
 
# File 'lib/vicary/gazetteer.rb', line 400

def title?(name)
  load.title?(name)
end

.title_head?(token) ⇒ Boolean

True when some title starts with token — the scan's cheap prefilter.

Deliberately uses downcase rather than normalize. This runs once per word of every essay, and normalize does an NFKD decomposition and a per-character rebuild. The heads are already folded and overwhelmingly plain ASCII, so the only cost is that a title beginning with an accented word fails the prefilter and is not matched. That loses a keep, never a redaction.

Returns:

  • (Boolean) 


412
413
414
 
# File 'lib/vicary/gazetteer.rb', line 412

def title_head?(token)
  load.title_heads.include?(token.downcase)
end

.title_prefix?(key) ⇒ Boolean

True when some title starts with the folded token sequence key.

Returns:

  • (Boolean) 


417
418
419
 
# File 'lib/vicary/gazetteer.rb', line 417

def title_prefix?(key)
  load.title_prefix?(key)
end