Module: Vicary::Gazetteer

Defined in:
lib/vicary/gazetteer.rb

Overview

Offline notability lookup: is this name a public figure, or somebody's cousin?

The Ruby port of python/src/vicary/gazetteer.py. Same tiers, same verdicts, same asymmetry — notable => KEEP, everything else => REDACT — so a miss here costs precision (a public figure masked) while a false positive costs recall, which is the gap the library exists to close.

Candidate generation over capitalised token sequences proposes Terrence Okonkwo and Vincent van Gogh with equal confidence, because in English prose they are the same thing: two capitalised words. No syntactic feature separates them, so the filter is a set-membership lookup, and this module is the lookup.

A candidate is never split into tokens and tested piecewise. If it were, Priya Raghunathan-Bell would resolve notable off Bell and a real student's name would leak. Whole-string matching is what makes the multi-token tier safe to populate broadly, and it is why honorifics are not stripped before lookup: Coach Bramwell matches no label and therefore redacts, where stripping the title would demote it to a bare surname — the shape most likely to collide with a public figure. The accepted cost is that President Lincoln over-redacts.

Two tiers are deliberately invisible to Index#notability: given points the other way (a common first name is evidence of a person, so on the inbound path it means redact), and settlement types a mask rather than granting one. Wiring either into notability would readmit the exact PII the tiers exist to remove, which is why each has its own guard test.

Defined Under Namespace

Classes: AssetError, Index

Constant Summary collapse

NOT_NOTABLE =

Lookup verdicts. Strings rather than symbols so they survive a JSON round trip into and out of the conformance spec unchanged.

"not_notable"
TITLE = 
"title"
FULL_NAME = 
"full_name"
ICONIC_SHORT = 
"iconic_short"
PLACE = 
"place"
DEMONYM =

A nationality or regional adjective — Cuban, Nigerian, Bostonian.

Its own verdict rather than folded into PLACE because it is not a place: it is a word derived from one, it is the only keep tier with no notability evidence behind it, and eval attribution needs to see it separately to tell whether this tier is where a leak came from.

"demonym"
TIER_NAMES =

Every tier this reader knows. An asset carrying a tier absent from this list is refused rather than ignored — a tier added to the builder and forgotten here would read back as an empty set, and an empty KEEP tier redacts everything it was built to protect while presenting as over-aggressive tuning.

%w[full short place given title demonym settlement].freeze
PARTICLES =

Name particles that may lead a two- or three-token partial surname.

Kept in sync with the Python runtime's list by a unit test rather than by import; the asset itself carries no copy.

Set.new(%w[
  van von de del della di da du la le
  les der den ten ter dos das al bin ibn
  mac mc st saint san abu ben op vander
]).freeze
ROLE_TITLES =

Honorifics and role titles. NOT stripped before lookup — exposed because a leading title is a positive signal that a candidate is a real person in the student's life, which is a candidate-generator concern.

Set.new(%w[
  mr mrs ms miss mx dr doctor prof professor
  coach principal officer sgt sergeant capt captain
  rev reverend father sister brother pastor rabbi
  imam nurse sen senator rep gov governor mayor
  sir dame lady lord aunt uncle grandma grandpa
]).freeze
SMART_QUOTES =

Curly quotes and dashes that NFKD leaves alone.

Student prose is full of them — a word processor turns every apostrophe curly — and without this mapping Lincoln’s folds to lincoln s and misses every tier, silently over-masking a notable name on the most ordinary punctuation there is. Identical to the Python runtime's _SMART_QUOTES; a unit test pins them together.

{
  "‘" => "'", "’" => "'", "ʼ" => "'", "′" => "'",
  "“" => '"', "”" => '"',
  "‐" => "-", "‑" => "-", "‒" => "-", "–" => "-",
  "—" => "-", "−" => "-"
}.freeze
SMART_QUOTE_PATTERN = 
Regexp.union(SMART_QUOTES.keys).freeze
ALNUM =

Letters and digits, matching Python's Unicode-aware str.isalnum().

/[[:alpha:][:digit:]]/.freeze
EMPTY = 
Set.new.freeze

Class Method Summary collapse

Class Method Details

.common_given_name?(token) ⇒ Boolean

True when token is a common given name — a REDACT signal, not a KEEP.

Returns:

  • (Boolean) 


362
363
364
 
# File 'lib/vicary/gazetteer.rb', line 362

def common_given_name?(token)
  load.common_given_name?(token)
end

.load(directory: nil) ⇒ Object

Load (and memoize) the notability index.

Lazy: requiring this file reads nothing. Call it at process init to move the decompression off the first request's latency; otherwise the first lookup pays it.



338
339
340
341
342
343
344
 
# File 'lib/vicary/gazetteer.rb', line 338

def load(directory: nil)
  return @cached if @cached && directory.nil?

  index = Index.new(Asset.load(directory: directory))
  @cached = index if directory.nil?
  index
end

.max_title_tokens ⇒ Object

Longest title in tokens — how far a title scanner must look ahead.



394
395
396
 
# File 'lib/vicary/gazetteer.rb', line 394

def max_title_tokens
  load.max_title_tokens
end

.normalize(name) ⇒ Object

Fold a name to its lookup key.

Accent-stripped, lower-cased, punctuation reduced to spaces. The apostrophe and internal hyphen survive because they belong to the name (O'Keeffe, Raghunathan-Bell) rather than surrounding it. A trailing possessive is dropped, because Terrence's older brother presents the name as Terrence's and a lookup that misses on the clitic is a leak.

Must fold identically to the Python runtime's normalize, because the asset is keyed by one fold and probed by the other. If they drift, every lookup silently misses and the gazetteer answers "nothing is notable" while looking perfectly healthy.

One documented divergence from Python, unreachable on this asset, shared with the TypeScript port for the same reason. Python drops characters whose canonical combining class is non-zero; this drops \p{M} — every mark. The two sets differ only for marks with a combining class of zero (some Thai and Indic vowel signs), which Python turns into a space and this drops outright. That changes a key only when such a mark sits between two alphanumerics, which cannot happen in a gazetteer whose keys are Latin-folded, nor in the English prose the conformance frames carry.



128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/vicary/gazetteer.rb', line 128

def self.normalize(name)
  folded = name.gsub(SMART_QUOTE_PATTERN) { |char| SMART_QUOTES.fetch(char, char) }
  folded = folded.unicode_normalize(:nfkd)
  folded = folded.gsub(/\p{M}/, "")
  folded = folded.downcase

  key = folded.each_char.map { |char|
    ALNUM.match?(char) || char == "'" || char == "-" ? char : " "
  }.join.split(" ").reject(&:empty?).join(" ")

  ["'s", "s'"].each do |clitic|
    next unless key.end_with?(clitic) && key.length > clitic.length + 1

    key = key[0...-clitic.length].sub(/'+\z/, "").strip
    break
  end

  key
end

.notability(name) ⇒ Object

Which tier matched name, for telemetry and for eval attribution.



357
358
359
 
# File 'lib/vicary/gazetteer.rb', line 357

def notability(name)
  load.notability(name)
end

.notable?(name) ⇒ Boolean

True when name is a public figure or a public place. notable => KEEP.

Returns:

  • (Boolean) 


352
353
354
 
# File 'lib/vicary/gazetteer.rb', line 352

def notable?(name)
  load.notable?(name)
end

.reset_cache ⇒ Object

Drop the memoized index. For tests that swap in a fixture asset.



347
348
349
 
# File 'lib/vicary/gazetteer.rb', line 347

def reset_cache
  @cached = nil
end

.settlement?(name) ⇒ Boolean

True when name is a town or city — a TYPING signal, not a keep.

Returns:

  • (Boolean) 


367
368
369
 
# File 'lib/vicary/gazetteer.rb', line 367

def settlement?(name)
  load.settlement?(name)
end

.title?(name) ⇒ Boolean

True when name is a published work or a fictional character.

Returns:

  • (Boolean) 


372
373
374
 
# File 'lib/vicary/gazetteer.rb', line 372

def title?(name)
  load.title?(name)
end

.title_head?(token) ⇒ Boolean

True when some title starts with token — the scan's cheap prefilter.

Deliberately uses downcase rather than normalize. This runs once per word of every essay, and normalize does an NFKD decomposition and a per-character rebuild. The heads are already folded and overwhelmingly plain ASCII, so the only cost is that a title beginning with an accented word fails the prefilter and is not matched. That loses a keep, never a redaction.

Returns:

  • (Boolean) 


384
385
386
 
# File 'lib/vicary/gazetteer.rb', line 384

def title_head?(token)
  load.title_heads.include?(token.downcase)
end

.title_prefix?(key) ⇒ Boolean

True when some title starts with the folded token sequence key.

Returns:

  • (Boolean) 


389
390
391
 
# File 'lib/vicary/gazetteer.rb', line 389

def title_prefix?(key)
  load.title_prefix?(key)
end