Class: Vicary::Gazetteer::Index

Inherits:
Object
  • Object
show all
Defined in:
lib/vicary/gazetteer.rb

Overview

An immutable, loaded notability index over the tiers the asset carries.

The derived indices (+title_heads+, title_prefixes) are memoized on first use rather than taken as constructor arguments, because they are functions of title and must never be able to disagree with it.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(asset) ⇒ Index

Returns a new instance of Index.



184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
 
# File 'lib/vicary/gazetteer.rb', line 184

def initialize(asset)
  asset.tiers.each_key do |name|
    next if TIER_NAMES.include?(name)

    raise AssetError,
          "unknown gazetteer tier #{name.inspect}. Refusing the asset " \
          "rather than ignoring the tier: a tier this reader drops is a " \
          "tier that reads back empty, and an empty keep tier redacts " \
          "everything it was built to protect while looking like " \
          "over-aggressive tuning."
  end

  @full = asset.tiers.fetch("full", EMPTY)
  @short = asset.tiers.fetch("short", EMPTY)
  @place = asset.tiers.fetch("place", EMPTY)
  # Common given names. The INVERSE signal — see #common_given_name?.
  @given = asset.tiers.fetch("given", EMPTY)
  # Works and fictional characters — multi-token only. See #title?.
  @title = asset.tiers.fetch("title", EMPTY)
  # English demonyms — `cuban`, `nigerian`. A KEEP, see DEMONYM.
  @demonym = asset.tiers.fetch("demonym", EMPTY)
  # Human settlements. Neither a keep nor a redact signal — the only tier
  # that is neither. See #settlement?.
  @settlement = asset.tiers.fetch("settlement", EMPTY)
  @meta = asset.meta
end

Instance Attribute Details

#demonym ⇒ Object (readonly)

Returns the value of attribute demonym.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def demonym
  @demonym
end

#full ⇒ Object (readonly)

Returns the value of attribute full.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def full
  @full
end

#given ⇒ Object (readonly)

Returns the value of attribute given.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def given
  @given
end

#meta ⇒ Object (readonly)

Returns the value of attribute meta.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def meta
  @meta
end

#place ⇒ Object (readonly)

Returns the value of attribute place.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def place
  @place
end

#settlement ⇒ Object (readonly)

Returns the value of attribute settlement.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def settlement
  @settlement
end

#short ⇒ Object (readonly)

Returns the value of attribute short.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def short
  @short
end

#title ⇒ Object (readonly)

Returns the value of attribute title.



182
183
184
 
# File 'lib/vicary/gazetteer.rb', line 182

def title
  @title
end

Instance Method Details

#common_given_name?(token) ⇒ Boolean

True when token is a first name lots of notable people share.

Not part of the notability decision, and deliberately not consulted by #notability — it points the other way. A given-name hit is evidence the token names a person, which on the inbound path means redact.

It exists for the two frames capitalisation cannot reach: then terrence okonkwo showed up and MY BEST FRIEND DESHAWN PRITCHARD score zero for any candidate generator keyed on capitalisation, by construction. A case-insensitive scan closes that, and a scan needs a list. This is the list; the scan belongs to the candidate generator.

Returns:

  • (Boolean) 


295
296
297
298
 
# File 'lib/vicary/gazetteer.rb', line 295

def common_given_name?(token)
  key = Gazetteer.normalize(token)
  !key.empty? && !key.include?(" ") && given.include?(key)
end

#entry_count ⇒ Object

Entries that can make something KEEP.

given and settlement are excluded on purpose: neither grants a keep, so counting them would inflate the one number that answers "how much notability does this asset carry".



216
217
218
 
# File 'lib/vicary/gazetteer.rb', line 216

def entry_count
  full.size + short.size + place.size + title.size + demonym.size
end

#max_title_tokens ⇒ Object

Longest title in tokens, so a scanner knows how far to look ahead.



256
257
258
 
# File 'lib/vicary/gazetteer.rb', line 256

def max_title_tokens
  @max_title_tokens ||= title.map { |key| key.count(" ") + 1 }.max || 0
end

#notability(name) ⇒ Object

Classify name. One of the verdict constants above.

Places are checked first: the string is being judged on what it names, and a place-name that is also a surname (Washington, Delaware) is keepable either way, so resolving it as a place costs nothing and saves a probe.



324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
 
# File 'lib/vicary/gazetteer.rb', line 324

def notability(name)
  key = Gazetteer.normalize(name)
  return NOT_NOTABLE if key.empty?

  tokens = key.split(" ")
  return PLACE if place.include?(key)

  if tokens.length == 1
    return ICONIC_SHORT if short.include?(key)

    # After `short`, because a token that is both — none today, but the
    # tiers are rebuilt from a moving upstream — should report the tier that
    # carries notability evidence rather than the one that does not.
    return demonym.include?(key) ? DEMONYM : NOT_NOTABLE
  end

  # "van Gogh", "de Gaulle" — a partial, not a full name, so it is held to
  # the strict short-tier threshold.
  return ICONIC_SHORT if tokens.length <= 3 && PARTICLES.include?(tokens[0]) && short.include?(key)
  return FULL_NAME if full.include?(key)

  # Titles resolve LAST. "Joan of Arc" and "van Gogh" are both also film
  # titles, and attributing them to the title tier would be true but less
  # specific — the person is who the student wrote about. Either way the
  # verdict is KEEP; only the reported tier changes, and that tier is what
  # eval attribution and telemetry read.
  return TITLE if tokens.length > 1 && title.include?(key)

  NOT_NOTABLE
end

#notable?(name) ⇒ Boolean

Returns:

  • (Boolean) 


355
356
357
 
# File 'lib/vicary/gazetteer.rb', line 355

def notable?(name)
  notability(name) != NOT_NOTABLE
end

#settlement?(name) ⇒ Boolean

True when name is a town, city or village.

Not part of the notability decision, and deliberately not consulted by #notability. A settlement is a student's hometown, so it must redact; that is the whole reason settlements are subtracted from the place tier. What this answers is the next question, asked only about a span already being masked: which placeholder does it get. A host that reads the type back writes "great job describing your trip to LOCATION", and before this tier existed it wrote "NAME".

The failure modes are not symmetric with a keep tier's: a miss types a place {NAME} and a false positive types a person {LOCATION}. Both are already redacted.

Returns:

  • (Boolean) 


313
314
315
316
 
# File 'lib/vicary/gazetteer.rb', line 313

def settlement?(name)
  key = Gazetteer.normalize(name)
  !key.empty? && settlement.include?(key)
end

#title?(name) ⇒ Boolean

True when name is a published work or a fictional character.

The full tier is P31 wd:Q5 — human — so before this tier existed every work title and every fictional character redacted: "Harry Potter taught me about friendship" came back as "NAME taught me about friendship".

Multi-token by construction, and that is a safety property rather than a convenience. "It", "Up", "Her", "Room", "Brave" and "Cats" are all films; a single-token title tier would make those ordinary words permanently notable, and notable means KEEP, so the cost would land on recall.

Returns:

  • (Boolean) 


279
280
281
282
 
# File 'lib/vicary/gazetteer.rb', line 279

def title?(name)
  key = Gazetteer.normalize(name)
  key.include?(" ") && title.include?(key)
end

#title_heads ⇒ Object

First tokens of every title, so a scanner can skip most positions.

Without this the title scan costs one lookup per candidate length at every token. With it the common case is a single set miss.



224
225
226
227
228
229
230
231
232
233
 
# File 'lib/vicary/gazetteer.rb', line 224

def title_heads
  @title_heads ||= begin
    heads = Set.new
    title.each do |key|
      space = key.index(" ")
      heads << (space.nil? ? key : key[0, space])
    end
    heads
  end
end

#title_prefix?(key) ⇒ Boolean

True when some title starts with (or equals) the token sequence key.

key is an already-folded lookup key — space-joined lower-cased tokens — not raw text. The scan folds each token of the document once and joins, rather than re-normalising a growing substring at every length.

Returns:

  • (Boolean) 


265
266
267
 
# File 'lib/vicary/gazetteer.rb', line 265

def title_prefix?(key)
  title_prefixes.include?(key) || title.include?(key)
end

#title_prefixes ⇒ Object

Every token-prefix of every title, so a scan can stop the moment no title can still be reached.

This is the automaton the per-position n-gram scan was standing in for: a walk advances only while some title still starts with what it has read, which on ordinary prose is one or two tokens. A flat set of pre-joined prefixes rather than a trie of objects — same asymptotics, a fraction of the allocations, built by a single pass over keys that are already normalised.



244
245
246
247
248
249
250
251
252
253
 
# File 'lib/vicary/gazetteer.rb', line 244

def title_prefixes
  @title_prefixes ||= begin
    prefixes = Set.new
    title.each do |key|
      tokens = key.split(" ")
      (1...tokens.length).each { |length| prefixes << tokens[0, length].join(" ") }
    end
    prefixes
  end
end