Class: Vicary::Gazetteer::Index

Inherits:
Object
  • Object
show all
Defined in:
lib/vicary/gazetteer.rb

Overview

An immutable, loaded notability index over the tiers the asset carries.

The derived indices (+title_heads+, title_prefixes) are memoized on first use rather than taken as constructor arguments, because they are functions of title and must never be able to disagree with it.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(asset) ⇒ Index

Returns a new instance of Index.



156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
 
# File 'lib/vicary/gazetteer.rb', line 156

def initialize(asset)
  asset.tiers.each_key do |name|
    next if TIER_NAMES.include?(name)

    raise AssetError,
          "unknown gazetteer tier #{name.inspect}. Refusing the asset " \
          "rather than ignoring the tier: a tier this reader drops is a " \
          "tier that reads back empty, and an empty keep tier redacts " \
          "everything it was built to protect while looking like " \
          "over-aggressive tuning."
  end

  @full = asset.tiers.fetch("full", EMPTY)
  @short = asset.tiers.fetch("short", EMPTY)
  @place = asset.tiers.fetch("place", EMPTY)
  # Common given names. The INVERSE signal — see #common_given_name?.
  @given = asset.tiers.fetch("given", EMPTY)
  # Works and fictional characters — multi-token only. See #title?.
  @title = asset.tiers.fetch("title", EMPTY)
  # English demonyms — `cuban`, `nigerian`. A KEEP, see DEMONYM.
  @demonym = asset.tiers.fetch("demonym", EMPTY)
  # Human settlements. Neither a keep nor a redact signal — the only tier
  # that is neither. See #settlement?.
  @settlement = asset.tiers.fetch("settlement", EMPTY)
  @meta = asset.meta
end

Instance Attribute Details

#demonym ⇒ Object (readonly)

Returns the value of attribute demonym.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def demonym
  @demonym
end

#full ⇒ Object (readonly)

Returns the value of attribute full.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def full
  @full
end

#given ⇒ Object (readonly)

Returns the value of attribute given.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def given
  @given
end

#meta ⇒ Object (readonly)

Returns the value of attribute meta.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def meta
  @meta
end

#place ⇒ Object (readonly)

Returns the value of attribute place.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def place
  @place
end

#settlement ⇒ Object (readonly)

Returns the value of attribute settlement.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def settlement
  @settlement
end

#short ⇒ Object (readonly)

Returns the value of attribute short.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def short
  @short
end

#title ⇒ Object (readonly)

Returns the value of attribute title.



154
155
156
 
# File 'lib/vicary/gazetteer.rb', line 154

def title
  @title
end

Instance Method Details

#common_given_name?(token) ⇒ Boolean

True when token is a first name lots of notable people share.

Not part of the notability decision, and deliberately not consulted by #notability — it points the other way. A given-name hit is evidence the token names a person, which on the inbound path means redact.

It exists for the two frames capitalisation cannot reach: then terrence okonkwo showed up and MY BEST FRIEND DESHAWN PRITCHARD score zero for any candidate generator keyed on capitalisation, by construction. A case-insensitive scan closes that, and a scan needs a list. This is the list; the scan belongs to the candidate generator.

Returns:

  • (Boolean) 


267
268
269
270
 
# File 'lib/vicary/gazetteer.rb', line 267

def common_given_name?(token)
  key = Gazetteer.normalize(token)
  !key.empty? && !key.include?(" ") && given.include?(key)
end

#entry_count ⇒ Object

Entries that can make something KEEP.

given and settlement are excluded on purpose: neither grants a keep, so counting them would inflate the one number that answers "how much notability does this asset carry".



188
189
190
 
# File 'lib/vicary/gazetteer.rb', line 188

def entry_count
  full.size + short.size + place.size + title.size + demonym.size
end

#max_title_tokens ⇒ Object

Longest title in tokens, so a scanner knows how far to look ahead.



228
229
230
 
# File 'lib/vicary/gazetteer.rb', line 228

def max_title_tokens
  @max_title_tokens ||= title.map { |key| key.count(" ") + 1 }.max || 0
end

#notability(name) ⇒ Object

Classify name. One of the verdict constants above.

Places are checked first: the string is being judged on what it names, and a place-name that is also a surname (Washington, Delaware) is keepable either way, so resolving it as a place costs nothing and saves a probe.



296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
 
# File 'lib/vicary/gazetteer.rb', line 296

def notability(name)
  key = Gazetteer.normalize(name)
  return NOT_NOTABLE if key.empty?

  tokens = key.split(" ")
  return PLACE if place.include?(key)

  if tokens.length == 1
    return ICONIC_SHORT if short.include?(key)

    # After `short`, because a token that is both — none today, but the
    # tiers are rebuilt from a moving upstream — should report the tier that
    # carries notability evidence rather than the one that does not.
    return demonym.include?(key) ? DEMONYM : NOT_NOTABLE
  end

  # "van Gogh", "de Gaulle" — a partial, not a full name, so it is held to
  # the strict short-tier threshold.
  return ICONIC_SHORT if tokens.length <= 3 && PARTICLES.include?(tokens[0]) && short.include?(key)
  return FULL_NAME if full.include?(key)

  # Titles resolve LAST. "Joan of Arc" and "van Gogh" are both also film
  # titles, and attributing them to the title tier would be true but less
  # specific — the person is who the student wrote about. Either way the
  # verdict is KEEP; only the reported tier changes, and that tier is what
  # eval attribution and telemetry read.
  return TITLE if tokens.length > 1 && title.include?(key)

  NOT_NOTABLE
end

#notable?(name) ⇒ Boolean

Returns:

  • (Boolean) 


327
328
329
 
# File 'lib/vicary/gazetteer.rb', line 327

def notable?(name)
  notability(name) != NOT_NOTABLE
end

#settlement?(name) ⇒ Boolean

True when name is a town, city or village.

Not part of the notability decision, and deliberately not consulted by #notability. A settlement is a student's hometown, so it must redact; that is the whole reason settlements are subtracted from the place tier. What this answers is the next question, asked only about a span already being masked: which placeholder does it get. A host that reads the type back writes "great job describing your trip to LOCATION", and before this tier existed it wrote "NAME".

The failure modes are not symmetric with a keep tier's: a miss types a place {NAME} and a false positive types a person {LOCATION}. Both are already redacted.

Returns:

  • (Boolean) 


285
286
287
288
 
# File 'lib/vicary/gazetteer.rb', line 285

def settlement?(name)
  key = Gazetteer.normalize(name)
  !key.empty? && settlement.include?(key)
end

#title?(name) ⇒ Boolean

True when name is a published work or a fictional character.

The full tier is P31 wd:Q5 — human — so before this tier existed every work title and every fictional character redacted: "Harry Potter taught me about friendship" came back as "NAME taught me about friendship".

Multi-token by construction, and that is a safety property rather than a convenience. "It", "Up", "Her", "Room", "Brave" and "Cats" are all films; a single-token title tier would make those ordinary words permanently notable, and notable means KEEP, so the cost would land on recall.

Returns:

  • (Boolean) 


251
252
253
254
 
# File 'lib/vicary/gazetteer.rb', line 251

def title?(name)
  key = Gazetteer.normalize(name)
  key.include?(" ") && title.include?(key)
end

#title_heads ⇒ Object

First tokens of every title, so a scanner can skip most positions.

Without this the title scan costs one lookup per candidate length at every token. With it the common case is a single set miss.



196
197
198
199
200
201
202
203
204
205
 
# File 'lib/vicary/gazetteer.rb', line 196

def title_heads
  @title_heads ||= begin
    heads = Set.new
    title.each do |key|
      space = key.index(" ")
      heads << (space.nil? ? key : key[0, space])
    end
    heads
  end
end

#title_prefix?(key) ⇒ Boolean

True when some title starts with (or equals) the token sequence key.

key is an already-folded lookup key — space-joined lower-cased tokens — not raw text. The scan folds each token of the document once and joins, rather than re-normalising a growing substring at every length.

Returns:

  • (Boolean) 


237
238
239
 
# File 'lib/vicary/gazetteer.rb', line 237

def title_prefix?(key)
  title_prefixes.include?(key) || title.include?(key)
end

#title_prefixes ⇒ Object

Every token-prefix of every title, so a scan can stop the moment no title can still be reached.

This is the automaton the per-position n-gram scan was standing in for: a walk advances only while some title still starts with what it has read, which on ordinary prose is one or two tokens. A flat set of pre-joined prefixes rather than a trie of objects — same asymptotics, a fraction of the allocations, built by a single pass over keys that are already normalised.



216
217
218
219
220
221
222
223
224
225
 
# File 'lib/vicary/gazetteer.rb', line 216

def title_prefixes
  @title_prefixes ||= begin
    prefixes = Set.new
    title.each do |key|
      tokens = key.split(" ")
      (1...tokens.length).each { |length| prefixes << tokens[0, length].join(" ") }
    end
    prefixes
  end
end