Module: Vicary::Asset

Defined in:
lib/vicary/asset.rb

Overview

Load the gazetteer asset — the same bytes the Python package loads.

The asset is a gzipped, line-oriented text file, chosen over a binary format precisely so three languages can read it without a schema compiler:

#!gazetteer 5                 format number, checked not sniffed
#!meta {"cut_date": ...}      provenance, one JSON object
#!tier demonym 1047           tier name and its DECLARED entry count
abidjanese                    one normalised entry per line

Two properties matter more than convenience.

The format number is refused, not tolerated. An unknown format means the file's meaning changed, and a reader that skips lines it does not recognise degrades into a smaller gazetteer — which redacts MORE, reads as privacy-safe, and is invisible to any test that only asks whether something was masked.

The declared tier count is checked against the parsed count. A truncated read is the same silent failure in a different costume: fewer notable people means fewer public figures kept, so an essay about Rosa Parks comes back with her name removed.

Defined Under Namespace

Classes: FormatError, Gazetteer, MissingAssetError

Constant Summary collapse

SUPPORTED_FORMAT =

Asset format this reader understands. Refuse anything else.

5
ASSET_FILENAME = 
"notability.txt.gz"
MANIFEST_FILENAME = 
"MANIFEST.json"
ASSET_PATH_ENV_VAR =

Environment override, spelled the same as the Python package's.

"VICARY_ASSET_PATH"

Class Method Summary collapse

Class Method Details

.load(directory: nil) ⇒ Object

Load and cache the gazetteer.



149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
 
# File 'lib/vicary/asset.rb', line 149

def load(directory: nil)
  return @cached if @cached && directory.nil?

  dir = Pathname.new(directory || locate)
  asset_path = dir.join(ASSET_FILENAME)
  compressed = asset_path.binread
  sha256 = Digest::SHA256.hexdigest(compressed)

  verify_against_manifest(dir, asset_path, sha256)

  format, meta, tiers = parse(Zlib::GzipReader.new(StringIO.new(compressed)).read)
  gazetteer = Gazetteer.new(format: format, meta: meta, tiers: tiers,
                            sha256: sha256, path: asset_path.to_s)
  @cached = gazetteer if directory.nil?
  gazetteer
end

.locate ⇒ Object

Raises:



68
69
70
71
72
73
74
75
76
77
 
# File 'lib/vicary/asset.rb', line 68

def locate
  tried = search_path
  found = tried.find { |dir| dir.join(ASSET_FILENAME).file? }
  return found if found

  raise MissingAssetError,
        "no #{ASSET_FILENAME} found. Looked in: " \
        "#{tried.join(', ')}. In a checkout, run `rake sync_assets`; set " \
        "#{ASSET_PATH_ENV_VAR} to override."
end

.parse(text) ⇒ Object

Parse the decompressed asset text.

Public so a test can feed it a deliberately malformed document. A parser reachable only through a 2.1 MB file on disk is a parser whose failure paths are never exercised.



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/vicary/asset.rb', line 84

def parse(text)
  lines = text.split("\n", -1)
  header = /\A\#!gazetteer (\d+)\z/.match(lines.first.to_s)
  unless header
    raise FormatError,
          "asset does not begin with a \#!gazetteer header (got " \
          "#{lines.first.to_s[0, 40].inspect})"
  end

  format = header[1].to_i
  unless format == SUPPORTED_FORMAT
    raise FormatError,
          "asset format #{format} is not #{SUPPORTED_FORMAT}. Refusing to " \
          "read it rather than skipping the parts that changed: a " \
          "partially understood gazetteer is a smaller one, and a smaller " \
          "one redacts more while looking correct."
  end

  meta = {}
  tiers = {}
  declared = {}
  current = nil

  lines.each_with_index do |line, index|
    next if index.zero? || line.empty?

    if line.start_with?("#!meta ")
      meta = JSON.parse(line.delete_prefix("#!meta "))
      next
    end

    if (tier = /\A\#!tier (\S+) (\d+)\z/.match(line))
      current = Set.new
      tiers[tier[1]] = current
      declared[tier[1]] = tier[2].to_i
      next
    end

    if line.start_with?("#!")
      raise FormatError,
            "unrecognised directive #{line[0, 40].inspect} at line " \
            "#{index + 1}; the asset format changed without its number " \
            "changing"
    end

    raise FormatError, "entry at line #{index + 1} appears before any \#!tier" if current.nil?

    current << line
  end

  declared.each do |name, count|
    actual = tiers.fetch(name).size
    next if actual == count

    raise FormatError,
          "tier #{name} declares #{count} entries and parsed #{actual}. A " \
          "short read here removes public figures from the keep list, so " \
          "an essay about a historical figure comes back with their name " \
          "redacted."
  end

  [format, meta, tiers]
end

.reset_cache ⇒ Object

Forget the cached gazetteer. For tests.



167
168
169
 
# File 'lib/vicary/asset.rb', line 167

def reset_cache
  @cached = nil
end

.search_path ⇒ Object

Candidate asset locations, most specific first.

The env override first, so an operator can point at a different cut without reinstalling. Then this gem's vendored copy, which is what an installed gem has. Then the monorepo's Python package, which is what a checkout has before rake sync_assets — so git clone && rake test works with no bootstrap step rather than failing in a way that reads as a broken port.



57
58
59
60
61
62
63
64
65
66
 
# File 'lib/vicary/asset.rb', line 57

def search_path
  gem_root = Pathname.new(__dir__).join("..", "..").expand_path
  repo_root = gem_root.join("..").expand_path
  candidates = []
  override = ENV.fetch(ASSET_PATH_ENV_VAR, "").strip
  candidates << Pathname.new(override) unless override.empty?
  candidates << gem_root.join("assets")
  candidates << repo_root.join("python", "src", "vicary", "data")
  candidates
end