Class: GovCodes::Dafecd::IndexBuilder

Inherits:
Object
  • Object
show all
Defined in:
lib/gov_codes/dafecd/index_builder.rb

Overview

Assembles parsed DAFECD records into a specialty-keyed index.

Verification gate (DEC-003) — read this before trusting "0 unverified". The gate confirms that every value emitted into the index appears verbatim in the source text. For the CURRENT purely-deterministic extraction every emitted code is a verbatim slice of the source, so the gate is guaranteed to pass — "0 unverified" is a tautology here, NOT evidence of correctness. The gate earns its keep as a REGRESSION GUARD for future value-transforming steps (notably the C.2 title de-gluer, whose invariant is "the de-glued title equals its source with only spaces changed"). Such a transform registers the values it emits via the #emitted_values_to_verify hook, and the gate flags any that are not grounded in the source.

Drops are surfaced, never hidden: any record that carries a specialty signal (a CEM code) but yields zero parsed ladder codes is collected in #dropped_records with a reason. The invariant records_split == index.size + merged_count + dropped_records.size holds.

Title de-gluing: a TitleDegluer supplies verified clean titles (see TitleDegluer). Each applied override is checked against the raw source title via the de-gluing invariant (spacing/case only); drift lands in #unverified_titles and must fail the build. Specialties without an override keep the auto-titlecased name and are listed in #specialties_needing_deglue.

Constant Summary collapse

ACRONYM_PATTERN =

A specialty acronym is a trailing parenthetical whose sole token is an uppercase abbreviation, e.g. "... (TACP)" -> TACP. Only trailing parens qualify; a mid-title paren ("... (SERE) Specialist") is prose, not the specialty's acronym.

/\(([A-Z][A-Z0-9]{1,7})\)\s*\z/

Instance Method Summary collapse

Constructor Details

#initialize(source_text, publication: Publication.dafecd, degluer: TitleDegluer.empty, shredout_degluer: ShredoutDegluer.empty) ⇒ IndexBuilder

Returns a new instance of IndexBuilder.



44
45
46
47
48
49
50
51
52
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 44

def initialize(source_text, publication: Publication.dafecd, degluer: TitleDegluer.empty,
  shredout_degluer: ShredoutDegluer.empty)
  @source_text = source_text
  @publication = publication
  @degluer = degluer
  @shredout_degluer = shredout_degluer
  @index = nil
  @unverified_codes = nil
end

Instance Method Details

#acronym_exclusions ⇒ Object

Specialties whose trailing parenthetical is a phrase-abbreviation, not a specialty acronym (per-publication; see Publication#acronym_exclusions).



56
57
58
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 56

def acronym_exclusions
  @publication.acronym_exclusions
end

#build ⇒ Hash{Symbol=>Hash}

Returns X-form specialty => index entry.

Returns:

  • (Hash{Symbol=>Hash}) —

    X-form specialty => index entry



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 61

def build
  @index = {}
  @unverified_codes = []
  @unverified_titles = []
  @unverified_acronyms = []
  @unverified_shredouts = []
  @needs_deglue = []
  @dropped_records = []
  @merge_conflicts = []
  @merged_count = 0

  records = RecordSplitter.new(@source_text, publication: @publication).records
  @records_split = records.size

  records.each do |record|
    header = SpecialtyParser.new(record, publication: @publication).parse
    specialty = header[:specialty]

    if specialty.nil?
      @dropped_records << drop_descriptor(record)
      next
    end

    shredouts = ShredoutParser.new(record, publication: @publication).parse
    entry = build_entry(header, shredouts, record)
    @merged_count += 1 if @index.key?(specialty)
    merge_entry(specialty, entry)
  end

  @index.each do |specialty, entry|
    apply_override(specialty, entry)
    capture_acronym(specialty, entry)
    verify(entry)
  end
  apply_shredout_overrides
  @index
end

#dropped_records ⇒ Object

Records that carried a specialty signal but produced no ladder codes, each a Hash of first_line:, reason:. Surfaced so drops can never hide (C1).



153
154
155
156
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 153

def dropped_records
  build if @dropped_records.nil?
  @dropped_records
end

#glued_titles ⇒ Object

Specialties whose title is flagged as a probable pdf-reader glue artifact (heuristic), deferred to the C.2 despacer.



200
201
202
203
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 200

def glued_titles
  build if @index.nil?
  @index.select { |_, e| e[:glued_title] }.transform_values { |e| e[:name] }
end

#merge_conflicts ⇒ Object

Merge collisions where an incoming record supplied a DIFFERENT title for a level digit already present, each a Hash of level:, kept:, discarded:, code:. The existing (first-seen) title always wins; the discarded title is recorded here so the collision is visible in the CLI reconciliation rather than silently dropped (I1 layer b).



175
176
177
178
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 175

def merge_conflicts
  build if @merge_conflicts.nil?
  @merge_conflicts
end

#merged_count ⇒ Object

Number of records that merged into an already-seen specialty.



165
166
167
168
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 165

def merged_count
  build if @merged_count.nil?
  @merged_count
end

#records_split ⇒ Object

Number of records the splitter produced.



159
160
161
162
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 159

def records_split
  build if @records_split.nil?
  @records_split
end

#specialties_missing_title ⇒ Object

Specialties whose title could not be extracted (for the coverage report).



186
187
188
189
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 186

def specialties_missing_title
  build if @index.nil?
  @index.select { |_, e| e[:name].nil? || e[:name].empty? }.keys
end

#specialties_needing_deglue ⇒ Object

Specialties that have an auto-titlecased name but no verified override (i.e. still need de-gluing). Should be empty once every specialty is covered; kept for robustness across future entity types.



120
121
122
123
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 120

def specialties_needing_deglue
  build if @needs_deglue.nil?
  @needs_deglue
end

#specialties_without_shredouts ⇒ Object

Specialties with no shredout table (for the coverage report). This is a normal condition for many specialties, not an error.



193
194
195
196
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 193

def specialties_without_shredouts
  build if @index.nil?
  @index.select { |_, e| e[:shredouts].empty? }.keys
end

#title_inventory ⇒ Array<Hash>

Full title inventory for the de-gluing step: every specialty with its current (title-cased) name and the raw pre-titlecase source title.

Returns:

  • (Array<Hash>) —

    name:, raw_title:, glued:



208
209
210
211
212
213
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 208

def title_inventory
  build if @index.nil?
  @index.sort_by { |k, _| k.to_s }.map do |specialty, e|
    {specialty: specialty, name: e[:name], raw_title: e[:raw_title], glued: e[:glued_title]}
  end
end

#unverified? ⇒ Boolean

True if any emitted value (code, applied title, acronym, or shredout value) failed verification. The CLI must fail the build when this is true.

Returns:

  • (Boolean) 


145
146
147
148
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 145

def unverified?
  unverified_codes.any? || unverified_titles.any? || unverified_acronyms.any? ||
    unverified_shredouts.any?
end

#unverified_acronyms ⇒ Object

Emitted :acronym values that could NOT be found (whitespace/case tolerant) in their specialty's raw source title. Non-empty means a drifting/hallucinated acronym; the build must fail. See #unverified?.



128
129
130
131
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 128

def unverified_acronyms
  build if @unverified_acronyms.nil?
  @unverified_acronyms
end

#unverified_codes ⇒ Object

Values emitted into the index that could NOT be found verbatim in the source. Empty by construction for the current deterministic extraction; non-empty means a transform (e.g. the de-gluer) emitted an ungrounded value. See the class docstring.



103
104
105
106
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 103

def unverified_codes
  build if @unverified_codes.nil?
  @unverified_codes
end

#unverified_shredouts ⇒ Object

Applied shredout-value overrides that violate the de-gluing invariant (their letters/digits/punctuation differ from the raw extracted value) or target a shredout absent from the source, each a Hash of suffix:, applied:, raw:, reason:. Non-empty means a stale/incorrect override; the build must fail. See #unverified?.



138
139
140
141
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 138

def unverified_shredouts
  build if @unverified_shredouts.nil?
  @unverified_shredouts
end

#unverified_titles ⇒ Object

Applied title overrides that violate the de-gluing invariant (their letters/digits/punctuation differ from the raw source title), each a Hash of applied:, raw_title:, reason:. Non-empty means a stale/incorrect override; the build must fail. See #unverified?.



112
113
114
115
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 112

def unverified_titles
  build if @unverified_titles.nil?
  @unverified_titles
end

#verified?(value) ⇒ Boolean

The verification-gate predicate: does value appear verbatim in source?

Returns:

  • (Boolean) 


181
182
183
 
# File 'lib/gov_codes/dafecd/index_builder.rb', line 181

def verified?(value)
  @source_text.include?(value.to_s)
end