Class: Wp2txt::FtsIndex

Inherits:
Object
  • Object
show all
Defined in:
lib/wp2txt/fts_index.rb

Overview

Tier 2: contentless FTS5 full-text index over cleaned section text. Stores only the inverted index plus a rowid -> (page_id, heading, ord) mapping; the dump itself remains the single source of truth for text (snippets are re-rendered on demand via multistream random access). Queries ATTACH the Tier 1 metadata DB so category/section/redirect filters compose with MATCH in plain SQL.

Constant Summary collapse

SCHEMA_VERSION = 
2
CACHE_SUFFIX = 
"_fts.sqlite3"
TRIGRAM_LANGS =

Languages without word delimiters: use character-trigram tokenization

%w[ja zh ko yue wuu th km lo my bo].freeze
TOKENIZERS = 
%w[unicode61 trigram porter].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(db_path, meta_db_path) ⇒ FtsIndex

Returns a new instance of FtsIndex.



31
32
33
34
35
 
# File 'lib/wp2txt/fts_index.rb', line 31

def initialize(db_path, meta_db_path)
  @db_path = db_path
  @meta_db_path = meta_db_path
  @db = nil
end

Instance Attribute Details

#db_path ⇒ Object (readonly)

Returns the value of attribute db_path.



29
30
31
 
# File 'lib/wp2txt/fts_index.rb', line 29

def db_path
  @db_path
end

#meta_db_path ⇒ Object (readonly)

Returns the value of attribute meta_db_path.



29
30
31
 
# File 'lib/wp2txt/fts_index.rb', line 29

def meta_db_path
  @meta_db_path
end

Class Method Details

.current_render_digest ⇒ Object

Digest of everything that determines how section text is rendered. Snippets re-render sections at query time and must reproduce what was indexed; a digest mismatch means the index was built by code whose rendering differs from the current code.

Parameters:

  • optimize (Boolean) —

    merge FTS segments into one (single-threaded and slow on large indexes; skip and run #optimize! later if build time matters)



146
147
148
 
# File 'lib/wp2txt/fts_index.rb', line 146

def self.current_render_digest
  Digest::MD5.hexdigest(SectionRenderer::RENDER_CONFIG.inspect)[0, 12]
end

.default_tokenizer(multistream_path) ⇒ Object

Pick a tokenizer from the dump's language (e.g. "jawiki-..." -> trigram)



45
46
47
48
 
# File 'lib/wp2txt/fts_index.rb', line 45

def self.default_tokenizer(multistream_path)
  lang = File.basename(multistream_path)[/\A([a-z_-]+?)wiki/, 1]
  TRIGRAM_LANGS.include?(lang) ? "trigram" : "unicode61"
end

.path_for(multistream_path, cache_dir: nil) ⇒ Object 



37
38
39
40
41
42
 
# File 'lib/wp2txt/fts_index.rb', line 37

def self.path_for(multistream_path, cache_dir: nil)
  dir = cache_dir || File.expand_path("~/.wp2txt/cache")
  basename = File.basename(multistream_path, ".*").sub(/\.xml\z/, "")
  path_hash = Digest::MD5.hexdigest(multistream_path)[0, 8]
  File.join(dir, "#{basename}_#{path_hash}#{CACHE_SUFFIX}")
end

Instance Method Details

#built? ⇒ Boolean

Status

Returns:

  • (Boolean) 


54
55
56
57
58
59
60
61
 
# File 'lib/wp2txt/fts_index.rb', line 54

def built?
  return false unless File.exist?(@db_path)

  meta = read_metadata
  !meta.nil? && meta[:schema_version].to_i == SCHEMA_VERSION && !meta[:built_at].nil?
rescue SQLite3::Exception
  false
end

#close ⇒ Object 



88
89
90
91
 
# File 'lib/wp2txt/fts_index.rb', line 88

def close
  @db&.close
  @db = nil
end

#finalize_build!(source_path, optimize: true) ⇒ Object 



150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
 
# File 'lib/wp2txt/fts_index.rb', line 150

def finalize_build!(source_path, optimize: true)
  db = open_db
  db.execute("CREATE INDEX IF NOT EXISTS idx_fts_map_page ON fts_map(page_id)")
  db.execute("INSERT INTO fts_sections(fts_sections) VALUES('optimize')") if optimize
  stat = File.stat(source_path)
  save_metadata(
    schema_version: SCHEMA_VERSION,
    wp2txt_version: Wp2txt::VERSION,
    render_digest: self.class.current_render_digest,
    tokenizer: @pending_tokenizer,
    source_path: source_path,
    source_size: stat.size,
    source_mtime: stat.mtime.to_i,
    optimized: optimize,
    built_at: Time.now.utc.iso8601
  )
  close
  if @build_path
    File.rename(@build_path, @db_path)
    FileUtils.rm_f(["#{@db_path}-wal", "#{@db_path}-shm"])
    @build_path = nil
  end
end

#insert_batch(rows) ⇒ Object

rows: [[page_id, heading, ord, text], ...]



125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/wp2txt/fts_index.rb', line 125

def insert_batch(rows)
  db = open_db
  db.transaction do
    fts_stmt = db.prepare("INSERT INTO fts_sections (rowid, text) VALUES (?, ?)")
    map_stmt = db.prepare("INSERT INTO fts_map (rowid, page_id, heading, ord) VALUES (?, ?, ?, ?)")
    rows.each do |page_id, heading, ord, text|
      @next_rowid += 1
      fts_stmt.execute([@next_rowid, text])
      map_stmt.execute([@next_rowid, page_id, heading, ord])
    end
    fts_stmt.close
    map_stmt.close
  end
end

#optimize! ⇒ Object

Merge all FTS segments of an existing index (idempotent). Queries work without this, just somewhat slower; run once, any time after a --no-fts-optimize build.

Raises:

  • (ArgumentError) 


183
184
185
186
187
188
189
190
191
 
# File 'lib/wp2txt/fts_index.rb', line 183

def optimize!
  raise ArgumentError, "index not built" unless built?

  db = open_db
  db.execute("INSERT INTO fts_sections(fts_sections) VALUES('optimize')")
  db.execute("PRAGMA wal_checkpoint(TRUNCATE)")
  save_metadata(optimized: true)
  true
end

#optimized? ⇒ Boolean

Returns:

  • (Boolean) 


193
194
195
 
# File 'lib/wp2txt/fts_index.rb', line 193

def optimized?
  read_metadata&.dig(:optimized) == "true"
end

#prepare_build!(tokenizer:) ⇒ Object

Build into a sidecar file and atomically rename in finalize_build!, so a failed multi-hour rebuild never destroys a working index

Raises:

  • (ArgumentError) 


99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/wp2txt/fts_index.rb', line 99

def prepare_build!(tokenizer:)
  raise ArgumentError, "unknown tokenizer: #{tokenizer}" unless TOKENIZERS.include?(tokenizer)

  FileUtils.mkdir_p(File.dirname(@db_path))
  close
  @build_path = "#{@db_path}.building"
  FileUtils.rm_f([@build_path, "#{@build_path}-wal", "#{@build_path}-shm"])
  db = open_db
  db.execute("CREATE TABLE metadata (key TEXT PRIMARY KEY, value TEXT)")
  db.execute("CREATE VIRTUAL TABLE fts_sections USING fts5(text, content='', tokenize='#{tokenizer}')")
  db.execute(<<~SQL)
    CREATE TABLE fts_map (
      rowid INTEGER PRIMARY KEY,
      page_id INTEGER,
      -- heading: normalized like page_sections.heading; '' for the lead text
      heading TEXT,
      -- ord: section position in the article; 0 = lead text (stored here,
      -- unlike page_sections which starts at 1). Same numbering otherwise.
      ord INTEGER
    )
  SQL
  @pending_tokenizer = tokenizer
  @next_rowid = 0
end

#render_current? ⇒ Boolean

True when the current code's rendering configuration matches what this index was built with (snippet fidelity guarantee)

Returns:

  • (Boolean) 


176
177
178
 
# File 'lib/wp2txt/fts_index.rb', line 176

def render_current?
  read_metadata&.dig(:render_digest) == self.class.current_render_digest
end

#search(query, mode: "phrase", sections: nil, category: nil, depth: 0, limit: 20, offset: 0, count: "capped", count_cap: 1000) ⇒ Hash

Returns { total:, total_is_capped:, hits: [title:, heading:, ord:] }.

Parameters:

  • query (String) —

    search string

  • mode (String) (defaults to: "phrase") —

    "phrase" (literal, default) or "query" (raw FTS5 syntax)

  • sections (Array<String>, nil) (defaults to: nil) —

    restrict to these headings

  • category (String, nil) (defaults to: nil) —

    category scope (recursion via depth)

  • count (String) (defaults to: "capped") —

    "capped" (default; stops at count_cap) or "exact"

Returns:

  • (Hash) —

    { total:, total_is_capped:, hits: [title:, heading:, ord:] }



207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
 
# File 'lib/wp2txt/fts_index.rb', line 207

def search(query, mode: "phrase", sections: nil, category: nil, depth: 0,
           limit: 20, offset: 0, count: "capped", count_cap: 1000)
  match_expr = mode == "query" ? query : phrase_query(query)

  conds = ["fts_sections MATCH ?", "p.namespace = 0", "p.redirect_to IS NULL"]
  params = [match_expr]
  cte = nil

  if category
    cte, cond, cat_params = category_condition(category, depth)
    conds << cond
    params = cat_params + params if cte     # CTE placeholders precede MATCH in SQL order
    params += cat_params unless cte
  end

  if sections && !sections.empty?
    placeholders = sections.map { "?" }.join(",")
    conds << "fm.heading COLLATE NOCASE IN (#{placeholders})"
    params += sections
  end

  base = "FROM fts_sections f JOIN fts_map fm ON fm.rowid = f.rowid " \
         "JOIN meta.pages p ON p.page_id = fm.page_id WHERE #{conds.join(' AND ')}"
  with = cte ? "WITH RECURSIVE #{cte} " : ""

  db = attached_db
  hits = db.execute(
    "#{with}SELECT fm.page_id, p.title, fm.heading, fm.ord #{base} ORDER BY rank LIMIT ? OFFSET ?",
    params + [limit, offset]
  ).map { |page_id, title, heading, ord| { page_id: page_id, title: title, heading: heading, ord: ord } }

  if count == "exact"
    total = db.get_first_value("#{with}SELECT COUNT(*) #{base}", params).to_i
    { total: total, total_is_capped: false, hits: hits }
  else
    capped = db.get_first_value(
      "#{with}SELECT COUNT(*) FROM (SELECT 1 #{base} LIMIT #{count_cap.to_i + 1})", params
    ).to_i
    { total: [capped, count_cap].min, total_is_capped: capped > count_cap, hits: hits }
  end
end

#stats ⇒ Object 



76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/wp2txt/fts_index.rb', line 76

def stats
  return nil unless File.exist?(@db_path)

  meta = read_metadata || {}
  { db_path: @db_path, db_size: File.size(@db_path),
    tokenizer: meta[:tokenizer], built_at: meta[:built_at],
    built_with: meta[:wp2txt_version],
    render_current: meta[:render_digest] == self.class.current_render_digest,
    optimized: meta[:optimized] == "true",
    section_count: (open_db.get_first_value("SELECT COUNT(*) FROM fts_map") rescue 0) }
end

#tokenizer ⇒ Object 



72
73
74
 
# File 'lib/wp2txt/fts_index.rb', line 72

def tokenizer
  read_metadata&.dig(:tokenizer)
end

#valid_for?(multistream_path) ⇒ Boolean

Returns:

  • (Boolean) 


63
64
65
66
67
68
69
70
 
# File 'lib/wp2txt/fts_index.rb', line 63

def valid_for?(multistream_path)
  return false unless built?
  return false unless File.exist?(multistream_path)

  meta = read_metadata
  stat = File.stat(multistream_path)
  meta[:source_size].to_i == stat.size && meta[:source_mtime].to_i == stat.mtime.to_i
end