Class: Wp2txt::Corpus

Inherits:
Object
  • Object
show all
Includes:
Wp2txt, Formatter
Defined in:
lib/wp2txt/corpus.rb

Overview

Facade over a local dump: single-article access (Tier 0, multistream), exhaustive metadata queries (Tier 1, MetadataIndex), and corpus extraction. Shared by the CLI and the MCP server so both expose identical behavior.

Defined Under Namespace

Classes: Cancelled, LazyTitleIndex

Constant Summary collapse

DEFAULT_MAX_SYNC_ARTICLES =

Sync extraction cap: larger requests need the (future) job API

5000
RENDER_CONFIG = 
{
  format: :text,
  title: true, heading: true, list: false, table: false, pre: false,
  ref: false, redirect: false, multiline: false,
  category: true, category_only: false, summary_only: false, metadata_only: false,
  marker: true, markers: true, extract_citations: false, expand_templates: true,
  sections: nil, section_output: "structured", min_section_length: 0,
  skip_empty: false, alias_file: nil, no_section_aliases: false,
  show_matched_sections: false
}.freeze
DEFAULT_MAX_CHARS =

Default cap on article text returned inline (LLM context economy); callers can raise it explicitly, and truncation is always flagged

40_000
GUARDRAIL_MAX_RATIO =

Guardrail thresholds: pairs whose co-occurrence ratio exceeds GUARDRAIL_MAX_RATIO (with both headings above GUARDRAIL_MIN_ARTICLES) coexist in the same articles and are likely NOT synonyms

0.2
GUARDRAIL_MIN_ARTICLES = 
100
SNIPPET_CONTEXT =

Save an alias set after mechanically verifying each group: high co-occurrence pairs block the save unless force is set, so protocol compliance does not depend on the calling model's discipline.

Tier 2: full-text search

80
EXTRACT_BATCH_SIZE =

Titles fetched/rendered per batch while streaming to disk

200
TITLES_MAX =

Max titles accepted by extract_corpus titles: (larger sets belong in filter-based extraction or a background job)

10_000
SQL_ROW_LIMIT =

Read-only SQL (escape hatch for queries the fixed tools cannot express)

200
SQL_CELL_LIMIT = 
2000
SQL_TIMEOUT_SECONDS = 
30
SQL_FORBIDDEN = 
/\b(ATTACH|DETACH|PRAGMA|INSERT|UPDATE|DELETE|DROP|CREATE|ALTER|REPLACE|VACUUM|REINDEX)\b/i
SQL_FILE_ROW_LIMIT =

File-output mode (query_sql output_path:): hard row cap and per-cell clip (insurance against runaway blobs, not context economy)

5_000_000
SQL_FILE_CELL_LIMIT = 
64 * 1024

Constants included from Wp2txt

AUTHORITY_CONTROL_REGEX, AUTHORITY_CONTROL_TEMPLATES, BLANK_LINE_REGEX, BLOCK_EXTENSION_TAGS, BLOCK_MARKERS, BYTES_PER_GB, BYTES_PER_KB, BYTES_PER_MB, CATEGORY_LINE_REGEX, CATEGORY_NAMESPACES, CATEGORY_NAMESPACE_REGEX, CATEGORY_REGEX, CHRREF_TO_UTF_REGEX, CITATION_TEMPLATES, CITATION_TEMPLATE_REGEX, CLEANUP_EMPTY_PARENS_REGEX, CLEANUP_FILE_INCOMPLETE_REGEX, CLEANUP_FILE_LINE_REGEX, CLEANUP_KEY_VALUE_LINE_REGEX, CLEANUP_MIXED_WHITESPACE_REGEX, CLEANUP_MULTIPLE_PIPES_REGEX, CLEANUP_MULTIPLE_SPACES_REGEX, CLEANUP_MULTI_BLANK_REGEX, CLEANUP_ORPHANED_BRACKETS_REGEX, CLEANUP_ORPHANED_CLOSE_REGEX, CLEANUP_ORPHANED_OPEN_REGEX, CLEANUP_PIPE_CLOSE_REGEX, CLEANUP_PIPE_LINE_REGEX, CLEANUP_REGEX_01, CLEANUP_REGEX_02, CLEANUP_REGEX_03, CLEANUP_REGEX_04, CLEANUP_REGEX_05, CLEANUP_REGEX_06, CLEANUP_REGEX_07, CLEANUP_REGEX_08, CLEANUP_REMNANTS_REGEX, CLEANUP_REMNANTS_TEMPLATES, CLEANUP_STANDALONE_CLOSE_REGEX, CLEANUP_TRAILING_PIPE_REGEX, COMPLEX_REGEX_01, COMPLEX_REGEX_02, COMPLEX_REGEX_03, COMPLEX_REGEX_04, COMPLEX_REGEX_05, CONVERT_TEMPLATES, COUNTRY_CODE_REGEX, CURLY_SQUARE_BRACKET_REGEX, DEFAULTSORT_KEYWORDS, DEFAULT_BUFFER_SIZE, DEFAULT_CATEGORY_CACHE_EXPIRY_DAYS, DEFAULT_DUMP_EXPIRY_DAYS, DEFAULT_HTTP_TIMEOUT, DEFAULT_MARKERS, DEFAULT_PROGRESS_INTERVAL, DEFAULT_TOP_N_SECTIONS, DEF_MARKS_REGEX, DISPLAYTITLE_KEYWORDS, DOUBLE_CURLY_BRACKET_REGEX, DOUBLE_SQUARE_BRACKET_REGEX, DOUBLE_UNDERSCORE_MAGIC_REGEX, DOUBLE_UNDERSCORE_PATTERNS, ENTITIES, ESCAPE_NOWIKI_REGEX, EXTENSION_TAGS, EXTRA_ENTITIES, EXTRA_ENTITIES_REGEX, FILE_NAMESPACES, FILE_NAMESPACES_REGEX, FLAG_TEMPLATES, FLAG_TEMPLATE_REGEX, FORMATTING_TEMPLATES, FORMATTING_TEMPLATE_REGEX, FORMAT_REF_REGEX, HEADING_CODA_REGEX, HEADING_ONSET_REGEX, HTML_COMMENT_REGEX, HTML_DECODER, HTML_ENTITIES_PATH, HTML_HASH, HTML_REGEX, IMAGEMAP_COORD_REGEX, IMAGE_PARAMS, IMAGE_PARAMS_REGEX, IMAGE_PARAM_KEYS, INDEX_PROGRESS_THRESHOLD, INLINE_MARKERS, INTERWIKI_LINK_TEMPLATES, INTERWIKI_PREFIX_REGEX, IN_DEFINITION_REGEX, IN_HEADING_REGEX, IN_HTML_TABLE_REGEX, IN_HTML_TABLE_REGEX1, IN_HTML_TABLE_REGEX2, IN_INPUTBOX_REGEX, IN_INPUTBOX_REGEX1, IN_INPUTBOX_REGEX2, IN_LINK_REGEX, IN_MATH_REGEX, IN_MATH_REGEX1, IN_MATH_REGEX2, IN_ORDERED_REGEX, IN_PRE_REGEX, IN_SOURCE_REGEX, IN_SOURCE_REGEX1, IN_SOURCE_REGEX2, IN_TABLE_REGEX1, IN_TABLE_REGEX2, IN_UNORDERED_REGEX, ISOLATED_TAG_REGEX, ISOLATED_TEMPLATE_REGEX, LIST_MARKS_REGEX, LONE_ASTERISK_REGEX, MAGIC_WORD_LINE_REGEX, MAGIC_WORD_TEMPLATE_REGEX, MAKE_REFERENCE_REGEX_A, MAKE_REFERENCE_REGEX_B, MAKE_REFERENCE_REGEX_C, MAKE_REFERENCE_REGEX_D, MARKER_PATTERNS, MARKER_TYPES, MATH_ENTITIES, MATH_ENTITIES_REGEX, MAX_BUFFER_SIZE, MAX_HTTP_RETRIES, MAX_NESTING_ITERATIONS, MEDIAWIKI_DATA_PATH, MIN_BUFFER_SIZE, MIXED_SCRIPT_TEMPLATES, ML_LINK_END_REGEX, ML_LINK_ONSET_REGEX, ML_TEMPLATE_END_REGEX, ML_TEMPLATE_ONSET_REGEX, MNDASH_REGEX, NON_ARTICLE_NAMESPACES, NON_ARTICLE_NAMESPACE_REGEX, ONSET_BAR_REGEX, PRE_MARKS_REGEX, RACTOR_SHAREABLE_EXCLUDES, REDIRECT_KEYWORDS, REDIRECT_REGEX, REMOVE_DIRECTIVES_REGEX, REMOVE_DIRECTIVES_REGEX_GENERIC, REMOVE_EMPHASIS_REGEX, REMOVE_HR_REGEX, REMOVE_INLINE_REGEX, REMOVE_ISOLATED_REGEX, REMOVE_TAG_REGEX, REMOVE_TEMPLATES, REMOVE_TEMPLATES_REGEX, RESUME_METADATA_MAX_AGE_DAYS, RUBY_TEXT_TEMPLATES, SECONDS_PER_DAY, SECONDS_PER_HOUR, SECONDS_PER_MINUTE, SELF_CLOSING_TAG_REGEX, SINGLE_CURLY_BRACKET_REGEX, SINGLE_SQUARE_BRACKET_REGEX, SISTER_PROJECTS, TEMPLATE_DATA_PATH, UNESCAPE_NOWIKI_REGEX, VERSION, WIKIMEDIA_PROJECT_NAMES, WIKIMEDIA_PROJECT_REGEX, WIKIPEDIA_ENTITIES_PATH

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Formatter

#build_text_content, #extract_redirect, #format_article, #format_article_json, #format_article_text, #format_category_only, #format_full_article, #format_metadata_only, #format_metadata_only_json, #format_metadata_only_text, #format_sections_combined_json, #format_sections_combined_text, #format_sections_structured_json, #format_sections_structured_text, #format_with_categories, #format_with_sections, #formatter_debug_mode, #process_element

Methods included from Wp2txt

#apply_markers, #apply_pipe_trick, article_page?, #batch_file_mod, build_extra_entities_regex, build_template_regex, #chrref_to_utf, #cleanup, #collect_files, #convert_characters, #correct_inline_template, #correct_separator, days_to_seconds, #escape_nowiki, #extract_template_content, file_age_days, file_fresh?, #file_mod, #finalize_markers, #format_citation, format_file_size, #format_wiki, #format_wiki_regex_transform, #html_decoder, load_html_entities, load_mediawiki_data, load_template_data, make_constants_ractor_shareable!, #make_reference, #marker_placeholder, #mndash, #parse_markers_config, #process_external_links, #process_interwiki_links, #process_nested_single_pass, #process_nested_structure, #remove_complex, #remove_directive, #remove_emphasis, #remove_hr, #remove_html, #remove_inbetween, #remove_ref, #remove_table, #remove_tag, #remove_templates, #rename, #replace_paired_templates_with_marker, #replace_template_with_marker, #replace_wiki_table_with_marker, #sec_to_str, #special_chr, ssl_safe_get, #template_matches?, #unescape_nowiki

Constructor Details

#initialize(multistream_path:, index_path:, cache_dir: nil) ⇒ Corpus

Returns a new instance of Corpus.



63
64
65
66
67
68
 
# File 'lib/wp2txt/corpus.rb', line 63

def initialize(multistream_path:, index_path:, cache_dir: nil)
  @multistream_path = multistream_path
  @index_path = index_path
  @cache_dir = cache_dir
  @metadata = MetadataIndex.new(MetadataIndex.path_for(multistream_path, cache_dir: cache_dir))
end

Instance Attribute Details

#index_path ⇒ Object (readonly)

Returns the value of attribute index_path.



61
62
63
 
# File 'lib/wp2txt/corpus.rb', line 61

def index_path
  @index_path
end

#metadata ⇒ Object (readonly)

Returns the value of attribute metadata.



61
62
63
 
# File 'lib/wp2txt/corpus.rb', line 61

def metadata
  @metadata
end

#multistream_path ⇒ Object (readonly)

Returns the value of attribute multistream_path.



61
62
63
 
# File 'lib/wp2txt/corpus.rb', line 61

def multistream_path
  @multistream_path
end

Class Method Details

.for_input(multistream_path, cache_dir: nil) ⇒ Object

Raises:

  • (ArgumentError) 


83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/wp2txt/corpus.rb', line 83

def self.for_input(multistream_path, cache_dir: nil)
  candidates = [
    multistream_path.sub(/multistream\.xml\.bz2\z/, "multistream-index.txt.bz2"),
    multistream_path.sub(/\.xml\.bz2\z/, "-index.txt.bz2"),
    multistream_path.sub(/\.xml\.bz2\z/, "-index.txt")
  ].uniq
  index = candidates.find { |c| c != multistream_path && File.exist?(c) }
  raise ArgumentError, "Multistream index file not found next to #{multistream_path}" unless index

  new(multistream_path: multistream_path, index_path: index, cache_dir: cache_dir)
end

.for_lang(lang, cache_dir: nil) ⇒ Object

Build a Corpus from a language code using the DumpManager cache. Raises with guidance when the dump has not been downloaded yet.



72
73
74
75
76
77
78
79
80
81
 
# File 'lib/wp2txt/corpus.rb', line 72

def self.for_lang(lang, cache_dir: nil)
  manager = DumpManager.new(lang, cache_dir: cache_dir)
  multistream = manager.cached_multistream_path
  index = manager.cached_index_path
  unless File.exist?(multistream) && File.exist?(index)
    raise ArgumentError, "No cached dump for '#{lang}'. Run: wp2txt --build-index -L #{lang}"
  end

  new(multistream_path: multistream, index_path: index, cache_dir: cache_dir)
end

Instance Method Details

#category_tree(category, depth: 2) ⇒ Object 



205
206
207
 
# File 'lib/wp2txt/corpus.rb', line 205

def category_tree(category, depth: 2)
  { dump: dump_name, tree: @metadata.category_tree(category, depth: depth) }
end

#close ⇒ Object 



549
550
551
 
# File 'lib/wp2txt/corpus.rb', line 549

def close
  close_read_connections
end

#describe_schema ⇒ Object

CREATE statements of all tables available to query_sql



540
541
542
543
544
545
546
547
 
# File 'lib/wp2txt/corpus.rb', line 540

def describe_schema
  db = readonly_db
  schemas = { meta: db.execute("SELECT sql FROM sqlite_master WHERE sql IS NOT NULL").map(&:first) }
  if fts.built?
    schemas[:fts] = db.execute("SELECT sql FROM fts.sqlite_master WHERE sql IS NOT NULL").map(&:first)
  end
  schemas
end

#dump_info ⇒ Object 



103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
 
# File 'lib/wp2txt/corpus.rb', line 103

def dump_info
  stats = @metadata.built? ? @metadata.stats : nil
  {
    multistream_path: @multistream_path,
    dump: stats&.dig(:dump_name) || File.basename(@multistream_path),
    tiers: {
      titles: File.exist?(@index_path),
      metadata: @metadata.built?,
      fulltext: fts.built?
    },
    metadata_current: @metadata.built? && @metadata.valid_for?(@multistream_path),
    fulltext_current: fts.built? && fts.valid_for?(@multistream_path),
    stats: stats,
    fulltext: fts.built? ? fts.stats : nil,
    langlinks: @metadata.built? ? @metadata.langlinks_provenance : nil
  }
end

#extract_corpus(output_path:, content: "sections", sections: nil, alias_set: nil, category: nil, depth: 0, categories: nil, category_match: nil, title_match: nil, limit: 0, titles: nil, chunk_size: nil, chunk_overlap: 0, max_articles: DEFAULT_MAX_SYNC_ARTICLES, num_processes: 4, progress: nil, cancel_check: nil) ⇒ Object

Parameters:

  • output_path (String) —

    JSONL destination (sidecar .meta.json is added)

  • content (String) (defaults to: "sections") —

    "sections" | "full" | "summary"

  • titles (Array<String>, nil) (defaults to: nil) —

    explicit article titles to extract (e.g. a set determined via query_sql). Normalized, deduplicated, one redirect hop resolved; missing titles are reported as not_found. Mutually exclusive with the filter arguments

  • chunk_size (Integer, nil) (defaults to: nil) —

    split text into ~N-char chunks (RAG-ready records)

  • chunk_overlap (Integer) (defaults to: 0) —

    overlap between consecutive chunks

  • max_articles (Integer, nil) (defaults to: DEFAULT_MAX_SYNC_ARTICLES) —

    sync cap (nil = unlimited, for jobs)

  • progress (Proc, nil) (defaults to: nil) —

    called with (titles_done, titles_total) after each batch

  • cancel_check (Proc, nil) (defaults to: nil) —

    polled between batches; truthy return aborts with Cancelled

Raises:

  • (ArgumentError) 


322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
 
# File 'lib/wp2txt/corpus.rb', line 322

def extract_corpus(output_path:, content: "sections", sections: nil, alias_set: nil,
                   category: nil, depth: 0, categories: nil, category_match: nil,
                   title_match: nil, limit: 0, titles: nil,
                   chunk_size: nil, chunk_overlap: 0,
                   max_articles: DEFAULT_MAX_SYNC_ARTICLES, num_processes: 4,
                   progress: nil, cancel_check: nil)
  if content == "sections" && Array(sections).empty? && alias_set.nil?
    raise ArgumentError, "content: \"sections\" requires sections or alias_set"
  end
  raise ArgumentError, "chunk_overlap must be smaller than chunk_size" if chunk_size && chunk_overlap >= chunk_size
  raise ArgumentError, "chunking is not supported for content: \"wikitext\"" if chunk_size && content == "wikitext"

  if titles
    raise ArgumentError, "titles must be an array of title strings" unless titles.is_a?(Array)
    if titles.size > TITLES_MAX
      raise ArgumentError, "titles accepts at most #{TITLES_MAX} titles (got #{titles.size}); " \
                           "use filter-based extraction or a background job for larger sets"
    end
    conflicts = []
    conflicts << "category" if category
    conflicts << "categories" if categories
    conflicts << "category_match" if category_match
    conflicts << "title_match" if title_match
    unless conflicts.empty?
      raise ArgumentError, "titles cannot be combined with #{conflicts.join(', ')} — the article " \
                           "set would be defined twice; perform set operations in query_sql and " \
                           "pass the resulting titles via titles:"
    end
  end

  filters = { category: category, depth: depth, categories: categories,
              category_match: category_match, sections: sections,
              alias_set: alias_set, title_match: title_match }
  not_found = nil
  titles_record = nil
  if titles
    normalized = titles.map { |t| MetadataIndex.normalize_title(t) }.reject(&:empty?).uniq
    titles_record = { titles_count: normalized.size,
                      titles_sha256: Digest::SHA256.hexdigest(normalized.sort.join("\n")) }
    titles_record[:titles] = normalized if normalized.size <= 100
    total = normalized.size
    resolved, missing = resolve_explicit_titles(normalized)
    not_found = { count: missing.size, sample: missing.first(20) }
    cap = if limit.positive?
            max_articles ? [limit, max_articles].min : limit
          else
            max_articles || resolved.size
          end
    titles = resolved.first(cap)
    # Truncated means "cut by the cap" only; shortfalls from missing
    # titles are explained by not_found, not by this flag
    truncated = resolved.size > titles.size
  else
    total = @metadata.count_articles(**filters)
    cap = if limit.positive?
            max_articles ? [limit, max_articles].min : limit
          else
            max_articles || total
          end
    titles = @metadata.find_articles(**filters, limit: cap)
    truncated = total > titles.size
  end

  resolved_sections = content == "summary" ? [SectionExtractor::SUMMARY_KEY] : expand_with_alias_set(sections, alias_set)
  alias_contents = alias_set ? get_alias_set(alias_set)&.dig(:groups) : nil

  # Close ALL SQLite connections before MultistreamReader forks workers
  # (children must not inherit open database handles)
  close_read_connections

  articles_extracted = 0
  records_written = 0
  titles_done = 0
  sample = []

  File.open(output_path, "w") do |f|
    titles.each_slice(EXTRACT_BATCH_SIZE) do |batch|
      raise Cancelled if cancel_check&.call

      pages = reader.extract_articles_parallel(batch, num_processes: num_processes)
      batch.each do |t|
        page = pages[t]
        next unless page

        records = build_records(page, content, resolved_sections, chunk_size, chunk_overlap)
        next if records.empty?

        articles_extracted += 1
        records.each do |record|
          f.puts(JSON.generate(record))
          records_written += 1
          sample << record if sample.size < 3
        end
      end
      titles_done += batch.size
      progress&.call(titles_done, titles.size)
    end
  end

  meta_path = "#{output_path}.meta.json"
  File.write(meta_path, JSON.pretty_generate(
    tool: "wp2txt #{Wp2txt::VERSION}",
    dump: dump_name,
    generated_at: Time.now.utc.iso8601,
    query: (titles_record || filters.compact).merge(
      content: content, resolved_sections: resolved_sections,
      chunk_size: chunk_size, chunk_overlap: chunk_size ? chunk_overlap : nil
    ).compact,
    alias_set_contents: alias_contents,
    total_matching: total,
    articles_extracted: articles_extracted,
    records_written: records_written,
    truncated: truncated,
    not_found: not_found
  ))

  { output_path: output_path, meta_path: meta_path, dump: dump_name,
    total_matching: total, articles_extracted: articles_extracted,
    records_written: records_written, truncated: truncated,
    bytes: File.size(output_path), sample: sample,
    not_found: not_found }.compact
end

#find_articles(**filters) ⇒ Object

Tier 1: exhaustive queries (delegated to MetadataIndex)



197
198
199
200
201
202
203
 
# File 'lib/wp2txt/corpus.rb', line 197

def find_articles(**filters)
  limit = filters.delete(:limit) || 0
  offset = filters.delete(:offset) || 0
  total = @metadata.count_articles(**filters)
  titles = @metadata.find_articles(**filters, limit: limit, offset: offset)
  { dump: dump_name, total: total, returned: titles.size, titles: titles }
end

#fts ⇒ Object 



121
122
123
124
125
126
 
# File 'lib/wp2txt/corpus.rb', line 121

def fts
  @fts ||= FtsIndex.new(
    FtsIndex.path_for(@multistream_path, cache_dir: @cache_dir),
    @metadata.db_path
  )
end

#get_alias_set(name) ⇒ Object 



288
289
290
 
# File 'lib/wp2txt/corpus.rb', line 288

def get_alias_set(name)
  @metadata.get_alias_set(name)
end

#get_article(title, format: "text", follow_redirect: true, max_chars: DEFAULT_MAX_CHARS) ⇒ Object

Parameters:

  • format (String) (defaults to: "text") —

    "text" (cleaned), "wikitext" (raw markup)

  • follow_redirect (Boolean) (defaults to: true) —

    resolve one redirect hop

  • max_chars (Integer, nil) (defaults to: DEFAULT_MAX_CHARS) —

    truncate text beyond this length (nil = no cap)



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
 
# File 'lib/wp2txt/corpus.rb', line 139

def get_article(title, format: "text", follow_redirect: true, max_chars: DEFAULT_MAX_CHARS)
  page = fetch_page(title, follow_redirect: follow_redirect)
  return nil unless page

  body = case format.to_s
         when "wikitext"
           page[:text]
         else
           render_text(page)
         end
  result = { id: page[:id], title: page[:title], format: format.to_s }
  if max_chars && body.length > max_chars
    result.merge(text: body[0, max_chars], truncated: true, total_chars: body.length)
  else
    result.merge(text: body)
  end
end

#get_categories(title) ⇒ Object

Categories of one article, resolved through the same title normalization



158
159
160
161
162
163
164
165
166
167
 
# File 'lib/wp2txt/corpus.rb', line 158

def get_categories(title)
  cats = @metadata.categories_of(title)
  if cats.nil?
    page = fetch_page(title)
    cats = page ? @metadata.categories_of(page[:title]) : nil
  end
  return nil unless cats

  { title: title, categories: cats }
end

#get_sections(title, sections, alias_set: nil) ⇒ Object

Extract specific sections from one article.

Parameters:

  • sections (Array<String>) —

    section names ("summary" for lead text)

  • alias_set (String, nil) (defaults to: nil) —

    saved alias set used to expand names



172
173
174
175
176
177
178
179
180
181
182
 
# File 'lib/wp2txt/corpus.rb', line 172

def get_sections(title, sections, alias_set: nil)
  page = fetch_page(title)
  return nil unless page

  resolved = expand_with_alias_set(sections, alias_set)
  config = RENDER_CONFIG.merge(format: :json, sections: resolved, title: page[:title])
  article = Article.new(page[:text], page[:title], false)
  result = format_with_sections(article, config)
  { id: page[:id], title: page[:title], requested: sections, resolved: resolved,
    sections: result ? result["sections"] : {} }
end

#list_alias_sets ⇒ Object 



292
293
294
 
# File 'lib/wp2txt/corpus.rb', line 292

def list_alias_sets
  @metadata.list_alias_sets
end

#list_headings(title) ⇒ Object 



184
185
186
187
188
189
190
191
 
# File 'lib/wp2txt/corpus.rb', line 184

def list_headings(title)
  page = fetch_page(title)
  return nil unless page

  article = Article.new(page[:text], page[:title], false)
  { id: page[:id], title: page[:title],
    headings: SectionExtractor.new.extract_headings_with_levels(article) }
end

#metadata_built? ⇒ Boolean

Info

Returns:

  • (Boolean) 


99
100
101
 
# File 'lib/wp2txt/corpus.rb', line 99

def metadata_built?
  @metadata.built?
end

#query_sql(sql, limit: SQL_ROW_LIMIT, timeout: SQL_TIMEOUT_SECONDS, attach: [], output_path: nil, overwrite: false) ⇒ Object

Run a read-only SELECT against the metadata DB (with the FTS DB attached as fts when built). Defense layers: keyword screening (outside string literals), an SQLITE_OPEN_READONLY connection so writes are impossible at the driver level, and subprocess execution with a hard wall-clock timeout — the sqlite3 gem holds the GVL during C execution, so a runaway query can only be stopped by killing the process running it. On timeout, the error message includes an EXPLAIN QUERY PLAN diagnosis when a likely cause (nested full scans, unbounded recursion) is recognizable.

Parameters:

  • attach (Array<String>) (defaults to: []) —

    language codes of other locally installed dumps to ATTACH read-only as lang_meta / lang_fts (e.g. ["en"] → en_meta.pages). Codes are validated and resolved server-side; user SQL itself can never contain ATTACH (SQL_FORBIDDEN).

  • output_path (String, nil) (defaults to: nil) —

    when given, write ALL rows (up to SQL_FILE_ROW_LIMIT) to a JSONL file plus a .meta.json sidecar, and return only a summary + 3-row sample; limit is ignored in this mode

  • overwrite (Boolean) (defaults to: false) —

    replace an existing output file (default: refuse)



509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
 
# File 'lib/wp2txt/corpus.rb', line 509

def query_sql(sql, limit: SQL_ROW_LIMIT, timeout: SQL_TIMEOUT_SECONDS, attach: [],
              output_path: nil, overwrite: false)
  raise ArgumentError, "only SELECT/WITH queries are allowed" unless sql =~ /\A\s*(SELECT|WITH)\b/i
  # Screen keywords outside string literals and quoted identifiers only, so
  # legitimate data values (e.g. title LIKE '%Update%') are not rejected
  screened = sql.gsub(/'(?:[^']|'')*'/m, "''").gsub(/"(?:[^"]|"")*"/m, '""')
  raise ArgumentError, "query contains a forbidden keyword" if screened.match?(SQL_FORBIDDEN)

  attachments = resolve_attachments(attach)
  return query_sql_to_file(sql, timeout, attachments, output_path, overwrite) if output_path

  limit = [[limit.to_i, 1].max, 1000].min
  result = if Process.respond_to?(:fork)
             run_sql_in_subprocess(sql, limit, timeout, attachments)
           elsif attachments.empty?
             run_sql_on(readonly_db, sql, limit)
           else
             db = build_readonly_connection(attach_fts: fts.built?, attachments: attachments)
             begin
               run_sql_on(db, sql, limit)
             ensure
               db.close
             end
           end
  result[:attached] = attachments_summary(attachments) unless attachments.empty?
  result
rescue SQLite3::Exception => e
  raise ArgumentError, "SQL error: #{e.message}"
end

#resolve_explicit_titles(titles) ⇒ Array(Array<String>, Array<String>)

Resolve explicit titles against the pages table: existence plus one redirect hop (same rule as get_article). Input order is preserved.

Returns:

  • (Array(Array<String>, Array<String>)) —

    [found_titles, missing_titles]



448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
 
# File 'lib/wp2txt/corpus.rb', line 448

def resolve_explicit_titles(titles)
  map = @metadata.redirect_map(titles)
  targets = titles.filter_map { |t| map[t] }
                  .map { |t| MetadataIndex.normalize_title(t) }.uniq
  target_map = targets.empty? ? {} : @metadata.redirect_map(targets)

  found = []
  missing = []
  titles.each do |t|
    if !map.key?(t)
      missing << t
    elsif (target = map[t])
      # Redirect: extract under the resolved title; a redirect whose
      # target does not exist counts as not found
      target = MetadataIndex.normalize_title(target)
      if target_map.key?(target)
        found << target
      else
        missing << t
      end
    else
      found << t
    end
  end
  # Resolution can collapse distinct inputs onto one article (two aliases
  # redirecting to the same target, or a direct title plus its alias):
  # dedupe so no article is extracted twice, preserving first-seen order
  [found.uniq, missing]
end

#save_alias_set(name, groups, force: false, max_ratio: GUARDRAIL_MAX_RATIO, min_articles: GUARDRAIL_MIN_ARTICLES) ⇒ Object 



265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
 
# File 'lib/wp2txt/corpus.rb', line 265

def save_alias_set(name, groups, force: false,
                   max_ratio: GUARDRAIL_MAX_RATIO, min_articles: GUARDRAIL_MIN_ARTICLES)
  unless groups.is_a?(Array) && !groups.empty? && groups.all? { |g| g.is_a?(Array) && !g.empty? }
    raise ArgumentError, "groups must be a non-empty array of arrays"
  end

  # Report the exact thresholds used so both the model and the user can see
  # why a group was accepted or rejected, not just that it was
  criteria = { max_cooccurrence_ratio: max_ratio, min_articles: min_articles }
  violations = check_alias_groups(groups, max_ratio: max_ratio, min_articles: min_articles)
  if violations.any? && !force
    return { saved: false, name: name, groups: groups, violations: violations, criteria: criteria,
             warning: "These heading pairs coexist in the same article more than " \
                      "#{(max_ratio * 100).round}% of the time (both headings appear in at least " \
                      "#{min_articles} articles), so they are likely different sections, not synonyms. " \
                      "Remove them from the group, or pass force: true if you verified them another " \
                      "way (e.g., by reading section contents with get_sections)." }
  end

  @metadata.save_alias_set(name, groups)
  { saved: true, name: name, groups: groups, violations: violations, criteria: criteria }
end

#search_text(query, mode: "phrase", sections: nil, alias_set: nil, category: nil, depth: 0, limit: 20, offset: 0, count: "capped", snippets: true) ⇒ Object

Exhaustive full-text search over cleaned section text.

Parameters:

  • mode (String) (defaults to: "phrase") —

    "phrase" (literal, default) or "query" (raw FTS5 syntax)

  • count (String) (defaults to: "capped") —

    "capped" (fast, default) or "exact" (may take seconds for common terms)

  • snippets (Boolean) (defaults to: true) —

    re-render matched sections from the dump for context



239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
 
# File 'lib/wp2txt/corpus.rb', line 239

def search_text(query, mode: "phrase", sections: nil, alias_set: nil,
                category: nil, depth: 0, limit: 20, offset: 0,
                count: "capped", snippets: true)
  unless fts.built?
    raise ArgumentError, "Full-text index not built. Run: wp2txt --build-index --fulltext"
  end

  resolved = expand_with_alias_set(sections, alias_set)
  resolved = nil if resolved.empty?
  result = fts.search(query, mode: mode, sections: resolved, category: category,
                      depth: depth, limit: limit, offset: offset, count: count)

  hits = result[:hits]
  attach_snippets(hits, query, mode) if snippets && !hits.empty?

  { dump: dump_name, query: query, mode: mode,
    total: result[:total], total_is_capped: result[:total_is_capped],
    returned: hits.size,
    hits: hits.map do |h|
      { page_id: h[:page_id], title: h[:title],
        section: h[:heading].to_s.empty? ? nil : h[:heading],
        section_path: h[:heading].to_s.empty? ? h[:title] : "#{h[:title]} > #{h[:heading]}",
        snippet: h[:snippet] }.compact
    end }
end

#section_cooccurrence(headings, category: nil, depth: 0) ⇒ Object 



215
216
217
218
 
# File 'lib/wp2txt/corpus.rb', line 215

def section_cooccurrence(headings, category: nil, depth: 0)
  @metadata.section_cooccurrence(headings, category: category, depth: depth)
           .merge(dump: dump_name)
end

#section_stats(category: nil, depth: 0, top_n: 50) ⇒ Object 



209
210
211
212
213
 
# File 'lib/wp2txt/corpus.rb', line 209

def section_stats(category: nil, depth: 0, top_n: 50)
  { dump: dump_name,
    sections: @metadata.section_stats(category: category, depth: depth, top_n: top_n)
                       .map { |h, c| { heading: h, articles: c } } }
end