Class: Wp2txt::LanglinksImporter

Inherits:
Object
  • Object
show all
Defined in:
lib/wp2txt/langlinks_importer.rb

Overview

Imports the official langlinks dump (langwiki-date-langlinks.sql.gz, MySQL dump format) into the Tier 1 metadata DB as a langlinks table (ll_from = source page_id, ll_lang = target language, ll_title = title in the target edition, normalized like pages.title).

Version pinning is the reason this feature exists: the langlinks file's dump name (e.g. jawiki-20260701) MUST equal the metadata DB's dump_name; a mismatch is rejected with no override.

Constant Summary collapse

BATCH_SIZE =

Rows inserted per transaction (index creation is deferred until after the load, so inserts stay fast)

10_000
SANITY_SAMPLE_SIZE =

Post-import sanity check (design doc ยง1.6): per target language, join a random sample of ll_title values against the target language's local meta DB (when installed) and report the match rate; a low rate signals a title-normalization mismatch

1000
SANITY_WARN_THRESHOLD = 
0.9
INSERT_PREFIX = 
/\A\s*INSERT\s+INTO\s+`langlinks`\s+VALUES\s+/i
UNESCAPES =

MySQL backslash escapes inside mysqldump string literals

{
  "0" => "\0", "'" => "'", '"' => '"', "b" => "\b", "n" => "\n",
  "r" => "\r", "t" => "\t", "Z" => "\x1A", "\\" => "\\"
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(db_path, cache_dir: nil) ⇒ LanglinksImporter

Returns a new instance of LanglinksImporter.



39
40
41
42
 
# File 'lib/wp2txt/langlinks_importer.rb', line 39

def initialize(db_path, cache_dir: nil)
  @db_path = db_path
  @cache_dir = cache_dir
end

Class Method Details

.dump_name_of(path) ⇒ Object

"jawiki-20260701-langlinks.sql.gz" => "jawiki-20260701" (same extraction rule as the dump_name recorded in the metadata DB)



46
47
48
 
# File 'lib/wp2txt/langlinks_importer.rb', line 46

def self.dump_name_of(path)
  File.basename(path)[/\A[a-z0-9_\-]+?-\d{8}/]
end

Instance Method Details

#import!(source_path, langs: nil, force: false, progress: nil) ⇒ Hash

Returns { status: :imported | :already_imported, ... }.

Parameters:

  • source_path (String) —

    langlinks .sql or .sql.gz file

  • langs (Array<String>, nil) (defaults to: nil) —

    target languages to import (nil = all)

  • force (Boolean) (defaults to: false) —

    drop and re-import an existing langlinks table

  • progress (Proc, nil) (defaults to: nil) —

    called with the running row count per batch

Returns:

  • (Hash) —

    { status: :imported | :already_imported, ... }



55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
 
# File 'lib/wp2txt/langlinks_importer.rb', line 55

def import!(source_path, langs: nil, force: false, progress: nil)
  raise ArgumentError, "langlinks file not found: #{source_path}" unless File.exist?(source_path)

  db = open_db
  dump_name = metadata_value(db, "dump_name")
  raise ArgumentError, "metadata index is not built: #{@db_path}" unless dump_name

  source_dump = self.class.dump_name_of(source_path)
  unless source_dump && source_dump == dump_name
    raise ArgumentError,
          "dump version mismatch: the metadata index is #{dump_name} but the langlinks file is " \
          "#{source_dump || File.basename(source_path)} (versions must match; there is no override)"
  end

  if !force && (existing = imported_at(db))
    return { status: :already_imported, imported_at: existing,
             row_count: db.get_first_value("SELECT COUNT(*) FROM langlinks").to_i }
  end

  lang_filter = langs && Set.new(langs)

  db.execute("DROP TABLE IF EXISTS langlinks")
  # Clear stale provenance immediately: if the load below fails midway,
  # the DB must be left as "not imported" (partial table only), so the
  # next non-force run re-imports instead of reporting a stale success
  db.execute("DELETE FROM metadata WHERE key LIKE 'langlinks\\_%' ESCAPE '\\'")
  db.execute(<<~SQL)
    CREATE TABLE langlinks (
      ll_from  INTEGER NOT NULL,
      ll_lang  TEXT    NOT NULL,
      ll_title TEXT    NOT NULL
    )
  SQL

  row_count = 0
  batch = []
  flush = lambda do
    db.transaction do
      stmt = db.prepare("INSERT INTO langlinks (ll_from, ll_lang, ll_title) VALUES (?, ?, ?)")
      batch.each { |row| stmt.execute(row) }
      stmt.close
    end
    row_count += batch.size
    progress&.call(row_count)
    batch.clear
  end

  skipped_invalid = each_source_row(source_path) do |ll_from, ll_lang, ll_title|
    next if lang_filter && !lang_filter.include?(ll_lang)

    batch << [ll_from, ll_lang, MetadataIndex.normalize_title(ll_title)]
    flush.call if batch.size >= BATCH_SIZE
  end
  flush.call unless batch.empty?

  # Indexes are created after the load, not before (insert speed)
  db.execute("CREATE INDEX idx_langlinks_from ON langlinks(ll_from, ll_lang)")
  db.execute("CREATE INDEX idx_langlinks_lang_title ON langlinks(ll_lang, ll_title)")

  stamp_provenance(db, source_path, langs, row_count, skipped_invalid)

  { status: :imported, row_count: row_count, skipped_invalid: skipped_invalid,
    provenance: read_provenance(db),
    sanity: sanity_check(db, dump_name) }
ensure
  db&.close
end