Module: TWFilter::Tables

Defined in:
lib/twfilter/tables.rb,
sig/twfilter.rbs

Overview

Reference tables. The bundled copy is the source of truth for behavior and is versioned with the gem; .dir replaces it wholesale and .overlay supplements it. Any deviation from the bundled copy changes what "Taiwan Mandarin" means, so both are reported by .provenance.

Constant Summary collapse

BUNDLED =

Returns:

  • (::Pathname) 
Pathname(__dir__).join("data").freeze
MANIFEST =

Returns:

  • (::String) 
"MANIFEST.json"
ADD =

Returns:

  • (::String) 
".add"
REMOVE =

Returns:

  • (::String) 
".remove"
SCHEMA =

Row shape per table; unknown names default to :terms so user tables are accepted.

Returns:

  • (::Hash[::String, ::Symbol]) 
{
  "moe_common.txt" => :chars,
  "moe_secondary.txt" => :chars,
  "moe_rare.txt" => :chars,
  "moe_exception.txt" => :chars,
  "simplified_only.txt" => :chars,
  "variants_used_in_taiwan.txt" => :chars,
  "converted_orthography.txt" => :chars,
  "cantonese.txt" => :chars,
  "wenyan.txt" => :chars,
  "mainland_hard.tsv" => :pairs,
  "mainland_soft.tsv" => :pairs,
  "mainland_exceptions.tsv" => :pairs
}.freeze

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.overlay ⇒ ::Pathname?

Returns the value of attribute overlay.

Returns:

  • (::Pathname, nil) 


35
36
37
 
# File 'lib/twfilter/tables.rb', line 35

def overlay
  @overlay
end

Class Method Details

.columns(name) ⇒ ::Array[::Array[::String]]

Parameters:

  • (::String)

Returns:

  • (::Array[::Array[::String]]) 


55
 
# File 'lib/twfilter/tables.rb', line 55

def columns(name) = cache[:"#{name}/columns"] ||= rows(name).map { |row| row.split("\t") }.freeze

.dir ⇒ ::Pathname

Returns:

  • (::Pathname) 


37
 
# File 'lib/twfilter/tables.rb', line 37

def dir = @dir ||= Pathname(ENV.fetch("TWFILTER_DATA_DIR", BUNDLED.to_s))

.dir=(path) ⇒ void

This method returns an undefined value.

Parameters:

  • (Object) 


39
40
41
42
 
# File 'lib/twfilter/tables.rb', line 39

def dir=(path)
  @dir = path.nil? ? nil : Pathname(path)
  TWFilter.reset!
end

.groups(name) ⇒ ::Hash[::String, ::Array[::String]]

Parameters:

  • (::String)

Returns:

  • (::Hash[::String, ::Array[::String]]) 


57
58
59
 
# File 'lib/twfilter/tables.rb', line 57

def groups(name) = cache[:"#{name}/groups"] ||= columns(name)
.to_h { |row| [row.first, row.drop(1).freeze] }
.freeze

.manifest ⇒ ::Hash[::String, untyped]

Returns:

  • (::Hash[::String, untyped]) 


69
70
71
72
 
# File 'lib/twfilter/tables.rb', line 69

def manifest
  path = dir.join(MANIFEST)
  path.exist? ? JSON.parse(path.read).freeze : {}
end

.names ⇒ ::Array[::String]

Returns:

  • (::Array[::String]) 


61
62
63
64
65
66
67
 
# File 'lib/twfilter/tables.rb', line 61

def names
  base = dir.children.map { |path| path.basename.to_s }
  extra = overlay ? overlay.children.map { |path|
    path.basename.to_s.sub(/#{Regexp.escape(ADD)}|#{Regexp.escape(REMOVE)}(?=\.)/, "")
  } : []
  (base + extra).uniq.reject { |name| name == MANIFEST }.sort
end

.pairs(name) ⇒ ::Hash[::String, ::String]

Parameters:

  • (::String)

Returns:

  • (::Hash[::String, ::String]) 


53
 
# File 'lib/twfilter/tables.rb', line 53

def pairs(name) = cache[:"#{name}/pairs"] ||= rows(name).to_h { |row| row.split("\t", 2) }.freeze

.provenance ⇒ ::Hash[::Symbol, untyped]

Fingerprint of the effective tables, for recording alongside derived measurements.

Returns:

  • (::Hash[::Symbol, untyped]) 


75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/twfilter/tables.rb', line 75

def provenance
  digest = Digest::SHA256.new
  names.each { |name| digest << name << rows(name).join("\n") }

  {
    version: VERSION,
    dir: dir.to_s,
    overlay: overlay&.to_s,
    tables: names.length,
    # 64 bits, sufficient to detect an accidental table change
    digest: digest.hexdigest[0, 16]
  }.freeze
end

.reset! ⇒ void

This method returns an undefined value.



89
90
91
92
 
# File 'lib/twfilter/tables.rb', line 89

def reset!
  @cache = nil
  ENV["TWFILTER_OVERLAY_DIR"]&.then { |path| @overlay ||= Pathname(path) }
end

.rows(name) ⇒ ::Array[::String]

Parameters:

  • (::String)

Returns:

  • (::Array[::String]) 


49
 
# File 'lib/twfilter/tables.rb', line 49

def rows(name) = cache[name] ||= resolve(name)

.set(name) ⇒ ::Set[::String]

Parameters:

  • (::String)

Returns:

  • (::Set[::String]) 


51
 
# File 'lib/twfilter/tables.rb', line 51

def set(name) = cache[:"#{name}/set"] ||= rows(name).to_set