Class: Kotoshu::Readers::ConvTable

Inherits:
Object
  • Object
show all
Defined in:
lib/kotoshu/readers/aff_data.rb

Overview

Conversion table for ICONV/OCONV.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(pairs) ⇒ ConvTable

Create a new conversion table.

Parameters:

  • pairs (Array<Array<String>>) —

    Array of [pattern, replacement] pairs



153
154
155
156
 
# File 'lib/kotoshu/readers/aff_data.rb', line 153

def initialize(pairs)
  @pairs = pairs
  @table = pairs.map { |pat1, pat2| compile_row(pat1, pat2) }.sort_by { |search, _| search.length }
end

Instance Attribute Details

#pairs ⇒ Array<Array<String>>

Array of [pattern, replacement] pairs

Returns:

  • (Array<Array<String>>) —

    the current value of pairs



147
148
149
 
# File 'lib/kotoshu/readers/aff_data.rb', line 147

def pairs
  @pairs
end

Instance Method Details

#call(word) ⇒ String

Apply conversions to word.

Note: Python's re.match(string, pos) anchors at pos, but Ruby's Regexp#match? searches from pos onward. We must check that the match actually begins at pos, otherwise short conversions fire on later positions in the word and produce nonsense like "ÉÉÉÉ" for "bébé".

Spylls uses Python's stable sorted(..., key=lambda r: len(r[0])) which preserves declaration order for ties. Ruby's sort_by is unstable, so we add the table index as a secondary key to mirror Spylls. Without this, Nepali's ICONV reorders the ZWNJ$ → ZWNJ no-op rule behind ZWNJ → U+FFF0, causing the trailing-ZWNJ word to be normalized to U+FFF0 (and then dropped by IGNORE) so it matches the dictionary.

Parameters:

  • word (String) —

    Input word

Returns:

  • (String) —

    Converted word



176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
 
# File 'lib/kotoshu/readers/aff_data.rb', line 176

def call(word)
  pos = 0
  result = ''

  while pos < word.length
    matches = @table.each_with_index.filter_map do |(search, pattern, _), idx|
      m = pattern.match(word, pos)
      next unless m && m.begin(0) == pos

      [search, idx]
    end.sort_by { |s, idx| [-s.length, idx] }

    if matches.any?
      search, idx = matches.first
      _, _, replacement = @table[idx]
      result += replacement
      pos += search.length
    else
      result += word[pos]
      pos += 1
    end
  end

  result
end