Module: Kotoshu::Readers::PhRepExtractor

Defined in:
lib/kotoshu/readers/lookup_builder.rb

Overview

Extracts REP entries from dictionary ph: morphological data.

Hunspell 1.7+ treats ph: as a phonetic misspelling hint and reuses the existing replchars permutation to surface corrections. To make that work, ph: payloads are converted to REP entries at dic-read time. Three payload forms are recognised (mirroring Spylls/Hunspell):

  • simple — ph:wich against stem "which" yields REP(wich, which) and registers wich as an alt_spelling for ngram scoring.
  • star — ph:prity* against stem "pretty" yields REP(prit, prett): the trailing char of the pattern (and the *) is stripped, and the trailing char of the stem is stripped, so the REP entry can attach suffixes on either side (matches "prity" → "pretty" and "pritiest" → "prettiest").
  • arrow — ph:hepi->happi is an explicit pair: REP(hepi, happi). The stem is irrelevant for arrow form.

The class is a pure transformer; side effects on the aff hash are performed by the caller via PhRepExtractor.append_to_aff.

Class Method Summary collapse

Class Method Details

.append_to_aff(aff, stem, ph_tokens) ⇒ void

This method returns an undefined value.

Convert each ph: payload into a RepPattern and append it to aff[:REP]. Mutates the aff hash.

Parameters:

  • aff (Hash) —

    Aff structure (mutated)

  • stem (String) —

    Dictionary stem

  • ph_tokens (Array<String>) —

    Raw ph: payloads



522
523
524
525
526
527
528
529
530
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 522

def append_to_aff(aff, stem, ph_tokens)
  return if ph_tokens.nil? || ph_tokens.empty?

  aff[:REP] ||= []
  ph_tokens.each do |token|
    rep = build_rep(stem, token)
    aff[:REP] << rep_hash(rep) if rep
  end
end

.arrow_rep(token) ⇒ RepPattern?

ph:hepi->happi → REP(hepi, happi).

Parameters:

  • token (String) —

    Arrow-form payload

Returns:

  • (RepPattern, nil) —

    nil if either side is empty



579
580
581
582
583
584
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 579

def arrow_rep(token)
  from, to = token.split('->', 2)
  return nil if from.nil? || to.nil? || from.empty? || to.empty?

  RepPattern.new(from, to)
end

.build_rep(stem, token) ⇒ RepPattern?

Build a RepPattern from a single ph: payload.

Parameters:

  • stem (String) —

    Dictionary stem

  • token (String) —

    Raw ph: payload

Returns:



537
538
539
540
541
542
543
544
545
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 537

def build_rep(stem, token)
  if token.end_with?('*')
    star_rep(stem, token)
  elsif token.include?('->')
    arrow_rep(token)
  else
    simple_rep(stem, token)
  end
end

.rep_hash(rep) ⇒ Hash{Symbol=>Object}

Convert a RepPattern to the Hash shape used by Permutations.replchars.

Parameters:

Returns:

  • (Hash{Symbol=>Object}) 


551
552
553
554
555
556
557
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 551

def rep_hash(rep)
  {
    regexp: rep.matcher,
    pattern: rep.pattern,
    replacement: rep.replacement
  }
end

.simple_alt_spellings(_stem, ph_tokens) ⇒ Array<String>

Return only the simple payloads — the ones that double as alt_spellings for ngram scoring. Star and arrow forms are excluded because their REP entries don't map 1:1 to the stem and would mislead ngram.

Parameters:

  • stem (String) —

    Dictionary stem (unused, present for symmetry)

  • ph_tokens (Array<String>) —

    Raw ph: payloads

Returns:

  • (Array<String>) —

    Simple alt spellings



511
512
513
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 511

def simple_alt_spellings(_stem, ph_tokens)
  ph_tokens.reject { |token| token.end_with?('*') || token.include?('->') }
end

.simple_rep(stem, token) ⇒ RepPattern?

ph:wich (against stem "which") → REP(wich, which).

Parameters:

  • stem (String) —

    Dictionary stem

  • token (String) —

    Simple payload

Returns:



591
592
593
594
595
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 591

def simple_rep(stem, token)
  return nil if token.empty?

  RepPattern.new(token, stem)
end

.star_rep(stem, token) ⇒ RepPattern?

ph:prity* (against stem "pretty") → REP(prit, prett).

Strip the last two chars of the pattern (y*) and the last char of the stem (y), so affixed forms on either side can match.

Parameters:

  • stem (String) —

    Dictionary stem

  • token (String) —

    Star-form payload (e.g. prity*)

Returns:

  • (RepPattern, nil) —

    nil if pattern or stem too short



567
568
569
570
571
572
573
 
# File 'lib/kotoshu/readers/lookup_builder.rb', line 567

def star_rep(stem, token)
  pattern = token[0...-2]
  replacement = stem[0...-1]
  return nil if pattern.empty? || replacement.empty?

  RepPattern.new(pattern, replacement)
end