Class: Kotoshu::Readers::Word

Inherits:
Struct
  • Object
show all
Defined in:
lib/kotoshu/readers/dic_reader.rb

Overview

Word entry from the dictionary file.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Attribute Details

#flags ⇒ Set<String>

Morphological flags

Returns:

  • (Set<String>) —

    the current value of flags



10
11
12
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 10

def flags
  @flags
end

#morph_data ⇒ Array<String>

Morphological data fields (e.g. "ph:wich")

Returns:

  • (Array<String>) —

    the current value of morph_data



10
11
12
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 10

def morph_data
  @morph_data
end

#stem ⇒ String

The word stem

Returns:

  • (String) —

    the current value of stem



10
11
12
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 10

def stem
  @stem
end

Class Method Details

.from_line(line, context = {}) ⇒ Word

Create a word from a dictionary line.

Parameters:

  • line (String) —

    The dictionary line

  • context (Hash) (defaults to: {}) —

    The reading context (for flag parsing)

Returns:

  • (Word) —

    The parsed word



16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 16

def self.from_line(line, context = {})
  # Format: stem[/flags][<SEP><morph_data>]
  # SEP is a tab per spec, but real-world fixtures (including the
  # Hunspell ph.* tests) sometimes use spaces. We split on the first
  # morphological token (key:value) regardless of separator.
  head, morph = split_stem_and_morph(line)
  head = head.strip

  # Find the first UNESCAPED slash to split stem from flags.
  # Hunspell allows `\/` to represent a literal `/` inside a word —
  # so we can't just use String#index('/'). Mirrors Spylls's
  # SLASH_REGEXP (a slash not preceded by a backslash).
  #
  # Special case: a word that STARTS with `/` is not an empty stem +
  # flags — it's a word whose first character is `/`. Without this,
  # dic entry `/` would be parsed as stem="" + flags="".
  if head.start_with?('/')
    stem = head
    flags_str = nil
  else
    slash_idx = unescaped_slash_index(head)
    if slash_idx
      stem = head[0...slash_idx]
      flags_str = head[(slash_idx + 1)..]
    else
      stem = head
      flags_str = nil
    end

    # Replace escaped slashes in the stem: `\/` → `/`
    stem = stem.gsub('\/', '/') if stem.include?('\/')
  end

  flags = if flags_str && !flags_str.empty? && context[:flag_format]
            parse_flags(flags_str, context[:flag_format], context[:flag_synonyms])
          elsif flags_str && !flags_str.empty?
            flags_str.chars.to_set
          else
            Set.new
          end

  morph_data = parse_morph_data(morph)
  new(stem:, flags:, morph_data:)
end

.parse_flags(string, flag_format, flag_synonyms = {}) ⇒ Set<String>

Parse flags from string.

Parameters:

  • string (String) —

    Flag string

  • flag_format (String) —

    Flag format ('short', 'long', 'num', 'UTF-8')

  • flag_synonyms (Hash) (defaults to: {}) —

    Flag synonyms map

Returns:

  • (Set<String>) —

    Parsed flags



102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 102

def self.parse_flags(string, flag_format, flag_synonyms = {})
  return Set.new if string.nil? || string.empty?

  # AF (flag aliases) only applies when aliases are actually defined
  # and the string is a positional alias index (pure digits). Without
  # the !empty? guard, a `FLAG num` dictionary with no AF would have
  # every numeric flag collapsed to the empty set.
  if flag_synonyms.is_a?(Hash) && !flag_synonyms.empty? && string =~ /^\d+$/
    return flag_synonyms[string] || Set.new
  end

  case flag_format
  when 'short'
    string.chars.to_set
  when 'long'
    string.scan(/../).to_set
  when 'num'
    string.scan(/\d+/).to_set
  when 'UTF-8'
    string.chars.to_set
  else
    string.chars.to_set
  end
end

.parse_morph_data(morph) ⇒ Array<String>

Parse morphological data from the post-tab portion of a dic line.

Hunspell morphological data is whitespace-separated key:value tokens such as ph:wich (phonetic), st:foo (stem), po:noun (part of speech). We preserve them as a list of raw strings — downstream consumers (e.g. PhonetSuggest via alt_spellings) extract what they need.

Parameters:

  • morph (String, nil) —

    The morphological portion

Returns:

  • (Array<String>) —

    List of morphological tokens



90
91
92
93
94
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 90

def self.parse_morph_data(morph)
  return [] if morph.nil?

  morph.split(/\s+/).reject(&:empty?)
end

.split_stem_and_morph(line) ⇒ Array(String, String)

Split a dictionary line into stem and morphological-data portions.

Hunspell specifies tab-separated morph data, but the ph.* test fixtures use spaces. We honor both: if there's a tab, split there; otherwise split before the first key:value token (which is the universal signature of morphological data).

Parameters:

  • line (String) —

    The raw dictionary line

Returns:

  • (Array(String, String)) —

    [stem portion, morph portion]



70
71
72
73
74
75
76
77
78
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 70

def self.split_stem_and_morph(line)
  tab_idx = line.index("\t")
  return line.split("\t", 2) if tab_idx

  match = line.match(/(.*?)(\s+[a-zA-Z]+:[^\s].*)$/)
  return [line, ''] unless match

  [match[1], match[2]]
end

.unescaped_slash_index(str) ⇒ Integer?

Find the index of the first unescaped slash in the string.

Parameters:

  • str (String) —

    Input string

Returns:

  • (Integer, nil) —

    Index of first unescaped /, or nil



131
132
133
134
135
136
137
138
139
140
 
# File 'lib/kotoshu/readers/dic_reader.rb', line 131

def self.unescaped_slash_index(str)
  i = 0
  while i < str.length
    c = str[i]
    return i if c == '/' && (i.zero? || str[i - 1] != '\\')

    i += 1
  end
  nil
end