Class: Kotoshu::Dictionary::PlainText

Inherits:
Base
  • Object
show all
Defined in:
lib/kotoshu/dictionary/plain_text.rb

Overview

Plain text dictionary backend.

This dictionary reads from simple plain text word lists, with support for comments and various formatting options.

File format:

  • One word per line
  • Lines starting with # are comments
  • Empty lines are ignored
  • Supports multi-word phrases (e.g., "New York")

Examples:

Creating from a file

dict = PlainText.new("words.txt", language_code: "en-US")
dict.lookup?("hello")  # => true

Creating from a URL

dict = PlainText.new("https://raw.githubusercontent.com/kotoshu/dictionaries/main/en_US/words.txt",
                     language_code: "en-US")

Creating from an array

dict = PlainText.from_words(%w[hello world test], language_code: "en")

Instance Attribute Summary collapse

Attributes inherited from Base

#language_code, #locale, #metadata

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Base

#<<, #all_words, #contains?, #each_word, #empty?, #has_word?, #include?, load, #lookup?, register_type, registry, #size, #to_s, #type, #words_matching, #words_with_prefix

Constructor Details

#initialize(path = nil, language_code:, locale: nil, case_sensitive: false, word_pattern: nil, metadata: {}, words: nil) ⇒ PlainText

Create a new PlainText dictionary.

Parameters:

  • path (String, nil) (defaults to: nil) —

    Path to the dictionary file or URL. Nil when constructing from in-memory words:.

  • language_code (String) —

    The language code

  • locale (String, nil) (defaults to: nil) —

    The locale (optional)

  • case_sensitive (Boolean) (defaults to: false) —

    Whether lookups are case-sensitive

  • word_pattern (Regexp, nil) (defaults to: nil) —

    Pattern to filter words (optional)

  • metadata (Hash) (defaults to: {}) —

    Additional metadata (optional)

  • words (Array<String>, nil) (defaults to: nil) —

    Initial word list (bypasses file loading; used by .from_words).



49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 49

def initialize(path = nil, language_code:, locale: nil, case_sensitive: false,
               word_pattern: nil, metadata: {}, words: nil)
  super(language_code, locale: locale, metadata: metadata)

  @original_path = path
  @path = path ? resolve_path(path) : nil
  @case_sensitive = case_sensitive
  @word_pattern = word_pattern
  @words = words ? normalize_words(words) : load_words(@path)
  @word_set = build_word_set

  # Register this dictionary type
  self.class.register_type(:plain_text) unless Dictionary.registry.key?(:plain_text)
end

Instance Attribute Details

#case_sensitive ⇒ Boolean (readonly)

Returns Whether lookups are case-sensitive.

Returns:

  • (Boolean) —

    Whether lookups are case-sensitive



33
34
35
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 33

def case_sensitive
  @case_sensitive
end

#path ⇒ String (readonly)

Returns The path to the dictionary file (or nil if created from array).

Returns:

  • (String) —

    The path to the dictionary file (or nil if created from array)



30
31
32
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 30

def path
  @path
end

#word_pattern ⇒ Regexp? (readonly)

Returns Pattern for word filtering.

Returns:

  • (Regexp, nil) —

    Pattern for word filtering



36
37
38
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 36

def word_pattern
  @word_pattern
end

Class Method Details

.from_string(text, language_code:, locale: nil, case_sensitive: false) ⇒ PlainText

Create a dictionary from a string.

Examples:

text = "hello\nworld\ntest"
dict = PlainText.from_string(text, language_code: "en")

Parameters:

  • text (String) —

    The text containing words (newline separated)

  • language_code (String) —

    The language code

  • locale (String, nil) (defaults to: nil) —

    The locale (optional)

  • case_sensitive (Boolean) (defaults to: false) —

    Whether lookups are case-sensitive

Returns:



168
169
170
171
172
173
174
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 168

def self.from_string(text, language_code:, locale: nil, case_sensitive: false)
  words = text.split("\n").reject { |l| l.empty? || l.strip.start_with?("#") }
    .map(&:strip)

  from_words(words, language_code: language_code, locale: locale,
                    case_sensitive: case_sensitive)
end

.from_words(words, language_code:, locale: nil, case_sensitive: false) ⇒ PlainText

Create a dictionary from an array of words.

Examples:

dict = PlainText.from_words(%w[hello world test], language_code: "en")

Parameters:

  • words (Array<String>) —

    The words

  • language_code (String) —

    The language code

  • locale (String, nil) (defaults to: nil) —

    The locale (optional)

  • case_sensitive (Boolean) (defaults to: false) —

    Whether lookups are case-sensitive

Returns:



152
153
154
155
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 152

def self.from_words(words, language_code:, locale: nil, case_sensitive: false)
  new(words: words, language_code: language_code,
      locale: locale, case_sensitive: case_sensitive)
end

Instance Method Details

#add_word(word, flags: []) ⇒ Boolean

Add a word to the dictionary.

Parameters:

  • word (String) —

    The word to add

  • flags (Array<String>) (defaults to: []) —

    Flags (ignored for PlainText)

Returns:

  • (Boolean) —

    True if added



107
108
109
110
111
112
113
114
115
116
117
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 107

def add_word(word, flags: [])
  return false if word.nil? || word.empty?

  lookup_word = @case_sensitive ? word : word.downcase
  return false if @word_set.key?(lookup_word)

  @words << lookup_word
  @word_set[lookup_word] = @words.length - 1

  true
end

#lookup(word) ⇒ Boolean

Check if a word exists in the dictionary.

Parameters:

  • word (String) —

    The word to look up

Returns:

  • (Boolean) —

    True if the word exists



68
69
70
71
72
73
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 68

def lookup(word)
  return false if word.nil? || word.empty?

  lookup_word = @case_sensitive ? word : word.downcase
  @word_set.key?(lookup_word)
end

#remove_word(word) ⇒ Boolean

Remove a word from the dictionary.

Parameters:

  • word (String) —

    The word to remove

Returns:

  • (Boolean) —

    True if removed



123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 123

def remove_word(word)
  return false if word.nil? || word.empty?

  lookup_word = @case_sensitive ? word : word.downcase
  return false unless @word_set.key?(lookup_word)

  index = @word_set.delete(lookup_word)
  @words.delete_at(index)

  true
end

#suggest(word, max_suggestions: 10) ⇒ Array<String>

Generate spelling suggestions.

Uses edit distance to find similar words in the dictionary.

Parameters:

  • word (String) —

    The misspelled word

  • max_suggestions (Integer) (defaults to: 10) —

    Maximum suggestions

Returns:

  • (Array<String>) —

    List of suggested words



82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 82

def suggest(word, max_suggestions: 10)
  return [] if word.nil? || word.empty?

  lookup_word = @case_sensitive ? word : word.downcase

  # Find words with same prefix
  prefix_len = [lookup_word.length - 1, 3].max
  prefix = lookup_word[0...prefix_len]
  candidates = @words.select { |w| w.start_with?(prefix) }

  # Calculate edit distances
  candidates.map do |dict_word|
    dist = edit_distance(lookup_word, dict_word)
    [dict_word, dist]
  end.select { |_, dist| dist.positive? && dist <= 2 }
    .sort_by { |_, dist| dist }
    .first(max_suggestions)
    .map(&:first)
end

#words ⇒ Array<String>

Get all words in the dictionary.

Returns:

  • (Array<String>) —

    All words



138
139
140
 
# File 'lib/kotoshu/dictionary/plain_text.rb', line 138

def words
  @words.dup
end