Class: Kotoshu::Languages::Hebrew::Tokenizer

Inherits:
Kotoshu::Language::Tokenizer::Base show all
Defined in:
lib/kotoshu/languages/he/language.rb

Overview

Minimal Hebrew tokenizer.

Splits on whitespace and standard punctuation — Hebrew text is space-delimited. Includes the Hebrew-specific geresh (׳) and gershayim (״) as word characters (they are used inside acronyms). Uses Base#skip_token? which correctly handles Unicode letters via \pL.

Constant Summary collapse

HEBREW_SEPARATORS = 
/[\s"()\[\]{}<>,.;:!?\\\/|`~@#$%^&*+\-·]/

Instance Method Summary collapse

Methods inherited from Kotoshu::Language::Tokenizer::Base

#skip_token?, #tokenize_with_positions, #word_boundary_regex, #word_char?

Instance Method Details

#tokenize(text) ⇒ Object 



63
64
65
66
67
68
69
 
# File 'lib/kotoshu/languages/he/language.rb', line 63

def tokenize(text)
  return [] if text.nil? || text.strip.empty?

  text.split(HEBREW_SEPARATORS)
    .map { |token| normalize(token) }
    .reject { |token| skip_token?(token) }
end