Class: Kotoshu::Languages::Arabic::Tokenizer

Inherits:
Kotoshu::Language::Tokenizer::Base show all
Defined in:
lib/kotoshu/languages/ar/language.rb

Overview

Minimal Arabic tokenizer.

Splits on whitespace and standard punctuation — Arabic text is space-delimited. Uses Base#skip_token? which correctly handles Unicode letters via \pL, so Arabic characters are kept while pure numbers and single non-letter chars are filtered.

Constant Summary collapse

ARABIC_SEPARATORS = 
/[\s"()\[\]{}<>,.;:!?\\\/|`~@#$%^&*+\-·،؛؟]/

Instance Method Summary collapse

Methods inherited from Kotoshu::Language::Tokenizer::Base

#skip_token?, #tokenize_with_positions, #word_boundary_regex, #word_char?

Instance Method Details

#tokenize(text) ⇒ Object 



64
65
66
67
68
69
70
 
# File 'lib/kotoshu/languages/ar/language.rb', line 64

def tokenize(text)
  return [] if text.nil? || text.strip.empty?

  text.split(ARABIC_SEPARATORS)
    .map { |token| normalize(token) }
    .reject { |token| skip_token?(token) }
end