Class: Kotoshu::Language::Tokenizer::JapaneseTokenizer

Inherits:
Base
  • Object
show all
Defined in:
lib/kotoshu/language/tokenizer/japanese_tokenizer.rb

Overview

Tokenizer for Japanese text.

Uses Suika gem for morphological analysis. Suika is a soft runtime dependency — see Suika for load status and SuikaUnavailable for the error raised when Japanese tokenization is requested without it.

See Also:

Direct Known Subclasses

Kotoshu::Languages::Japanese::Tokenizer

Constant Summary collapse

WORD_SEPARATORS =

Japanese word separators - keep it simple since Suika handles tokenization

/[\s"()\[\]{}<>,.;:!?\\\/|`~@#$%^&*·]/

Instance Method Summary collapse

Methods inherited from Base

#normalize, #skip_token?, #tokenize_with_positions, #word_boundary_regex, #word_char?

Instance Method Details

#tokenize(text) ⇒ Object 



18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
 
# File 'lib/kotoshu/language/tokenizer/japanese_tokenizer.rb', line 18

def tokenize(text)
  return [] if text.nil? || text.strip.empty?

  # Suika::Tagger is process-wide memoized in Language::Suika;
  # raises SuikaUnavailable when the gem is missing.
  tagger = Kotoshu::Language::Suika.tagger

  # Suika.parse returns an array of "surface\tfeatures" strings
  tokens = []
  parsed = tagger.parse(text)

  parsed.each do |token|
    # Suika returns: "すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ"
    # The surface form is tab-separated from the POS features
    surface = token.split("\t").first
    tokens << surface if surface && !surface.strip.empty?
  end

  tokens
end