Class: Kotoshu::Languages::Japanese::Tokenizer

Inherits:

Kotoshu::Language::Tokenizer::JapaneseTokenizer

• Object
• Kotoshu::Language::Tokenizer::Base
• Kotoshu::Language::Tokenizer::JapaneseTokenizer
• Kotoshu::Languages::Japanese::Tokenizer

Defined in:

lib/kotoshu/languages/ja/language.rb

Overview

Japanese tokenizer with morphological analysis.

Constant Summary

Constants inherited from Kotoshu::Language::Tokenizer::JapaneseTokenizer

Kotoshu::Language::Tokenizer::JapaneseTokenizer::WORD_SEPARATORS

Method Summary

Methods inherited from Kotoshu::Language::Tokenizer::JapaneseTokenizer

#tokenize

Methods inherited from Kotoshu::Language::Tokenizer::Base

#normalize, #skip_token?, #tokenize, #tokenize_with_positions, #word_boundary_regex, #word_char?