Class: Kotoshu::Languages::German::Tokenizer

Inherits:

Kotoshu::Language::Tokenizer::GermanTokenizer

• Object
• Kotoshu::Language::Tokenizer::Base
• Kotoshu::Language::Tokenizer::GermanTokenizer
• Kotoshu::Languages::German::Tokenizer

Defined in:

lib/kotoshu/languages/de/language.rb

Overview

German tokenizer with special character handling.

Constant Summary

Constants inherited from Kotoshu::Language::Tokenizer::GermanTokenizer

Kotoshu::Language::Tokenizer::GermanTokenizer::WORD_SEPARATORS

Method Summary

Methods inherited from Kotoshu::Language::Tokenizer::GermanTokenizer

#tokenize

Methods inherited from Kotoshu::Language::Tokenizer::Base

#normalize, #skip_token?, #tokenize, #tokenize_with_positions, #word_boundary_regex, #word_char?