Class: Kotoshu::Languages::French::Tokenizer

Inherits:

Kotoshu::Language::Tokenizer::FrenchTokenizer

• Object
• Kotoshu::Language::Tokenizer::Base
• Kotoshu::Language::Tokenizer::FrenchTokenizer
• Kotoshu::Languages::French::Tokenizer

Defined in:

lib/kotoshu/languages/fr/language.rb

Overview

French tokenizer with contraction handling.

Constant Summary

Constants inherited from Kotoshu::Language::Tokenizer::FrenchTokenizer

Kotoshu::Language::Tokenizer::FrenchTokenizer::CONTRACTION_PATTERNS, Kotoshu::Language::Tokenizer::FrenchTokenizer::DO_NOT_SPLIT, Kotoshu::Language::Tokenizer::FrenchTokenizer::WORD_SEPARATORS

Method Summary

Methods inherited from Kotoshu::Language::Tokenizer::FrenchTokenizer

#tokenize

Methods inherited from Kotoshu::Language::Tokenizer::Base

#normalize, #skip_token?, #tokenize, #tokenize_with_positions, #word_boundary_regex, #word_char?