Module: Kotoshu::Language

Defined in:
lib/kotoshu/language.rb,
lib/kotoshu/language/suika.rb,
lib/kotoshu/language/detector.rb,
lib/kotoshu/language/registry.rb,
lib/kotoshu/language/segmenter.rb,
lib/kotoshu/language/identifier.rb,
lib/kotoshu/language/languages/base.rb,
lib/kotoshu/language/tokenizer/base.rb,
lib/kotoshu/language/normalizer/base.rb,
lib/kotoshu/language/normalizer/arabic.rb,
lib/kotoshu/language/normalizer/hebrew.rb,
lib/kotoshu/language/normalizer/persian.rb,
lib/kotoshu/language/tokenizer/latin_tokenizer.rb,
lib/kotoshu/language/tokenizer/french_tokenizer.rb,
lib/kotoshu/language/tokenizer/german_tokenizer.rb,
lib/kotoshu/language/tokenizer/russian_tokenizer.rb,
lib/kotoshu/language/tokenizer/spanish_tokenizer.rb,
lib/kotoshu/language/tokenizer/japanese_tokenizer.rb,
lib/kotoshu/language/tokenizer/portuguese_tokenizer.rb

Overview

Language module for multi-language support.

Provides language detection, tokenization, and normalization for different languages with proper OOP design.

Examples:

Detect language

Kotoshu::Language.detect("Hello world")  # => "en"

Get language class

lang_class = Kotoshu::Language.get("en-US")

List supported languages

Kotoshu::Language.supported_codes  # => ["de-DE", "en-US", ...]

Defined Under Namespace

Modules: Normalizer, Suika, Tokenizer Classes: Base, Detector, LanguageIdentifier, Registry, Segment, Segmenter

Class Method Summary collapse

Class Method Details

.detect(text) ⇒ String?

Detect language from text.

Delegates to Detector.

Parameters:

  • text (String) —

    Text to analyze

Returns:

  • (String, nil) —

    Detected language code



54
55
56
 
# File 'lib/kotoshu/language.rb', line 54

def detect(text)
  Detector.detect(text)
end

.detect_with_confidence(text) ⇒ Array<String, Float>

Detect with confidence score.

Parameters:

  • text (String) —

    Text to analyze

Returns:

  • (Array<String, Float>) —

    Language code and confidence



62
63
64
 
# File 'lib/kotoshu/language.rb', line 62

def detect_with_confidence(text)
  Detector.detect_with_confidence(text)
end

.get(code) ⇒ Class?

Get language class by code.

Delegates to Registry.

Parameters:

  • code (String) —

    Language code

Returns:

  • (Class, nil) —

    Language class or nil



72
73
74
 
# File 'lib/kotoshu/language.rb', line 72

def get(code)
  Registry.get(code)
end

.info(code) ⇒ Hash?

Get language info.

Parameters:

  • code (String) —

    Language code

Returns:

  • (Hash, nil) —

    Language info or nil



95
96
97
 
# File 'lib/kotoshu/language.rb', line 95

def info(code)
  Registry.info(code)
end

.register(code, klass) ⇒ void

This method returns an undefined value.

Register a language.

Parameters:

  • code (String) —

    Language code

  • klass (Class) —

    Language class



104
105
106
 
# File 'lib/kotoshu/language.rb', line 104

def register(code, klass)
  Registry.register(code, klass)
end

.registered?(code) ⇒ Boolean

Check if language is registered.

Parameters:

  • code (String) —

    Language code

Returns:

  • (Boolean) —

    True if registered



80
81
82
 
# File 'lib/kotoshu/language.rb', line 80

def registered?(code)
  Registry.registered?(code)
end

.supported_codes ⇒ Array<String>

Get all supported language codes.

Returns:

  • (Array<String>) —

    List of codes



87
88
89
 
# File 'lib/kotoshu/language.rb', line 87

def supported_codes
  Registry.supported_codes
end