Class: Kotoshu::Language::Segmenter

Inherits:
Object
  • Object
show all
Defined in:
lib/kotoshu/language/segmenter.rb

Overview

Splits a Documents::Document into language-tagged segments.

Walks document.text_nodes, detects the dominant language for each node via the configured detector (default: Detector, the pure-Ruby character-set-based detector — no fasttext needed), and groups consecutive same-language nodes into Segments.

The detector is pluggable via the detector: constructor argument. Any object that responds to detect_with_confidence and returns [code, confidence] will do — the FastText-backed LanguageIdentifier is a drop-in for higher accuracy at the cost of a model download.

Constant Summary collapse

DEFAULT_DETECTOR = 
Detector
MIN_NODE_LENGTH_FOR_DETECTION =

Minimum text length (chars) for a node to participate in detection. Shorter nodes ("a", "the", whitespace) don't carry enough signal — they inherit the surrounding language.

3

Instance Method Summary collapse

Constructor Details

#initialize(detector: nil, min_confidence: 0.3, fallback_language: "en") ⇒ Segmenter

Returns a new instance of Segmenter.

Parameters:

  • detector (Object, nil) (defaults to: nil) —

    responds to detect_with_confidence. Defaults to DEFAULT_DETECTOR.

  • min_confidence (Float) (defaults to: 0.3) —

    detection results below this threshold are treated as unknown (segment language becomes the fallback language, default "en").

  • fallback_language (String) (defaults to: "en") —

    language code to use when detection is inconclusive.



63
64
65
66
67
 
# File 'lib/kotoshu/language/segmenter.rb', line 63

def initialize(detector: nil, min_confidence: 0.3, fallback_language: "en")
  @detector = detector || DEFAULT_DETECTOR
  @min_confidence = min_confidence
  @fallback_language = fallback_language
end

Instance Method Details

#segment(document) ⇒ Array<Segment>

Segment a document.

Parameters:

Returns:

  • (Array<Segment>) —

    language-tagged segments, in source order



73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
 
# File 'lib/kotoshu/language/segmenter.rb', line 73

def segment(document)
  unless document.is_a?(Kotoshu::Documents::Document)
    raise ArgumentError,
          "document must be a Kotoshu::Documents::Document"
  end

  segments = []
  current = nil

  document.text_nodes.each do |node|
    code, confidence = detect_for_node(node)
    if current && current.language_code == code
      current.text_nodes << node
    else
      segments << current if current && !current.empty?
      current = Segment.new(language_code: code, text_nodes: [node],
                            confidence: confidence)
    end
  end
  segments << current if current && !current.empty?
  segments
end