Class: LangExtract::Core::FuzzyTokenStream

Inherits:
Object
  • Object
show all
Defined in:
lib/langextract/core/fuzzy_token_stream.rb

Overview

Builds the temporary source-token stream used only by fuzzy alignment. Canonical tokenizer output remains unchanged for exact and token offsets.

Constant Summary collapse

DASH_PUNCTUATION = 
/\p{Pd}/u
NUMERIC_SEPARATOR =

Commas are the grouping variant covered by the fuzzy contract. Keep periods intact so sentence-boundary barriers remain visible.

/,/u
APOSTROPHE = 
/[\u0027\u2019]/u
SEPARATOR_PUNCTUATION = 
/\A(?:\p{Pd}|,)\z/u

Instance Method Summary collapse

Constructor Details

#initialize(tokens) ⇒ FuzzyTokenStream

Returns a new instance of FuzzyTokenStream.



18
19
20
 
# File 'lib/langextract/core/fuzzy_token_stream.rb', line 18

def initialize(tokens)
  @tokens = tokens
end

Instance Method Details

#tokens_in(range) ⇒ Object 



22
23
24
25
26
27
28
 
# File 'lib/langextract/core/fuzzy_token_stream.rb', line 22

def tokens_in(range)
  in_range = tokens.filter do |token|
    token.char_interval.start_pos >= range.begin && token.char_interval.end_pos <= range.end
  end
  pieces = in_range.flat_map { |token| split_boundary_punctuation(token) }
  merge_apostrophe_tokens(pieces).reject { |token| separator_punctuation?(token.text) }
end