Module: LangExtract::Core::TokenSimilarity

Defined in:
lib/langextract/core/token_similarity.rb

Overview

Per-token similarity gate used by the token-level fuzzy aligner. Prevents near-word substitutions (humane→human, chart→cart) by requiring both a length-ratio floor and a character-similarity floor.

Constant Summary collapse

MIN_LENGTH_RATIO = 
0.85
MIN_CHAR_SIMILARITY = 
0.78

Class Method Summary collapse

Class Method Details

.char_similarity(left, right) ⇒ Object 



88
89
90
91
92
93
94
 
# File 'lib/langextract/core/token_similarity.rb', line 88

def char_similarity(left, right)
  return 1.0 if left == right
  return 0.0 if left.empty? || right.empty?

  matches = SequenceMatcher.match_count(left.each_char.to_a, right.each_char.to_a)
  (2.0 * matches) / (left.length + right.length)
end

.edit_distance_at_most_one?(left, right) ⇒ Boolean

Returns:

  • (Boolean) 


52
53
54
55
56
57
58
59
 
# File 'lib/langextract/core/token_similarity.rb', line 52

def edit_distance_at_most_one?(left, right)
  return true if left == right
  return false if (left.length - right.length).abs > 1

  return equal_length_one_mismatch?(left, right) if left.length == right.length

  one_insertion_or_deletion?(left, right)
end

.equal_length_one_mismatch?(left, right) ⇒ Boolean

Returns:

  • (Boolean) 


61
62
63
 
# File 'lib/langextract/core/token_similarity.rb', line 61

def equal_length_one_mismatch?(left, right)
  left.each_char.zip(right.each_char).count { |a, b| a != b } <= 1
end

.normalize(value) ⇒ Object 



33
34
35
36
 
# File 'lib/langextract/core/token_similarity.rb', line 33

def normalize(value)
  normalized = value.to_s.unicode_normalize(:nfc).downcase
  normalized.gsub(/[\u0027\u2019]/u, "").gsub(/\s+/, " ").strip
end

.one_insertion_or_deletion?(left, right) ⇒ Boolean

Returns:

  • (Boolean) 


65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/langextract/core/token_similarity.rb', line 65

def one_insertion_or_deletion?(left, right)
  shorter, longer = left.length < right.length ? [left, right] : [right, left]
  shorter_chars = shorter.each_char.to_a
  longer_chars = longer.each_char.to_a
  short_index = 0
  long_index = 0
  edits = 0

  # Keep this bounded to one edit.  The early exits make the routine
  # linear in the short token length and avoid allocating a matrix.
  while short_index < shorter.length && long_index < longer.length
    if shorter_chars[short_index] == longer_chars[long_index]
      short_index += 1
    else
      edits += 1
      return false if edits > 1
    end
    long_index += 1
  end

  edits + (longer.length - long_index) <= 1
end

.short_typo?(target_token, source_token) ⇒ Boolean

A one-character edit is intentionally limited to short words. Long near-words (for example human/humane) remain subject to the stricter ratio and character-similarity gates above.

Returns:

  • (Boolean) 


41
42
43
44
45
46
47
48
49
50
 
# File 'lib/langextract/core/token_similarity.rb', line 41

def short_typo?(target_token, source_token)
  target = normalize(target_token)
  source = normalize(source_token)
  return false if target.empty? || source.empty?
  return false if target == source
  return false if target.length < 3 || source.length < 3
  return false if [target.length, source.length].max > 5

  edit_distance_at_most_one?(target, source)
end

.similar?(target_token, source_token) ⇒ Boolean

Returns:

  • (Boolean) 


14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 
# File 'lib/langextract/core/token_similarity.rb', line 14

def similar?(target_token, source_token)
  target = normalize(target_token)
  source = normalize(source_token)
  return true if target == source
  return false if target.empty? || source.empty?

  # Short one-edit substitutions are useful when they are anchored by
  # another exact token in the same extraction.  The alignment policy enforces
  # that contextual anchor; keeping the broader lexical gate here lets
  # it consider those candidates without weakening single-token
  # grounding.
  return true if short_typo?(target, source)

  ratio = [target.length, source.length].min.to_f / [target.length, source.length].max
  return false if ratio < MIN_LENGTH_RATIO

  char_similarity(target, source) >= MIN_CHAR_SIMILARITY
end