Class: Ast::Merge::TokenMatchRefiner

Inherits:
MatchRefinerBase show all
Includes:
JaccardSimilarity
Defined in:
lib/ast/merge/token_match_refiner.rb

Overview

Match refiner using Jaccard token-overlap similarity.

Pairs unmatched nodes by extracting significant tokens from their text content and computing the Jaccard similarity index. Useful for matching nodes with minor wording differences (e.g., "Commit changes" vs "Commit your changes").

Uses greedy_match from MatchRefinerBase for optimal 1:1 pairing.

Examples:

Basic usage

refiner = TokenMatchRefiner.new(threshold: 0.35)
matches = refiner.call(template_nodes, dest_nodes)

With specific node types

refiner = TokenMatchRefiner.new(
  threshold: 0.4,
  node_types: [:list_item, :paragraph],
)

With custom text extraction

refiner = TokenMatchRefiner.new(
  text_extractor: ->(node) { node.inner_text.strip },
)

See Also:

Constant Summary collapse

DEFAULT_TOKEN_THRESHOLD =

Default threshold for Jaccard token overlap

0.35

Constants included from JaccardSimilarity

JaccardSimilarity::DEFAULT_MIN_TOKEN_LENGTH, JaccardSimilarity::DEFAULT_STOPWORDS

Constants inherited from MatchRefinerBase

MatchRefinerBase::DEFAULT_THRESHOLD

Instance Attribute Summary collapse

Attributes inherited from MatchRefinerBase

#node_types, #threshold

Instance Method Summary collapse

Methods included from JaccardSimilarity

extract_tokens, jaccard

Methods inherited from MatchRefinerBase

#handles_type?

Constructor Details

#initialize(threshold: DEFAULT_TOKEN_THRESHOLD, node_types: [], text_extractor: nil, stopwords: JaccardSimilarity::DEFAULT_STOPWORDS, **_options) ⇒ TokenMatchRefiner

Initialize a token match refiner.

Parameters:

  • threshold (Float) (defaults to: DEFAULT_TOKEN_THRESHOLD) —

    Minimum Jaccard score to accept a match (default: 0.35)

  • node_types (Array<Symbol>) (defaults to: []) —

    Node types to process (empty = all)

  • text_extractor (Proc, nil) (defaults to: nil) —

    Custom function to extract text from a node. Should accept a node and return a String. Default uses node.text.to_s.

  • stopwords (Set<String>) (defaults to: JaccardSimilarity::DEFAULT_STOPWORDS) —

    Words to exclude (default: JaccardSimilarity::DEFAULT_STOPWORDS)

  • options (Hash) —

    Additional options for forward compatibility



51
52
53
54
55
56
57
58
59
60
61
 
# File 'lib/ast/merge/token_match_refiner.rb', line 51

def initialize(
  threshold: DEFAULT_TOKEN_THRESHOLD,
  node_types: [],
  text_extractor: nil,
  stopwords: JaccardSimilarity::DEFAULT_STOPWORDS,
  **_options
)
  super(threshold: threshold, node_types: node_types)
  @text_extractor = text_extractor
  @stopwords = stopwords
end

Instance Attribute Details

#stopwords ⇒ Set<String> (readonly)

Returns Stopwords to exclude from token extraction.

Returns:

  • (Set<String>) —

    Stopwords to exclude from token extraction



41
42
43
 
# File 'lib/ast/merge/token_match_refiner.rb', line 41

def stopwords
  @stopwords
end

#text_extractor ⇒ Proc? (readonly)

Returns Custom function to extract text from a node.

Returns:

  • (Proc, nil) —

    Custom function to extract text from a node



38
39
40
 
# File 'lib/ast/merge/token_match_refiner.rb', line 38

def text_extractor
  @text_extractor
end

Instance Method Details

#call(template_nodes, dest_nodes, _context = {}) ⇒ Array<MatchResult>

Match unmatched nodes by Jaccard token similarity.

Parameters:

  • template_nodes (Array) —

    Unmatched template nodes

  • dest_nodes (Array) —

    Unmatched destination nodes

  • context (Hash) —

    Additional context (unused)

Returns:



69
70
71
72
73
74
75
76
77
78
 
# File 'lib/ast/merge/token_match_refiner.rb', line 69

def call(template_nodes, dest_nodes, _context = {})
  t_filtered = node_types.empty? ? template_nodes : filter_nodes(template_nodes)
  d_filtered = node_types.empty? ? dest_nodes : filter_nodes(dest_nodes)

  greedy_match(t_filtered, d_filtered) do |t_node, d_node|
    t_tokens = node_tokens(t_node)
    d_tokens = node_tokens(d_node)
    jaccard(t_tokens, d_tokens)
  end
end