Module: Ast::Merge::JaccardSimilarity

Included in:
TokenMatchRefiner
Defined in:
lib/ast/merge/jaccard_similarity.rb

Overview

Jaccard set-similarity utilities for fuzzy matching of text nodes.

Provides token extraction and Jaccard index computation, reusable across format-specific mergers (Markdown lists, TOML blocks, etc.).

Examples:

Computing similarity between two strings

include Ast::Merge::JaccardSimilarity

a = extract_tokens("Commit changes to the branch")
b = extract_tokens("Commit your changes")
jaccard(a, b)  # => 0.667

With custom stopwords

tokens = extract_tokens("the quick brown fox", stopwords: %w[the].to_set)
# => #<Set: {"quick", "brown", "fox"}>

Constant Summary collapse

DEFAULT_STOPWORDS =

Common English stopwords excluded from token matching.

%w[
  a
  an
  and
  are
  as
  at
  be
  but
  by
  for
  from
  has
  have
  in
  is
  it
  its
  of
  on
  or
  so
  that
  the
  their
  then
  there
  they
  this
  to
  up
  was
  will
  with
].to_set.freeze
DEFAULT_MIN_TOKEN_LENGTH =

Minimum word length for a token to be considered significant.

3

Class Method Summary collapse

Class Method Details

.extract_tokens(text, stopwords: DEFAULT_STOPWORDS, min_length: DEFAULT_MIN_TOKEN_LENGTH) ⇒ Set<String>

Extract significant tokens from text for set-based comparison.

Extracts words of at least min_length characters, lowercased, with stopwords removed.

Parameters:

  • text (String) —

    The text to tokenize

  • stopwords (Set<String>) (defaults to: DEFAULT_STOPWORDS) —

    Words to exclude (default: DEFAULT_STOPWORDS)

  • min_length (Integer) (defaults to: DEFAULT_MIN_TOKEN_LENGTH) —

    Minimum token length (default: 3)

Returns:

  • (Set<String>) —

    Set of significant lowercase tokens



70
71
72
73
74
75
76
 
# File 'lib/ast/merge/jaccard_similarity.rb', line 70

def extract_tokens(text, stopwords: DEFAULT_STOPWORDS, min_length: DEFAULT_MIN_TOKEN_LENGTH)
  text.to_s
      .downcase
      .scan(/[[:alpha:]][[:alnum:]_-]{#{min_length - 1},}/)
      .reject { |t| stopwords.include?(t) }
      .to_set
end

.jaccard(a, b) ⇒ Float

Compute Jaccard similarity index between two sets.

J(A,B) = |A ∩ B| / |A ∪ B|

Returns 0.0 if either set is empty.

Parameters:

  • a (Set) —

    First token set

  • b (Set) —

    Second token set

Returns:

  • (Float) —

    Similarity score between 0.0 and 1.0



87
88
89
90
91
 
# File 'lib/ast/merge/jaccard_similarity.rb', line 87

def jaccard(a, b)
  return 0.0 if a.empty? || b.empty?

  (a & b).size.to_f / (a | b).size
end