☯️ Ast::Merge

if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know on Discord or RubyForum, as I may have missed the notification.

---

if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.

👣 How will this project approach the September 2025 hostile takeover of RubyGems? 🚑️

I've summarized my thoughts in this blog post.

🌻 Synopsis

Ast::Merge is not typically used directly - instead, use one of the format-specific gems built on top of it.

Architecture: tree_haver + ast-merge

The *-merge gem family is built on a two-layer architecture:

Layer 1: tree_haver (Parsing Foundation)

tree_haver provides cross-Ruby parsing capabilities:

• Universal Backend Support: Automatically selects the best parsing backend for your Ruby implementation (MRI, JRuby, TruffleRuby)
• Backend Registry: Tracks tree-sitter adapters (mri, rust, ffi, java, tslp, kreuzberg-language-pack), native parsers (prism, psych, commonmarker, markly, rbs), PEG parsers (citrus, parslet), and binary schema support (kaitai-struct)
• Unified API: Write parsing code once, run on any Ruby implementation
• Grammar Discovery: Built-in GrammarFinder for platform-aware grammar library discovery
• Thread-Safe: Language registry with thread-safe caching

Layer 2: ast-merge (Merge Infrastructure)

Ast::Merge builds on tree_haver to provide:

• Base Classes: FreezeNode, MergeResult base classes with unified constructors
• Shared Modules: FileAnalysisBase, FileAnalyzable, MergerConfig, DebugLogger
• Freeze Block Support: Configurable marker patterns for multiple comment syntaxes (preserve sections during merge)
• Node Typing System: NodeTyping for canonical node type identification across different parsers
• Conflict Resolution: ConflictResolverBase with pluggable strategies
• Error Classes: ParseError, TemplateParseError, DestinationParseError
• Region Detection: RegionDetectorBase, FencedCodeBlockDetector for text-based analysis
• RSpec Shared Examples: Test helpers for implementing new merge gems

Merge Gem Backend and Behavior Patterns

Every merge gem in this family must enter parsing through tree_haver. Do not add direct parser fallbacks in merge gems. If a requested backend or grammar is unavailable, fail closed with an unsupported feature or parse diagnostic instead of switching to another parser outside tree_haver.

Merge gems also form language or format families. Each family has a substrate gem that owns shared merge behavior for that language or format. Provider gems register alternate AST backends and delegate family-specific behavior back to the substrate. This keeps parser differences behind TreeHaver while preserving one place for the merge semantics that make a format peculiar.

These are two separate axes:

• Behavior role answers where shared merge semantics live. A substrate gem owns common behavior for a language or format family. A provider gem exposes a backend and delegates family behavior to the substrate when one exists.
• Backend role answers how parsing is registered. A gem may register a TSLP/tree-sitter grammar, a native/provider backend, both, or no parser at all for byte or line oriented merge behavior.

Current examples:

• Substrate with TSLP/tree-sitter registration: markdown-merge, yaml-merge, ruby-merge, and toml-merge register tree-sitter grammar paths and host shared family behavior. json-merge, bash-merge, go-merge, html-merge, rust-merge, and typescript-merge currently use the same TSLP-backed shape for formats without alternate provider gems.
• Provider backed by a native or Ruby parser: commonmarker-merge, kramdown-merge, and markly-merge provide Markdown backends and share behavior with markdown-merge; psych-merge provides a YAML backend and should share behavior with yaml-merge; prism-merge provides a Ruby backend and shares behavior with ruby-merge; citrus-toml-merge and parslet-toml-merge provide TOML backends and share behavior with toml-merge.
• Dual-backend single gem: rbs-merge registers the official RBS parser backend and the tree-sitter RBS grammar in one gem because both surfaces are intentional and tested there.
• Kaitai/binary family: binary-merge is the shared substrate for schema-aware binary merge behavior: byte ranges, preservation reports, unsafe diagnostics, render policies, and nested dispatch conventions. Concrete parser gems such as zip-merge depend on that substrate, register their schema parser path, and produce normalized TreeHaver binary report nodes.
• Synthetic line AST merge gems: plain-merge registers :text and :plain line-backed parser paths for normalized block analysis, and dotenv-merge registers :dotenv and :env parser paths that build dotenv ownership on the plain line substrate.
• Thin substrates are still substrates: binary-merge is intentionally thin while ZIP is the only Kaitai-backed family member. Shared binary behavior should move there as additional binary schema gems prove common patterns.

Backend fallback inside tree_haver is allowed when it is explicit TreeHaver backend selection. Backend fallback outside tree_haver is not allowed, because it bypasses owner identity, backend diagnostics, parser capability reporting, and shared merge-stack behavior.

Partial document insertion, replacement, and removal belong in ast-crispr. PartialTemplateMerger classes should be adapters over structural owners and source-preserving edit plans, not native object round-trips. A YAML partial merge must not parse to Ruby Hashes and emit with Psych.dump; a Markdown partial merge must not slice headings with regular expressions; and a tree-sitter-backed merge must not call the grammar library directly outside TreeHaver.

Creating a New Merge Gem

Pick the merge-gem shape before writing parser code:

• Use a substrate gem when the format needs shared behavior across one or more backends. Register the TSLP/tree-sitter grammar there when that backend is supported, parse through TreeHaver.parser_for, and keep shared merge behavior in the substrate.
• Use a provider gem when the gem exists to expose one concrete parser backend. Register that backend with TreeHaver.register_language, then delegate shared language or format behavior back to the substrate gem.
• Use a dual-backend gem only when both the native provider and TSLP/tree-sitter path are intentional, tested TreeHaver backends in that same gem.

Merge gems should not call parser libraries directly during merge analysis. Direct parser calls belong inside TreeHaver backend adapters.

require "tree_haver"
require "ast/merge"

module MyFormat
  module Merge
    BACKEND_REFERENCE = TreeHaver::BackendReference.new(
      id: "kreuzberg-language-pack",
      family: "tree-sitter"
    ).freeze

    def self.register_backend!
      TreeHaver::GrammarFinder.new(:my_format).register!
    end

    # Inherit from base classes and pass **options for forward compatibility

    class SmartMerger < Ast::Merge::SmartMergerBase
      DEFAULT_FREEZE_TOKEN = "myformat-merge"

      def initialize(template, dest, my_custom_option: nil, **options)
        @my_custom_option = my_custom_option
        super(template, dest, **options)
      end

      protected

      def analysis_class
        FileAnalysis
      end

      def default_freeze_token
        DEFAULT_FREEZE_TOKEN
      end

      def perform_merge
        # Implement format-specific merge logic
        # Returns a MergeResult
      end
    end

    class FileAnalysis
      include Ast::Merge::FileAnalyzable

      def initialize(source, freeze_token: nil, signature_generator: nil, **options)
        @source = source
        @freeze_token = freeze_token
        @signature_generator = signature_generator
        Merge.register_backend!
        @tree = TreeHaver.parser_for(:my_format).parse(source)
        # Project TreeHaver nodes into merge ownership records.
      end

      def compute_node_signature(node)
        # Return signature array for node matching
      end
    end

    class ConflictResolver < Ast::Merge::ConflictResolverBase
      def initialize(template_analysis, dest_analysis, preference: :destination,
        add_template_only_nodes: false, match_refiner: nil, **options)
        super(
          strategy: :batch,  # or :node, :boundary
          preference: preference,
          template_analysis: template_analysis,
          dest_analysis: dest_analysis,
          add_template_only_nodes: add_template_only_nodes,
          match_refiner: match_refiner,
          **options
        )
      end

      protected

      def resolve_batch(result)
        # Implement batch resolution logic
      end
    end

    class MergeResult < Ast::Merge::MergeResultBase
      def initialize(**options)
        super(**options)
        @statistics = {merged_count: 0}
      end

      def to_my_format
        to_s
      end
    end

    class MatchRefiner < Ast::Merge::MatchRefinerBase
      def initialize(threshold: 0.7, node_types: nil, **options)
        super(threshold: threshold, node_types: node_types, **options)
      end

      def similarity(template_node, dest_node)
        # Return similarity score between 0.0 and 1.0
      end
    end
  end
end

Base Classes Reference

Base Class	Purpose	Key Methods to Implement
SmartMergerBase	Main merge orchestration	analysis_class, perform_merge
ConflictResolverBase	Resolve node conflicts	resolve_batch or resolve_node_pair
MergeResultBase	Track merge results	to_s, format-specific output
MatchRefinerBase	Fuzzy node matching	similarity
ContentMatchRefiner	Text content fuzzy matching	Ready to use
FileAnalyzable	File parsing/analysis	compute_node_signature

ContentMatchRefiner

Ast::Merge::ContentMatchRefiner is a built-in match refiner for fuzzy text content matching using Levenshtein distance. Unlike signature-based matching which requires exact content hashes, this refiner allows matching nodes with similar (but not identical) content.

# Basic usage - match nodes with 70% similarity
refiner = Ast::Merge::ContentMatchRefiner.new(threshold: 0.7)

# Only match specific node types
refiner = Ast::Merge::ContentMatchRefiner.new(
  threshold: 0.6,
  node_types: [:paragraph, :heading],
)

# Custom weights for scoring
refiner = Ast::Merge::ContentMatchRefiner.new(
  threshold: 0.7,
  weights: {
    content: 0.8,   # Levenshtein similarity (default: 0.7)
    length: 0.1,    # Length similarity (default: 0.15)
    position: 0.1,   # Position in document (default: 0.15)
  },
)

# Custom content extraction
refiner = Ast::Merge::ContentMatchRefiner.new(
  threshold: 0.7,
  content_extractor: ->(node) { node.text_content.downcase.strip },
)

# Use with a merger
merger = MyFormat::SmartMerger.new(
  template,
  destination,
  preference: :template,
  match_refiner: refiner,
)

This is particularly useful for:

• Paragraphs with minor edits (typos, rewording)
• Headings with slight changes
• Comments with updated text
• Any text-based node that may have been slightly modified

JaccardSimilarity

Ast::Merge::JaccardSimilarity provides set-based fuzzy matching of text blocks using Jaccard index with bigram and token overlap metrics. This is the foundation for detecting renamed or refactored nodes that share similar content.

# Calculate similarity between two text strings
Ast::Merge::JaccardSimilarity.jaccard("def process_users(data)", "def handle_users(data)")
# => 0.75 (high overlap due to shared tokens)

# Extract tokens from text for comparison
tokens = Ast::Merge::JaccardSimilarity.extract_tokens("data.each { |u| validate(u) }")
# => ["data", "each", "validate"]

TokenMatchRefiner

Ast::Merge::TokenMatchRefiner extends MatchRefinerBase for Jaccard-based fuzzy refinement of unmatched node pairs during alignment. It uses greedy best-first matching to pair orphan nodes that have similar body text.

refiner = Ast::Merge::TokenMatchRefiner.new(
  threshold: 0.6,                  # Minimum Jaccard similarity (default: 0.6)
  node_types: [:def, :class],       # Only match these node types
)

merger = MyFormat::SmartMerger.new(
  template,
  destination,
  match_refiner: refiner,
)

CompositeMatchRefiner

Ast::Merge::CompositeMatchRefiner chains multiple refiners sequentially, enabling multi-strategy matching in a single alignment pass. Each refiner operates on the residual unmatched nodes from the previous refiner.

composite = Ast::Merge::CompositeMatchRefiner.new(refiners: [
  Ast::Merge::ContentMatchRefiner.new(threshold: 0.8),  # strict text match first
  Ast::Merge::TokenMatchRefiner.new(threshold: 0.5),    # then looser token match
])

merger = MyFormat::SmartMerger.new(
  template,
  destination,
  match_refiner: composite,
)

Namespace Reference

The Ast::Merge module is organized into several namespaces, each with detailed documentation:

Namespace	Purpose	Documentation
Ast::Merge::Detector	Region detection and merging	lib/ast/merge/detector/README.md
Ast::Merge::Recipe	YAML-based merge recipes	lib/ast/merge/recipe/README.md
Ast::Merge::Comment	Comment parsing and representation	lib/ast/merge/comment/README.md
Ast::Merge::Text	Plain text AST parsing	lib/ast/merge/text/README.md
Ast::Merge::RSpec	Shared RSpec examples	lib/ast/merge/rspec/README.md

Key Classes by Namespace:

• Detector: Region, Base, Mergeable, FencedCodeBlock, YamlFrontmatter, TomlFrontmatter
• Recipe: Config, Runner, ScriptLoader
• Comment: Line, Block, Empty, Parser, Style
• Text: SmartMerger, FileAnalysis, LineNode, WordNode, Section
• RSpec: Shared examples and dependency tags for testing *-merge implementations

💡 Info you can shake a stick at

Tokens to Remember
Works with MRI Ruby 4
Support & Community
Source
Documentation
Compliance
Style
Maintainer 🎖️
... 💖	🧊 🐙 🛖 🧪

Compatibility

Compatible with MRI Ruby 4.0.0+, and concordant releases of JRuby, and TruffleRuby. CI workflows and Appraisals are generated for MRI Ruby 4.0.0+. This test floor is configured by ruby.test_minimum in .kettle-jem.yml and may be higher than the gem's runtime compatibility floor when legacy Rubies are not practical for the current toolchain.

The amazing test matrix is powered by the kettle-dev stack.

How kettle-dev manages complexity in tests

Gem	Source	Role	Total downloads
appraisal2	GitHub	multi-dependency Appraisal matrix generation
appraisal2-rubocop	GitHub	RuboCop Appraisal generator integration
kettle-dev	GitHub	development, release, and CI workflow tooling
kettle-jem	GitHub	Appraisals & CI workflow templates
kettle-soup-cover	GitHub	SimpleCov coverage policy and reporting
kettle-test	GitHub	standard test runner and coverage harness
rubocop-lts	GitHub	Ruby-version-aware linting
turbo_tests2	GitHub	parallel test execution

✨ Installation

Install the gem and add to the application's Gemfile by executing:

bundle add ast-merge

If bundler is not being used to manage dependencies, install the gem by executing:

gem install ast-merge

⚙️ Configuration

ast-merge provides base classes and shared interfaces for building format-specific merge tools. Each implementation (like prism-merge, psych-merge, etc.) has its own SmartMerger with format-specific configuration.

Common Configuration Options

All SmartMerger implementations share these configuration options:

merger = SomeFormat::Merge::SmartMerger.new(
  template,
  destination,
  # When conflicts occur, prefer template or destination values
  preference: :template,            # or :destination (default), or a Hash for per-node-type
  # Add nodes that only exist in template (Boolean or callable filter)
  add_template_only_nodes: true,    # default: false, or ->(node, entry) { ... }
  # Custom node type handling
  node_typing: {},                # optional, for per-node-type preference
)

Signature Match Preference

Control which source wins when both files have the same structural element:

• :template - Template values replace destination values
• :destination (default) - Destination values are preserved
• Hash - Per-node-type preference (see Advanced Configuration)

Template-Only Nodes

Control whether to add nodes that only exist in the template:

• true - Add all template-only nodes
• false (default) - Skip template-only nodes
• Callable - Filter which template-only nodes to add

Callable Filter

When you need fine-grained control over which template-only nodes are added, pass a callable (Proc/Lambda) that receives (node, entry) and returns truthy to add or falsey to skip:

# Only add nodes with gem_family signatures
merger = SomeFormat::Merge::SmartMerger.new(
  template,
  destination,
  add_template_only_nodes: ->(node, entry) {
    sig = entry[:signature]
    sig.is_a?(Array) && sig.first == :gem_family
  },
)

# Only add link definitions that match a pattern
merger = Markly::Merge::SmartMerger.new(
  template,
  destination,
  add_template_only_nodes: ->(node, entry) {
    entry[:template_node].type == :link_definition &&
      entry[:signature]&.last&.include?("gem")
  },
)

The entry hash contains:

• :template_node - The node being considered for addition
• :signature - The node's signature (Array or other value)
• :template_index - Index in the template statements
• :dest_index - Always nil for template-only nodes

🔧 Basic Usage

Using Shared Examples in Tests

# spec/spec_helper.rb
require "ast/merge/rspec/shared_examples"

# spec/my_format/merge/freeze_node_spec.rb
RSpec.describe(MyFormat::Merge::FreezeNode) do
  it_behaves_like "Ast::Merge::FreezeNode" do
    let(:freeze_node_class) { described_class }
    let(:default_pattern_type) { :hash_comment }
    let(:build_freeze_node) do
      lambda { |start_line:, end_line:, **opts|
        # Build a freeze node for your format
      }
    end
  end
end

Available Shared Examples

• "Ast::Merge::FreezeNode" - Tests for FreezeNode implementations
• "Ast::Merge::MergeResult" - Tests for MergeResult implementations
• "Ast::Merge::DebugLogger" - Tests for DebugLogger implementations
• "Ast::Merge::FileAnalysisBase" - Tests for FileAnalysis implementations
• "Ast::Merge::MergerConfig" - Tests for SmartMerger implementations

🔐 Security

See SECURITY.md.

🤝 Contributing

If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check issues or PRs, or use the gem and think about how it could be better.

We so if you make changes, remember to update it.

See CONTRIBUTING.md for more detailed instructions.

Code Coverage

Coverage service badges

📌 Versioning

This library follows for its public API where practical. For most applications, prefer the Pessimistic Version Constraint with two digits of precision.

For example:

spec.add_dependency("ast-merge", "~> 7.0")

📌 Is "Platform Support" part of the public API? More details inside.

Dropping support for a platform can be a breaking change for affected users. If a release changes supported platforms, it should be called out clearly in the changelog and versioned with that impact in mind.

To get a better understanding of how SemVer is intended to work over a project's lifetime, read this article from the creator of SemVer:

• "Major Version Numbers are Not Sacred"

See CHANGELOG.md for a list of releases.

📄 License

The gem is available under the following licenses: AGPL-3.0-only, PolyForm-Small-Business-1.0.0. See LICENSE.md for details.

If none of the available licenses suit your use case, please contact us to discuss a custom commercial license.