rspec-flake-classifier

rspec-flake-classifier records failed RSpec examples, normalizes their failure signatures, and classifies likely flaky-test causes. It can also rerun examples in isolated RSpec subprocesses to separate repeatable regressions from flaky failures.

Installation

Add this line to your application's Gemfile:

gem "rspec-flake-classifier"

And then execute:

bundle install

Or install it yourself as:

gem install rspec-flake-classifier

Requirements:

  • Ruby 3.2 or newer
  • RSpec 3.10 or newer

Runtime dependencies:

  • rspec-core
  • rspec-covers
  • rspec-hermetic

Usage

Configure it from spec/spec_helper.rb or spec/rails_helper.rb:

require "rspec/flake/classifier"

RSpec.configure do |config|
  RSpec::FlakeClassifier.configure(config) do |flake|
    flake.store = ".rspec_flake_store"
    flake.auto_rerun = false
    flake.deflaker = true
    flake.run_sensitivity = false
  end
end

Run your suite normally:

bundle exec rspec

Failure records are written to .rspec_flake_store/failures.jsonl. Each record contains a normalized signature digest, occurrence counts, labels, example IDs, and evidence metadata.

Features

  • Append-only JSONL failure signature store
  • iDFlakies-style rerun protocol for same-order, isolated, OD, and NOD checks
  • Luo taxonomy cause labels such as network, time, io, resource_leak, async_wait, randomness, and test_order_dependency
  • DeFlaker-style coverage and git diff suspicion signal
  • Runtime integration with rspec-covers and rspec-hermetic
  • JSON and JUnit formatters for CI ingestion
  • Buildkite Ruby test collector tags when the collector is active
  • Lightweight feature extraction, prediction, training, and evaluation commands

Configuration

Common options:

RSpec::FlakeClassifier.configure(config) do |flake|
  flake.store = ".rspec_flake_store"
  flake.auto_rerun = { failures: 3 }
  flake.rspec_command = ["bundle", "exec", "rspec"]
  flake.deflaker = true
  flake.changed_lines_provider = -> { { "app/user.rb" => [10, 11] } }
  flake.coverage_provider = ->(example) { example.metadata[:covered_lines] }
  flake.probe_provider = ->(_example) { { files: ["tmp/cache"] } }
  flake.skip_known_flakes = false
  flake.run_sensitivity = false
  flake.sensitivity_factors = %i[time randomness network]
end

auto_rerun starts isolated subprocess reruns for failed examples. Keep it off unless you are comfortable with the extra runtime cost in CI.

rspec-covers and rspec-hermetic

When all three gems are enabled, configure rspec-covers and rspec-hermetic before rspec-flake-classifier. That lets the classifier observe their after-example records from its outer RSpec hook.

require "rspec/covers"
require "rspec/hermetic"
require "rspec/flake/classifier"

RSpec.configure do |config|
  RSpec::Covers.configure(config) do |covers|
    covers.root = Dir.pwd
    covers.production_paths = %w[app lib]
    covers.risky = :report
  end

  RSpec::Hermetic.configure(config) do |hermetic|
    hermetic.root_path = Dir.pwd
    hermetic.probes = %i[filesystem resources]
    hermetic.on_pollution = :report
  end

  RSpec::FlakeClassifier.configure(config) do |flake|
    flake.store = ".rspec_flake_store"
    flake.deflaker = true
  end
end

The classifier reads executed locations from RSpec::Covers.reporter.results and filesystem or resource changes from RSpec::Hermetic.runner.records. It also accepts compatible third-party adapters and explicit metadata such as :flake_classifier_coverage, :flake_classifier_files, :flake_classifier_resources, and :flake_classifier_sockets.

CI output

Require the formatter explicitly when you want machine-readable output:

bundle exec rspec \
  --require rspec/flake/classifier/formatter \
  --format progress \
  --format RSpec::FlakeClassifier::Formatter \
  --out tmp/rspec-flakes.json

For JUnit XML:

bundle exec rspec \
  --require rspec/flake/classifier/formatter \
  --format progress \
  --format RSpec::FlakeClassifier::JUnitFormatter \
  --out test-results/rspec/rspec.xml

The JUnit formatter adds flaky, flaky_labels, flake_signature, buildkite.*, and circleci.* metadata.

Buildkite

Use the official Buildkite Ruby test collector as usual:

require "buildkite/test_collector"

Buildkite::TestCollector.configure(hook: :rspec)

When the collector is active, rspec-flake-classifier tags failed executions with:

  • rspec_flake_classifier.flaky
  • rspec_flake_classifier.labels
  • rspec_flake_classifier.signature

CircleCI

Generate JUnit XML and store the results directory:

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/ruby:3.3
    steps:
      - checkout
      - run: bundle install
      - run:
          name: Run RSpec
          command: |
            bundle exec rspec \
              --require rspec/flake/classifier/formatter \
              --format progress \
              --format RSpec::FlakeClassifier::JUnitFormatter \
              --out test-results/rspec/rspec.xml
      - store_test_results:
          path: test-results

CLI

Investigate a failed example:

rspec-flake investigate 'spec/models/user_spec.rb[1:2]' --seed 1234 --prior 'spec/models/user_spec.rb[1:1]'

Classify a message or existing store records:

rspec-flake classify "Net::HTTP timed out" --json
rspec-flake classify --from-store --store .rspec_flake_store --json

Export features, rank files, train weights, and evaluate results:

rspec-flake features spec/models/user_spec.rb --json
rspec-flake predict spec/models/user_spec.rb spec/system/search_spec.rb --json
rspec-flake train tmp/flake_features.jsonl --out tmp/flake_weights.json --json
rspec-flake evaluate --predictions tmp/predictions.json --ground-truth tmp/ground_truth.json --json

Run sensitivity checks or summarize the store:

rspec-flake sensitivity 'spec/models/user_spec.rb[1:2]' --factor network --json
rspec-flake report --store .rspec_flake_store --json

Labels

Labels are multi-value and confidence-scored. A single failure can receive more than one label.

Common labels include:

  • test_order_dependency
  • async_wait
  • concurrency
  • resource_leak
  • network
  • time
  • io
  • randomness
  • floating_point
  • unordered_collections
  • infrastructure
  • suspected_flaky_deflaker
  • known_flaky

Development

After checking out the repo, run:

bin/setup

Run the test suite:

bundle exec rake spec

Validate RBS signatures:

bundle exec rbs validate

Build the gem:

bundle exec rake build

Open an interactive console:

bin/console

To install this gem onto your local machine:

bundle exec rake install

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ydah/rspec-flake-classifier.

License

The gem is available as open source under the terms of the MIT License.