rspec-hermetic

rspec-hermetic detects examples that leave process-global state changed after they finish. It snapshots selected probes before and after each example, diffs the snapshots, applies allowlists, and reports the example as a polluter when state is still dirty.

The implementation follows the PolDet idea from state-pollution testing: compare shared state around one test and show the before/after evidence.

Installation

Install the gem and add it to the application's Gemfile:

bundle add rspec-hermetic

Or install from a local checkout:

gem "rspec-hermetic", path: "../rspec-hermetic"

Usage

Configure it in spec/spec_helper.rb:

require "rspec/hermetic"

RSpec.configure do |config|
  RSpec::Hermetic.configure(config) do |hermetic|
    hermetic.probes = %i[env constants globals ruby_runtime rails time randomness resources]
    hermetic.on_pollution = :risky # :risky, :fail, or :report
    hermetic.constant_namespaces = [MyApp]
    hermetic.filesystem_paths = ["app", "config"]
    hermetic.randomness_seed_probe = true # opt in to Kernel.srand seed sampling
    hermetic.resource_process_probe = true # opt in to child process scanning
    hermetic.auto_reset = false # true or probe list, e.g. %i[env rails time]
    hermetic.candidate_report_path = "tmp/rspec_hermetic_candidates.json"

    hermetic.allow do |allow|
      allow.env "CI", "RAILS_*"
      allow.path "tmp/**", "log/test.log"
      allow.constant(/\AFactoryBot::/)
      allow.append_only "Warning.deduplicated"
    end
  end
end

Use metadata for intentional pollution in one example:

it "changes locale globally", hermetic: { allow: [:i18n_locale] } do
  I18n.locale = :ja
end

Available probes:

  • env: ENV.to_h
  • constants: configured namespaces, or top-level constants by default using bounded ObjectSpace reachable-object fingerprints
  • globals: selected Ruby globals such as $LOAD_PATH and stdio
  • ruby_runtime: warning, encoding, GC, and thread runtime flags
  • rails: I18n, Time.zone, Rails.cache, mailer/job counts, CurrentAttributes
  • time: wall-clock/monotonic offset, ActiveSupport time helper state, and Timecop travel state when present
  • randomness: Random/FactoryBot state that can be read safely, with optional Kernel.srand seed sampling via randomness_seed_probe
  • resources: live threads, open IO objects, ActiveRecord pool state, and optional child process scanning via resource_process_probe
  • filesystem: configured paths under root_path (. by default when enabled), with metadata and small-file SHA256 content hashes

append_only patterns only downgrade a change to a warning when the new value preserves the old value as a prefix/subset. Replacing existing values is still reported as pollution.

The RSpec hook is installed outside existing around(:each) hooks on RSpec 3 by using RSpec's hook collection when available. On unsupported RSpec versions it falls back to normal around(:each) registration.

auto_reset is opt-in. Set it to true or a probe list to restore supported state after reporting: ENV, selected globals/runtime flags, Rails globals, time helpers, randomness seed, added constants, leaked threads/IO objects, leaked child processes, and filesystem additions/modifications/deletions when file contents were captured.

Resource origin tracking is enabled by default. It annotates resources created while an example is running so leaked threads and IO objects can include the creation callsite in the report.

For order-dependent failures, require forensic mode:

bundle exec rspec --seed 1234 --require rspec/hermetic/forensic

It compares a failing example's starting state with the suite baseline and reports the last example that changed each dirty key when known. Candidate pairs are written to candidate_report_path as JSON.

To verify known polluter/victim candidates:

bundle exec rake hermetic:verify[tmp/rspec_hermetic_candidates.json]

The JSON file can be an array of objects or newline-delimited JSON:

[
  { "polluter": "spec/models/user_spec.rb:12", "victim": "spec/models/user_spec.rb:48" }
]

Minitest can use the same detector:

require "rspec/hermetic/minitest"

RSpec::Hermetic.configure_minitest do |hermetic|
  hermetic.probes = %i[env globals ruby_runtime resources filesystem]
  hermetic.on_pollution = :fail
end

Run the built-in seeded pollution evaluation:

bundle exec rake hermetic:evaluate[tmp/rspec_hermetic_evaluation.json]

The report includes probe-level seeded recall and is intended as the local baseline for the evaluation workflow described in the design document.

Run an external corpus evaluation by providing commands through the environment:

HERMETIC_BASELINE_COMMAND="bundle exec rspec" \
HERMETIC_COMMAND="bundle exec rspec --require rspec/hermetic/forensic" \
HERMETIC_CANDIDATES="tmp/rspec_hermetic_candidates.json" \
bundle exec rake hermetic:evaluate_corpus[tmp/rspec_hermetic_corpus.json]

If HERMETIC_JUDGMENTS points to a JSON file keyed by polluter|victim|probe|key, the corpus report includes false-alarm counts and rates alongside overhead and candidate counts.

Development

After checking out the repo, run bin/setup to install dependencies. Then run rake spec to run the tests. You can also run bin/console for an interactive prompt.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/ydah/rspec-hermetic.

License

The gem is available as open source under the terms of the MIT License.