active_mutator

Gem Version

Mutation testing for Ruby, built on Prism. Open source, RSpec-integrated, Rails-first. Available on RubyGems.

active_mutator mutates your code one small change at a time (> becomes >=, && becomes ||, a statement gets deleted, a condition gets forced, and so on). It runs exactly the examples that cover the mutated line, and reports every mutant your suite fails to kill. A surviving mutant is a behavior change no test notices: a precise, machine-verified test gap.

A surviving mutant, in one example

def discount(total)
  return 0 if total < 100
  total / 10
end

it { expect(calc.discount(50)).to eq(0) }
it { expect(calc.discount(200)).to eq(20) }

Both examples pass. Line coverage on discount is 100%. Run active_mutator and one mutant survives anyway:

Surviving mutants:

  Calculator#discount (lib/calculator.rb:11)
    replace `<` with `<=`
    - total < 100
    + total <= 100

Nothing in the test suite calls discount(100), the one input where < and <= disagree. The tests pass, and coverage is green. But the boundary is still unverified. That gap is invisible to coverage and obvious to mutation testing. Add it { expect(calc.discount(100)).to eq(0) } and the mutant is killed.

What is mutation testing?

Coverage answers "did a test run this line?" Mutation testing answers "would a test notice if this line were wrong?" That is a different, and usually more useful, question.

active_mutator applies one small, syntactically valid change to your code (a "mutant") and re-runs only the examples that cover it. If a test fails, the mutant is killed: your tests correctly reject that wrong behavior. If every covering test still passes, the mutant survived: something changed and nothing noticed. A survivor is not a hypothetical. It is the exact line, the exact before and after diff, and proof that no assertion depends on the difference.

Mutation score is (killed + timeout) / (killed + timeout + survived). 100% is usually not the right target. Some mutants are behaviorally equivalent to the original and can never be killed by any test. That is why active_mutator has a committed acceptance ledger. It lets you close survivors out with a stated reason instead of chasing an unreachable score.

Full primer, including the origin of the technique and further reading: docs/guides/what-is-mutation-testing.md.

Install

# Gemfile
group :development, :test do
  gem "active_mutator"
end

Requires Ruby 3.2 or later, RSpec, and a green suite. Linux/macOS (MRI fork).

Quick start

bundle install
bundle exec active_mutator app/models/calculator.rb

The first run performs an instrumented baseline of your suite to build the coverage map. The map is cached in .active_mutator/ and refreshed incrementally after that (see docs/guides/how-it-works.md). Then each mutant runs in its own fork against only its covering examples.

Reading the output

$ bundle exec active_mutator app/models/calculator.rb

.....S..T...U..A....

killed: 14
survived: 1
timeout: 1
error: 0
uncovered: 1
accepted: 1
invalid (discarded): 2
Mutation score: 93.8%

Surviving mutants:

  Calculator#discount (app/models/calculator.rb:9)
    replace `<` with `<=`
    - total < 100
    + total <= 100

Each character on the progress line is one mutant, printed as it finishes:

Char Status Meaning
. killed a covering test failed. Good, the mutant is dead
S survived every covering test passed. This is a test gap
T timeout ran past its time budget. Counted as detected (likely an infinite loop)
E error the worker crashed, or the mutated code raised outside a test assertion
U uncovered no test executes the mutated line at all. This is coverage debt, worse than a survivor
A accepted matches a known-equivalent entry in the acceptance ledger. Excluded from the score

invalid mutants (edits that don't even re-parse as valid Ruby) are discarded before scheduling and reported as a count only. Exit code is 1 if unaccepted survivors exist (or, with --fail-at, if the score is below the threshold), 0 otherwise, including when there are only uncovered, accepted, or error results. The JSON report's exit_reason field reflects survivor presence, independent of the --fail-at gate.

When survivors exist, the summary also prints a per-operator table showing how often each operator's mutants survive, to help spot likely-equivalent mutant patterns.

How it works, compactly

  1. Subject discovery: a Prism visitor finds every method (def) in your target files.
  2. Source-span edits: each operator emits byte-range text edits against the original file, not a rewritten AST. Every mutant is re-parsed with Prism and discarded (invalid) if the edit produced something that doesn't parse. No unparser is ever built or maintained.
  3. Coverage-mapped test selection: one instrumented baseline run maps every source line to the examples that cover it. Incremental runs refresh only what changed instead of re-running the whole suite.
  4. Fork-per-mutant kill runs: the parent preloads your app and spec helper once. Each mutant is inserted and exercised in its own fork against just its covering examples, so results can't bleed state between mutants.

Full architecture, including the coverage-cache format, the fork pipeline, the serial lane for browser specs, timeout budgets, and every status, is in docs/guides/how-it-works.md.

Usage

active_mutator                          # mutate app/ and lib/, full run
active_mutator app/models               # scope by path (directory)
active_mutator app/models/document.rb   # scope to a single file
active_mutator --changed                # uncommitted work only (dev loop)
active_mutator --since origin/main      # PR scope (CI)
active_mutator --subject 'Foo::Bar#baz' # one method
active_mutator --exclude 'lib/generated' # skip a subtree (repeatable)

--subject also takes broader expressions: Foo::Bar (all methods of that constant), Foo::Bar* (raw name prefix — matches Foo::Bar::Qux and also Foo::Barn), Foo::Bar#* (instance methods only), Foo::Bar.* (singleton methods only).

--exclude PAT is a glob relative to the project root, applied during subject discovery, and gitignore-like: lib/generated, lib/generated/, and lib/generated/** all exclude the whole subtree. File globs like **/legacy/* work too.

Skip a single method by putting # active_mutator:skip on the line above its def:

# active_mutator:skip
def legacy_delegator
  target.call
end

Statuses: killed (test failed, this is good), survived (test gap), timeout (counts as detected), uncovered (no covering example, this is coverage debt), accepted (known-equivalent, see ledger), error, invalid (discarded). Exit code is 1 if unaccepted survivors exist (or, with --fail-at, if the score is below the threshold). Mistyped positional paths (a file that doesn't exist, or a non-.rb file) are an error (exit 2) instead of a vacuous green run.

Score = (killed + timeout) / (killed + timeout + survived).

The dev loop

TDD until green, then verify the tests constrain the behavior:

bundle exec active_mutator --changed --format json

Kill survivors by writing the missing tests. For genuine equivalent mutants:

bundle exec active_mutator --changed --accept-survivors   # records to ledger
git add .active_mutator_accepted.json                     # committed state

Acceptance takes effect on the next run. The accepting run still exits 1. Scoped accepting runs (--changed, --subject, path args) are safe: the ledger only prunes entries in files fully scanned by non-narrowed runs, so out-of-scope acceptances are never dropped. Agent workflow: see docs/skills/mutation-check.md.

Reports

--format stryker-json writes .active_mutator/mutation-report.json in the Stryker mutation-testing-report-schema v2 format. Open it in the Stryker report viewer for per-file mutant maps with inline diffs, filterable by status.

--format github prints one ::warning annotation per surviving mutant, so survivors show inline on the PR diff. Pairs with the CI recipe:

bundle exec active_mutator --since origin/main --format github

CI recipe

  • Per-PR: active_mutator --since origin/main --format github (minutes; survivors annotate the PR diff)
  • Nightly: active_mutator --force-baseline (full run; also recovers the residual blind spot — constant-reference detection handles the common newly-covering-example case since 0.2)

Flags

Flag Default Meaning
--jobs N half the cores fork-pool width
--changed none mutate uncommitted + untracked work
--since REF none mutate methods changed since REF
--subject EXPR none subject expression, e.g. Foo#bar, Foo::Bar, Foo::Bar*, Foo#*, Foo.*
--exclude PAT none skip files matching glob during subject discovery (repeatable, gitignore-like)
--max-mutants N none deterministic sample of the first N mutants (quick smoke run on huge scopes; accepted/uncovered mutants count against N)
--debug-plan off print planned mutants as JSON and exit without running
`--format terminal\ json\ stryker-json\ github` terminal report format
--accept-survivors off record survivors to the acceptance ledger
--force-baseline off ignore cached coverage map
--preload-helper FILE / --no-preload-helper auto-detect parent spec-helper preload
--serial-pattern PAT spec/system/, spec/features/ covering-path prefixes forced serial
--browser-boot-seconds S 15 serial-lane timeout bump
--timeout-factor F / --timeout-floor S 8 / 10 mutation timeout budget
--[no-]adaptive-timeout on scale timeout budgets from observed worker wall times (median utilization, grow-only, clamped 1x–4x; --timeout-factor/--timeout-floor set the starting budget)
--require FILE none preload files (repeatable)
--operator FILE none load a custom operator file before analysis (repeatable)
--[no-]class-level on mutate class-level code (macros, constants, DSL/scope lambdas) via class-body subjects
--fail-at SCORE none (strict) exit 0 if score >= SCORE even with survivors (opt-in relaxation for gradual adoption; 0 = report-only)

--debug-plan prints the planned mutant list as one JSON document ({"planned": [...], "pre_resolved": {...}}) and exits without running anything. A coverage baseline is still built or loaded, since timeouts and covering examples come from it.

Every active_mutator process sets ENV["ACTIVE_MUTATOR"] = "1". Use it to guard SimpleCov or other tooling in your spec helper:

SimpleCov.start "rails" unless ENV["ACTIVE_MUTATOR"]

Configuration file

Put team-wide settings in .active_mutator.yml at the project root; CLI flags override file values (--require and --exclude add to the file's lists; the first --serial-pattern replaces them). Recognized keys: jobs, format, timeout_factor, timeout_floor, browser_boot_seconds, fail_at, exclude, serial_patterns, requires, operators (custom operator files, loaded before analysis; see Custom operators), preload_helper (a path, or false to skip preload), adaptive_timeout (true/false), class_level (true/false, default true — mutate class-level code), class_level_closure_cap (integer, default 10 — max constants a class-body mutant may reload before it is skipped). Unknown keys and wrong types are errors, not silent no-ops.

# .active_mutator.yml
jobs: 4
exclude:
  - lib/generated
serial_patterns:
  - spec/system/
fail_at: 90   # legacy suite: gate on score instead of zero-survivors

Class-level mutation

Class-level code — macros (validates, scope, has_many), constants, and DSL/scope lambdas — IS mutated. Each Zeitwerk-shaped file gets a … (class body) subject alongside its method subjects, and the same operator set runs over its class-level statements. Because re-running a macro accumulates rather than replaces (calling validates twice adds a second validator), a class-body mutant can't be inserted with class_eval the way a def mutant is. Instead active_mutator removes the target constant and re-evaluates the whole mutated file, reloading anything attached to it (includers, subclasses, extenders) in dependency order. See docs/guides/how-it-works.md for the full closure-reload pipeline.

Disable it with --no-class-level (or class_level: false in the config file). A class-body mutant whose closure can't be reloaded faithfully — the closure exceeds class_level_closure_cap (default 10), the constant was reopened elsewhere, or an attacher is anonymous/native — is reported skipped (progress char -): listed but not counted in the score, because a mutant we can't insert faithfully must not be called survived or killed.

Known limits

Method bodies and Zeitwerk-shaped class bodies are mutated; the remaining limits are:

  • Class-body mutation requires a Zeitwerk-shaped file — exactly one top-level class/module per file. Multi-constant files and core-class monkey-patches/reopens are not class-body-mutated (issue #32). Their method bodies still are.
  • Most code inside blocks is not mutated. ActiveSupport::Concern DSL blocks (included/prepended/class_methods do … end) ARE mutated — their bodies re-run as class-level code in the includer (issue #31). Every other block (has_many :x do … end and any do … end/{ … } body) is pruned to avoid false survivors from mutating code whose run-time context is unknown.
  • Constants captured by value go stale. A reference that holds the target by value rather than by ancestry — an alias (ALIAS = SomeClass), a registry the class was pushed into, a memoized instance, a class variable captured at load — keeps pointing at the pre-reload object after the closure reload. Such stale references can produce false survivors.
  • Whole-file re-eval re-runs class-body side effects. The reload re-evaluates the target and every attacher's class body, so non-idempotent load-time side effects (global self-registration, descendant tracking) run twice — which can double or mask a count a spec asserts on.
  • refine-based modules are not discovered or reloaded. Refinements are anonymous and don't appear in normal ancestors.
  • RSpec only. Test selection, worker setup, and the world-group filter are all RSpec-API-shaped.
  • Method-body scope details: plain heredoc bodies ARE mutated (emptied); interpolated heredocs are skipped. class << self bodies are mutated as singleton subjects (class << obj and top-level class << self are skipped). Nested defs mutate as part of the enclosing method's body — they get no subject of their own (a directly-inserted mutant would be reverted whenever the outer method re-runs the def).
  • The incremental baseline's residual blind spot: constant-reference detection handles the common case since 0.2; a few residual cases (pure indirection, partially-covering files, leaf-only or wrapper-only references, class ::Foo, Data.define/Struct.new value objects) are caught by nightly --force-baseline.

Guides

  • What is mutation testing?: the concepts. Kill/survive, score, equivalent mutants, further reading.
  • How it works: architecture. Subject discovery, source-span edits, the coverage map, the fork pipeline, and honest limits.
  • Operator reference: every mutation active_mutator can generate, with before/after examples and what a survivor of each one means.
  • Custom operators: write and load your own mutation operators with --operator / the operators: config key.
  • Mutation-check skill: the agent-facing workflow. Run, read survivors, strengthen tests, or accept with a reason.

Contributing

Issues and pull requests welcome. Run bundle exec rspec before sending a change. Also run bundle exec active_mutator --changed on your own diff before sending a change that touches lib/. This is a good idea for the same reason you'd want it run on any other codebase.

License

MIT.