llm-experiment

Run controlled, reproducible experiments on coding agents

Each trial runs in a fresh container with no host state. The host only orchestrates.

Supports Claude Code and Codex CLI.

How it works

llmx runs repeatable coding-agent experiments inside disposable containers. Each trial starts from the same code and task. It records the agent's work, reruns the tests, and collects comparable results.

  1. Define tasks, agents, and conditions in experiment.yml.
  2. Build pinned container images for reproducibility.
  3. Generate the task × agent × condition trial matrix.
  4. Run every trial in a clean, isolated container.
  5. Capture transcripts, diffs, test results, and metrics.
  6. Compare conditions and export sanitized, publishable results.

Requirements

  • macOS on Apple silicon, using Apple's container runtime
  • The container CLI installed and available on your PATH (verified with 0.5.0)
  • Ruby 3.4+

The gem has no runtime dependencies.

Installation

Run:

gem install llm-experiment

This installs the llmx command.

Quick start

llmx new my-question
cd my-question
$EDITOR experiment.yml
export LLMX_APP_CAMPFIRE=/path/to/checkout
llmx doctor
llmx build base
llmx login
llmx build app campfire
llmx run --all --dry-run
llmx run --all
llmx parse --all
llmx metrics
llmx sanitize

Write your prompts to prompts/<app>/<task-id>/<condition>.txt before running.

The gem runs trials. It does not write your prompts or plant your tasks.

Commands

Every command takes --help, --experiment DIR, and --verbose. Use --verbose to echo each container command; it is off by default because a grid runs hundreds of them.

Command Description
llmx new NAME Scaffold an experiment directory
llmx doctor Check CLI, services, disk, auth, images, and prompts
llmx build base Build and smoke-check the base image
llmx build app KEY Build a per-app image from a git bundle
llmx login Log the agent CLIs in to your plan subscriptions
llmx run Run one trial, or the whole grid with --all
llmx status Show which grid cells have results
llmx shell [APP] Open a shell in a trial-shaped container
llmx parse Turn transcripts into events.jsonl and metrics.json
llmx metrics Report medians and exact Mann-Whitney U
llmx sanitize Gate results-raw/ into the committed results/
llmx clean Reclaim disk, verified by observation
llmx version Show the gem version and the default pins

Destructive commands take --dry-run. The grid resumes by default, so pass --redo to re-run a finished cell.

experiment.yml

One file configures the experiment.

name: at-file-mentions
question: "Does @path change what an agent does and what it costs?"

# Optional. Defaults ship with the gem; `llmx version` prints them.
pins:
  claude_code: "2.1.233"
  codex: "0.147.0"
  opencode: "1.18.15"
  ruby_versions: ["3.4.5", "4.0.1"]
  node: "22"

# Trial agents. Only claude and codex are supported.
agents:
  claude:
    model: claude-opus-5
  codex:
    model: gpt-5.6-sol

# One prompt variant per condition.
conditions: [none, bare_noline, at_noline]

trial:
  timeout_seconds: 900
  memory: 6g              # a unit suffix is required
  cpus: 4

apps:
  campfire:
    ruby: "3.4.5"
    bundler: "4.0.13"                                 # or "default"
    database: sqlite3                                 # sqlite3 or postgresql
    test_command: bin/rails test
    db_prepare: bin/rails db:test:prepare
    suite_ran_pattern: '\d+ runs?, \d+ assertions?'   # default shown
    publish_transcripts: true                         # false for private code
    branch_prefix: exp/path-hints                     # source branches on the host
    neutralize: [CLAUDE.md, AGENTS.md, .mcp.json, .claude]

tasks:
  - id: campfire-01
    app: campfire
    test_file: test/lib/restricted_http/private_network_guard_test.rb
    impl_files: [lib/restricted_http/private_network_guard.rb]

Unknown keys are errors. A typo that silently does nothing is how an experiment measures the wrong thing.

postgresql is available in the config format, but that full in-container path has not yet been validated end to end against a real app. Prefer sqlite3 unless you plan to verify the PostgreSQL path yourself.

Host checkout paths never go in this file. They come from LLMX_APP_<KEY>.

Set one per app:

export LLMX_APP_CAMPFIRE=/path/to/campfire

The image builder expects two branch names per app: <branch_prefix>/base and <branch_prefix>/<task-id>.

How a trial stays clean

A trial measures the agent, not the machine it ran on.

  • Every trial gets a fresh container. Nothing survives it.
  • No host state crosses: no user CLAUDE.md, no MCP servers, no hooks, no project memory.
  • Credentials live in ~/.llmx/auth and are mounted, never baked into an image.
  • The mount is a seed, not a home. Each trial copies the credential files and repoints CLAUDE_CONFIG_DIR or CODEX_HOME at the copy.
  • Codex also runs with --ignore-user-config --ephemeral.
  • Application code enters as a git bundle, so untracked secrets and local bundler overrides stay behind.
  • History flattens to one anonymous orphan commit per branch, named "Import application source". The build fails if any other message survives.
  • Coaching files listed under neutralize are deleted during the image build.
  • Prompts cross as base64 in LLMX_PROMPT_B64, never as shell text.
  • A trial aborts with exit 3 unless the suite ran and failed before the agent started.
  • llmx parse exits non-zero on any trial that reports an MCP server, or a memory path inside the shared mount. It checks where a path points, not whether one exists. Pass --lenient to report and continue.

Disk

Budget about 10 GB for the base image and 4-8 GB per app image. The BuildKit cache reached 9.8 GB building one Rails app.

Plan against the unpacked snapshot, not the sizes container image inspect reports. Deleting three app images returned about 16 GB against manifests summing to 1.2-1.6 GB each.

Run llmx clean for a report that measures the snapshot:

llmx clean                  # report only, deletes nothing
llmx clean --images         # every llmx-app-* image, about 5 GB each
llmx clean --builder        # the BuildKit cache, which is regenerable

Both llmx build base and llmx build app refuse to start without room: 10 GB for the base, LLMX_MIN_FREE_GB (25 by default) for an app. Running out mid-build does not fail cleanly. The 25 is measured, not guessed — a Rails app image took about 24 GB, roughly 17 GB of it the BuildKit builder VM rather than the image.

llmx clean --builder verifies with container ls -a afterwards. container builder stop can exit 0 and do nothing, so the exit code is never the evidence.

Private code

Images built from private repositories are never pushed to a registry.

llmx sanitize is the gate between results-raw/ and the committed results/. It refuses the whole publish on a host path or anything shaped like a secret. It refuses rather than redacts, because a surprise in a transcript should stop the commit.

Set publish_transcripts: false for a private app. Only derived measurements cross; raw transcripts and diffs stay in results-raw/, which is gitignored.

License

MIT