Ruby Skill Bench

A high-fidelity evaluation engine for benchmarking AI agent skills across any stack (Rails-first, but extensible).

Part of the AI Skill Ecosystem

This repo is one of 6 in a composable AI skill ecosystem:

Repo	Role
ruby-core-skills	15 shared Ruby skills + process discipline
rails-agent-skills	28 Rails-specific skills + 9 agents
hanakai-yaku	35 Hanami/dry-rb skills + 10 agents
agnostic-planning-skills	10 planning skills + 4 agents
agent-mcp-runtime	Rust CLI runtime (pack resolution, MCP)
ruby-skill-bench	Benchmark/eval engine

See the Ecosystem Overview for the full architecture.

---

Features

• Side-by-Side Evaluation: Quantify the "ROI of Context" by comparing baseline vs. skill-enhanced agent runs.
• Isolated Git Sandboxes: Every run operates in a temporary repo. Clean diffs, zero side-effects, 100% reproducibility.
• Blind Judging with Dimensions: LLM judge scores baseline and context independently across 5 canonical dimensions (Correctness, Skill Adherence, Code Quality, Test Coverage, Documentation). Eval authors configure weights and thresholds via criteria.json.
• Sophisticated ReAct Loop: Employs a robust Thought → Tool → Observation loop to handle complex, multi-step engineering tasks.
• Multi-Provider Ecosystem: Native support for OpenAI, Anthropic, Google Gemini, Azure OpenAI, Ollama, Groq, DeepSeek, Mistral, and OpenCode.
• Standardized Intelligence: Consistent reporting format regardless of the underlying LLM provider.

---

Architecture Overview

The system decoupling allows the reasoning engine to remain agnostic of the execution environment.

CLI / API → RunnerService → Sandbox + ReAct Agent → LLM Client Layer → Provider
                                                              ↓
                                         EvaluationRunner (baseline + context)
                                                              ↓
                                                    Judge (blind scoring)
                                                              ↓
                                                    DeltaReport

---

Configuration & Orchestration

Environment Variable Mapping

Provider	Required Env Variables	Registry Key
OpenAI	SKILL_BENCH_OPENAI_API_KEY	:openai
Anthropic	SKILL_BENCH_ANTHROPIC_API_KEY	:anthropic
Gemini	SKILL_BENCH_GEMINI_API_KEY	:gemini
Azure	SKILL_BENCH_AZURE_API_KEY	:azure
Ollama	—	:ollama
Groq	SKILL_BENCH_GROQ_API_KEY	:groq
DeepSeek	SKILL_BENCH_DEEPSEEK_API_KEY	:deepseek
Mistral	SKILL_BENCH_MISTRAL_API_KEY	:mistral
OpenCode	SKILL_BENCH_OPENCODE_API_KEY, SKILL_BENCH_OPENCODE_BASE_URL	:opencode

Note: Environment variables are loaded automatically. You can also configure provider settings in skill-bench.json (created by skill-bench init).

OpenCode requires a custom base_url: OpenCode does not host a public LLM API. You must provide your own OpenAI-compatible endpoint (e.g. a LiteLLM proxy, self-hosted vLLM, or company gateway) via the base_url config key. Without it, the provider will fail with "Base URL not set for Opencode".

Mistral uses Mistral's OpenAI-compatible chat completions API (default model mistral-large-latest). Set SKILL_BENCH_MISTRAL_API_KEY and scaffold it with skill-bench init --mistral.

Command Allowlist

By default, no shell commands are permitted. You must configure allowed_commands in skill-bench.json:

{
  "provider": "openai",
  "max_execution_time": 30,
  "allowed_commands": ["rspec", "bundle", "ruby", "git"],
  "allow_host_execution": false,
  "config": {
    "api_key": null,
    "model": "gpt-4o"
  }
}

Security: The agent can only execute commands on this list. Dangerous commands (bash, curl, sudo, etc.) are always blocked regardless of configuration.

Where commands run: Allowed commands run inside a temporary git sandbox directory on the host — a copy of your eval files, not your project. True container isolation (Docker) is not yet shipped, so the sandbox directory is the only boundary. Because of this, host execution fails closed: it is disabled by default and must be explicitly enabled with "allow_host_execution": true. With it disabled (the default), run_command refuses to execute and returns an error instead of running un-isolated. Enable it only when you accept that allowed commands run directly on your machine.

Configuration Hierarchy

Configuration is loaded in this order (later sources override earlier ones):

1. Code defaults — built-in defaults for provider, model, and timeout
2. Home JSON — ~/.skill-bench.json for user-wide settings
3. Local JSON — ./skill-bench.json for project-specific settings
4. Environment variables — provider API keys and models from ENV

---

Getting Started

Installation

gem install ruby-skill-bench

Or add to your Gemfile:

gem 'ruby-skill-bench'

Usage: The 4-Step Flow

Each command creates specific files. Here is exactly what lands on disk after each step.

1. Initialize Configuration

skill-bench init --openai

Creates: skill-bench.json (provider configuration)

{
  "provider": "openai",
  "max_execution_time": 30,
  "allowed_commands": ["rspec", "bundle", "ruby", "git"],
  "config": {
    "api_key": null,
    "model": "gpt-4o"
  }
}

Available providers: --openai, --anthropic, --gemini, --ollama, --azure, --groq, --deepseek, --mistral, --opencode

Zero-config offline path: skill-bench init --mock scaffolds a minimal offline config that needs no API key and no network — {"provider":"mock","max_execution_time":30}. Use it to try the full flow (and run the bundled examples) before wiring up a real provider.

Use --force to overwrite an existing config.

---

2. Create a Skill

skill-bench skill new my-service --mode=rails --template=service_object

Creates:

skills/
└── my-service/
    └── SKILL.md          # <- Your skill instructions go here

SKILL.md is free-form Markdown. It typically contains:

• What pattern the skill implements (e.g., "Service Object with .call")
• Hard rules the agent must follow
• Code examples
• Response format expectations

Example SKILL.md:

# Service Object Skill

## Pattern

All service objects use the `.call` class method and return a standardized hash:

```ruby
{ success: true, response: { data: ... } }

Hard Rules

1. Every .rb file begins with # frozen_string_literal: true
2. Every public method has YARD docs (@param, @return, @raise)
3. rescue StandardError blocks must log backtrace

---

### Using TemplateRegistry for Rapid Eval Scaffolding

For programmatic eval creation, use `SkillBench::Services::TemplateRegistry` to generate scaffolding from pre-built templates. This is ideal for automating eval creation or building tools on top of SkillBench.

**Basic Usage:**

```ruby
require 'skill_bench'

# Generate a task template for a CRUD service
task_content = SkillBench::Services::TemplateRegistry.call(
  :task_md, 
  :crud, 
  skill_name: "UserCreator"
)

# Generate criteria JSON for an API client
criteria_content = SkillBench::Services::TemplateRegistry.call(:criteria_json, :api)

# Generate skill instructions for a background job
skill_content = SkillBench::Services::TemplateRegistry.call(
  :skill_md, 
  :background_job, 
  skill_name: "OrderProcessor"
)

Available Template Types:

Type	Output	Purpose
task_md	Markdown	Agent prompt with requirements
criteria_json	JSON	Scoring rules and dimensions
skill_md	Markdown	Skill instructions for the agent

Supported Categories:

Category	Use Case
crud	Service Objects with Create, Read, Update, Delete
api	API clients with authentication and error handling
background_job	ActiveJob/Sidekiq workers with retry logic
controller	RESTful controllers with strong parameters
model	ActiveRecord models with validations
migration	Database migrations with indexes
concern	ActiveSupport::Concern modules
policy	Authorization policies (Pundit-style)
form_object	Form objects with validations
view_component	ViewComponent components with previews

Variable Interpolation:

Templates support {{variable_name}} syntax for dynamic content:

# Custom variables are interpolated into templates
task = SkillBench::Services::TemplateRegistry.call(
  :task_md, 
  :api, 
  skill_name: "PaymentGateway",
  endpoint: "/api/v1/payments"
)

Complete Workflow Example:

require 'fileutils'
require 'skill_bench'

# Define your skill name
skill_name = "OrderService"

# Generate all eval scaffolding
task_md = SkillBench::Services::TemplateRegistry.call(:task_md, :crud, skill_name: skill_name)
criteria_json = SkillBench::Services::TemplateRegistry.call(:criteria_json, :crud)
skill_md = SkillBench::Services::TemplateRegistry.call(:skill_md, :crud, skill_name: skill_name)

# Write to disk
FileUtils.mkdir_p("evals/order-service")
File.write("evals/order-service/task.md", task_md)
File.write("evals/order-service/criteria.json", criteria_json)

FileUtils.mkdir_p("skills/order-service")
File.write("skills/order-service/SKILL.md", skill_md)

puts "Eval scaffolding created for #{skill_name}!"

Note: TemplateRegistry is a pure function with no side effects. It returns template strings that you can customize before writing to disk.

---

3. Create an Eval

You have two options: manual or auto-generated.

Option A — Manual (full control):

skill-bench eval new my-first-eval --runtime=rails

Creates:

evals/
└── my-first-eval/
    ├── task.md           # <- The task description for the agent
    └── criteria.json     # <- Scoring rules and dimension weights

task.md tells the agent what to build. Be specific — the agent receives this as its user prompt.

Example task.md:

Create a `UserRegistrationService` that:

1. Accepts `email` and `password`
2. Validates email format with a regex
3. Validates password length (minimum 8 characters)
4. Returns `{ success: true, response: { user_id: ... } }` on success
5. Returns `{ success: false, response: { error: { message: ... } } }` on failure
6. Includes YARD documentation for every public method
7. Includes RSpec tests that cover both success and failure paths

criteria.json tells the judge how to score the agent's output. See the Scoring Engine section for the full format.

Option B — Auto-Generated (from a skill):

skill-bench eval generate my-service --name my-first-eval

Reads skills/my-service/SKILL.md, sends it to the LLM, and auto-generates task.md + criteria.json. The generated eval is immediately validated against the same rules as manual evals.

---

4. Run the Eval

skill-bench run my-first-eval --skill=my-service

What happens internally:

1. Resolve — Load eval (task.md + criteria.json), skill (SKILL.md), and provider config
2. Baseline run — Agent receives task.md as a prompt, no skill context → produces output A
3. Context run — Agent receives task.md + SKILL.md as prompt → produces output B
4. Blind judging — LLM judge scores output A and output B independently across the dimensions defined in criteria.json
5. Delta computation — Compare scores, compute deltas, apply pass/fail logic
6. History recording — Store result in .skill-bench-trends.json for trend tracking

Provider is read from skill-bench.json — no --provider flag needed.

Run with multiple skills (skill chaining):

skill-bench run my-first-eval --skill=skill-a --skill=skill-b

Both skill contexts are concatenated and sent to the agent. The judge evaluates whether the combined context improves results.

Output Formats: --format human (default), json, junit, or html.

• Human-readable (default) — full delta table, iteration timeline, and a Tokens: N | Est. Cost: $X.XXXX line.

• JSON: --format json — machine-readable, including top-level tokens and cost fields.

• JUnit XML: --format junit — for CI test reporting.

• HTML: --format html — a self-contained, shareable report (styles inlined, no external assets) with the delta table and iteration timeline. Redirect it to a file:

  skill-bench run my-first-eval --skill=my-service --format html > report.html
  

---

Pre-flight Checks: validate / doctor

Before spending tokens on a run, sanity-check your setup. skill-bench validate (aliased as doctor) runs read-only pre-flight checks — it never runs an eval and never makes a network call:

skill-bench validate
# or, identically:
skill-bench doctor

It runs three checks and prints a PASS / FAIL / SKIP line for each:

1. criteria — validates the criteria JSON (default criteria.json, override with --criteria PATH). Skipped if the default file is absent.
2. config — schema-checks skill-bench.json (default, override with --config PATH): provider is required and must be a known provider, max_execution_time must be a positive integer, and config (when present) must be an object.
3. provider key — reports whether the configured provider's API key is present (the mock provider needs none).

A passing report exits 0:

skill-bench validate

[PASS] criteria      criteria.json is valid
[PASS] config        skill-bench.json matches the expected shape
[PASS] provider key  openai credentials present

All checks passed.

A failure exits non-zero and names what is wrong:

[FAIL] provider key  openai is missing: api_key

1 check(s) failed.

---

Multi-Repo Skill Benchmarking

Skills in the ecosystem are split across multiple repos:

• ruby-core-skills — 15 shared Ruby skills (DDD, patterns, process discipline)
• rails-agent-skills — 28 Rails-specific skills
• hanakai-yaku — 35 Hanami/dry-rb skills

To benchmark a skill from an external repo, use the --skill flag:

# Benchmark a core skill
skill-bench run evals/skills/write-yard-docs/basic \
  --skill /path/to/ruby-core-skills/skills/patterns/write-yard-docs

# Benchmark a Rails skill
skill-bench run evals/skills/code-review/pr-review \
  --skill /path/to/rails-agent-skills/skills/code-quality/code-review

Config-Based Multi-Repo Resolution

Configure skill_sources in skill-bench.json to automatically resolve skills across repos without --skill every time:

{
  "provider": "openai",
  "model": "gpt-4o",
  "skill_sources": {
    "core": "../ruby-core-skills/skills",
    "rails": "../rails-agent-skills/skills",
    "hanami": "../hanakai-yaku/skills"
  }
}

Each key is a source name (for logging), each value is a path to a skills/ directory. When a skill is not found locally, SkillBench iterates through skill_sources and uses the first match.

Pack-Based Resolution (--pack)

Resolve skills via the ecosystem registry manifest (from agent-mcp-runtime):

# Run an eval using the Rails pack's version of code-review
skill-bench run evals/skills/code-review/basic \
  --skill code-review \
  --pack rails

# Override the default registry manifest path
skill-bench run evals/skills/code-review/basic \
  --skill code-review \
  --pack rails \
  --registry-manifest /path/to/registry.json

Variant Comparison (compare)

Compare the same skill across two pack variants to measure context-dependent performance:

skill-bench compare code-review \
  --variant-a "pack:rails" \
  --variant-b "pack:hanami" \
  --eval evals/skills/code-review/basic

The --variant spec supports two forms:

• pack:<name> — resolve via registry manifest
• /absolute/path or relative/path — use a direct path

Response Caching (opt-in, --cache)

LLM responses can be cached so identical requests reuse a previous result instead of calling the provider again. Caching is off by default. Enable it per run with --cache, or set the SKILL_BENCH_CACHE environment variable to a truthy value (1, true, yes, or on):

# Per-run flag
skill-bench run my-first-eval --skill=my-service --cache

# Or via the environment
SKILL_BENCH_CACHE=1 skill-bench run my-first-eval --skill=my-service

The cache is in-memory (process-lifetime) and content-addressed: the key is a SHA-256 digest of the provider, model, system prompt, messages, tools, and temperature, so only truly identical requests share an entry. The mock and null providers are never cached.

This pays off most with compare, which runs the skill-less baseline twice with identical inputs — with caching enabled, the repeated baseline reuses the cached response instead of making a second call.

---

File Reference: What Lives on Disk

SkillBench creates and manages three files in your project. Understanding them helps you iterate faster.

skill-bench.json — Your Configuration

What it is: The config file you create with skill-bench init. It tells SkillBench which LLM provider to use, your API key, timeout settings, and which shell commands the agent is allowed to run.

Who edits it: You. This is the only file SkillBench expects you to write by hand.

Typical contents:

{
  "provider": "openai",
  "max_execution_time": 300,
  "allowed_commands": ["rspec", "bundle", "ruby", "git"],
  "allow_host_execution": false,
  "config": {
    "api_key": "sk-...",
    "model": "gpt-4o",
    "max_iterations": 25
  }
}

Key rules:

• Configuration is loaded in this order: code defaults → ~/.skill-bench.json (user-wide) → ./skill-bench.json (local) → environment variables. Later sources override earlier ones.
• If api_key is null, SkillBench looks for the matching environment variable (e.g. SKILL_BENCH_OPENAI_API_KEY).
• allowed_commands is a safeguard, not a convenience. By default the agent cannot run any shell command. Add only what your evals need.
• allow_host_execution (default false) gates whether run_command may run on the host when no container isolation is active. Since container isolation is not yet shipped, leaving it false means run_command fails closed (refuses to execute). Set it to true only if you accept that allowed commands run directly on your machine inside the temporary sandbox directory.

---

.skill-bench-trends.json — Evaluation History (Auto-Generated)

What it is: A JSON array that records every successful eval run. SkillBench appends to it automatically. It stores the timestamp, eval name, skill names, scores, and deltas so you can track improvement over time.

Who edits it: Nobody. SkillBench writes it; you read it. If you delete it, you lose your trend data.

Example entry:

[
  {
    "timestamp": "2026-05-12T10:30:00Z",
    "eval_name": "my-first-eval",
    "skill_names": ["my-service"],
    "verdict": true,
    "baseline_total": 32,
    "context_total": 87,
    "deltas": {
      "correctness": 16,
      "skill_adherence": 17,
      "code_quality": 6,
      "test_coverage": 10,
      "documentation": 6
    }
  }
]

Why it matters: This file powers the TREND line you see in human-readable output:

TREND: baseline ↑ (+2), context ↑ (+7)

The trend compares the current run against the previous run of the same eval + skill. This tells you at a glance whether your latest skill edit made things better or worse.

Pro tip: .skill-bench-trends.json is git-ignored by default (via the .skill-bench-trends.json* line in .gitignore). If you want to share trend data with your team, remove that line so the file can be committed.

---

.skill-bench-trends.json.bak — Backup (Auto-Generated)

What it is: A snapshot of the previous good version of .skill-bench-trends.json, copied just before each new write. (The first run has no prior version yet, so it creates no .bak.) If the main file gets corrupted (e.g. you kill the process mid-write), SkillBench automatically falls back to the .bak file.

Who edits it: Nobody. It is a safety net.

When to care: Almost never. If you see a "History file corrupted" warning, SkillBench has already recovered from the .bak for you.

---

Iterating on Skills: A Practical Workflow

Writing a good skill is rarely a one-shot process. Here is a tested workflow that uses the history file to guide your improvements.

Step 1: Write a V1 Skill

Create a skill and an eval that exercises it:

skill-bench skill new my-service --mode=rails --template=service_object
skill-bench eval new my-first-eval --runtime=rails
# ... edit SKILL.md, task.md, and criteria.json ...

Step 2: Run the Eval (Baseline + Context)

skill-bench run my-first-eval --skill=my-service

This executes the full evaluation pipeline: a baseline run (agent receives the task without the skill) and a context run (agent receives the task with the skill). The two outputs are scored independently by the judge and compared.

Read the output carefully. Look at two things:

1. Verdict: Did it pass? If not, which dimension failed?
2. Delta: Which dimensions improved the most? Which improved the least?

Step 3: Inspect the History

cat .skill-bench-trends.json | jq '.[-1]'

This shows the latest entry. Focus on the dimension with the smallest delta — that is where your skill is weakest.

Step 4: Edit the Skill

Suppose test_coverage only improved by +3. Open skills/my-service/SKILL.md and add a concrete rule:

## Hard Rules

... existing rules ...

5. Every service must include RSpec tests with at least:
   - One happy-path test
   - One error-path test
   - Use of `let` and `subject` blocks

Step 5: Re-run and Compare Trends

skill-bench run my-first-eval --skill=my-service

Watch the TREND line:

TREND: baseline → (0), context ↑ (+5)

The context score went up by 5 points compared to the previous run. If test_coverage delta jumped from +3 to +8, your edit worked.

Step 6: Iterate Until Stable

Repeat steps 4-5 until:

• The eval passes consistently (2-3 runs in a row)
• Deltas are stable (not swinging wildly)
• The trend line shows context → (0) or small positive deltas

When to Stop Iterating

Situation	Action
Context score is ~95+ and deltas are flat	Your skill is mature. Move on.
Context score is stuck below threshold	Your eval task might be too hard, or your skill rules are too vague. Rewrite task.md with clearer acceptance criteria.
Baseline score is already high	The task is too easy. Make task.md harder so the skill has room to show value.
One dimension is always low	Add a specific rule to SKILL.md targeting that dimension.

---

Scoring Engine

The engine runs every eval twice — once without skill context (baseline) and once with skill context — then uses an LLM judge to score both outputs independently across configurable dimensions.

How It Works (Visual Walkthrough)

┌────────────────────────────────────────────────────────────────────────┐
│                         EVALUATION PIPELINE                            │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  Step 1: Baseline Run                                                  │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │  task.md    │───→│   Agent     │───→│  Output A   │                 │
│  └─────────────┘    │  (no skill) │    │  (git diff) │                 │
│                     └─────────────┘    └─────────────┘                 │
│                                                                        │
│  Step 2: Context Run                                                   │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │  task.md    │───→│   Agent     │───→│  Output B   │                 │
│  │  SKILL.md   │───→│  (+ skill)  │───→│  (git diff) │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
│                                                                        │
│  Step 3: Blind Judging (two independent calls)                         │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │ Output A    │───→│   Judge     │───→│  Score A    │                 │
│  │ criteria    │    │  (baseline) │    │  per dim    │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
│                                                                        │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐                 │
│  │ Output B    │───→│   Judge     │───→│  Score B    │                 │
│  │ criteria    │    │  (context)  │    │  per dim    │                 │
│  └─────────────┘    └─────────────┘    └─────────────┘                 │
│                                                                        │
│  Step 4: Verdict                                                       │
│  Delta = Score B - Score A                                             │
│  Pass if: Score B >= pass_threshold AND Delta >= minimum_delta         │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

Key principle: The judge never sees both outputs in the same call. This eliminates "halo effect" bias — the judge scores each output on its own merits, not by direct comparison.

Canonical Dimensions

These 5 dimensions are mandatory in every criteria.json. You can add custom dimensions beyond these, but you cannot remove any of the core 5.

Dimension	Default Description	Typical Weight
Correctness	Does the output fulfill the task requirements? Are all specified behaviors present and correct?	25-35
Skill Adherence	Did the agent follow the specific patterns, hard gates, and workflows defined in the skill?	20-30
Code Quality	Is the code clean, well-structured, free of smells, follows SRP, and avoids duplication?	15-25
Test Coverage	Are there meaningful tests? Do they test the right things? Are they following TDD/best practices?	10-20
Documentation	Is there adequate YARD documentation, clear intent, and helpful inline comments where needed?	5-15

Why these weights? Correctness and Skill Adherence are usually the highest because they directly measure "did the agent do the right thing" and "did the skill help." Test Coverage and Documentation are lower because they are supporting qualities.

criteria.json Format

{
  "context": "Evaluate whether the skill helps build a proper API REST collection",
  "dimensions": [
    { "name": "correctness", "max_score": 30 },
    { "name": "skill_adherence", "max_score": 25 },
    { "name": "code_quality", "max_score": 20 },
    { "name": "test_coverage", "max_score": 15 },
    { "name": "documentation", "max_score": 10 }
  ],
  "pass_threshold": 70,
  "minimum_delta": 10
}

Field-by-field breakdown:

Field	Type	Required	Description
context	string	Yes	Human-readable description of what this eval measures. Shown in the judge prompt.
dimensions	array	Yes	List of dimension objects. Must include all 5 canonical dimensions. Each needs name and max_score. max_score values must sum to exactly 100.
pass_threshold	integer	No	Minimum total context score (0-100) to pass. Default: 70.
minimum_delta	integer	No	Minimum total improvement (context - baseline) required to pass. Default: 10.

Rules:

1. Sum to 100: dimensions max_score values must sum to exactly 100. The engine rejects any eval where they don't.
2. All 5 core dimensions required: You cannot omit correctness, skill_adherence, code_quality, test_coverage, or documentation.
3. Custom dimensions allowed: You can add dimensions beyond the core 5. Their max_score values still count toward the 100 total.
4. Pass/fail logic: Both conditions must be true:
   • context_total >= pass_threshold (the agent with skill scored high enough)
   • total_delta >= minimum_delta (the skill made a meaningful difference)

Example with custom dimension descriptions:

{
  "context": "Evaluate REST API collection skill",
  "dimensions": [
    { "name": "correctness", "max_score": 30 },
    { "name": "skill_adherence", "max_score": 25, "description": "Did the agent use the `.call` pattern and return the standardized hash?" },
    { "name": "code_quality", "max_score": 20 },
    { "name": "test_coverage", "max_score": 15 },
    { "name": "documentation", "max_score": 10 }
  ],
  "pass_threshold": 70,
  "minimum_delta": 10
}

Example with a custom dimension (6 total, still summing to 100):

{
  "context": "Evaluate with performance considerations",
  "dimensions": [
    { "name": "correctness", "max_score": 25 },
    { "name": "skill_adherence", "max_score": 20 },
    { "name": "code_quality", "max_score": 15 },
    { "name": "test_coverage", "max_score": 15 },
    { "name": "documentation", "max_score": 10 },
    { "name": "performance", "max_score": 15, "description": "Is the solution performant? Are N+1 queries avoided?" }
  ],
  "pass_threshold": 70,
  "minimum_delta": 10
}

Understanding the Output

Human-readable format:

═══════════════════════════════════════════════════════
  Eval: my-first-eval
  Skill: my-service
  Provider: openai
  Tokens: 18432  |  Est. Cost: $0.0934
═══════════════════════════════════════════════════════

  === BASELINE ITERATIONS ===
  Step 1: Read task → Tool: read_file → Observation: content...
  Step 2: Plan changes → Tool: write_file → Observation: Success...
  Step 3: Run tests → Tool: run_command → Observation: 3 runs, 0 failures
  Step 4: Final answer

  === CONTEXT ITERATIONS ===
  Step 1: Read task → Tool: read_file → Observation: content...
  Step 2: Apply skill pattern → Tool: write_file, run_command → Observation: Success...
  Step 3: Final answer

  DIMENSION                BASELINE   CONTEXT    DELTA
  ──────────────────────── ───────── ───────── ───────
  Correctness (30)                12        28     +16
  Skill Adherence (25)             5        22     +17
  Code Quality (20)               10        16      +6
  Test Coverage (15)               3        13     +10
  Documentation (10)               2         8      +6
  ──────────────────────── ───────── ───────── ───────
  TOTAL                          32/100    87/100   +55

  TREND: baseline ↑ (+2), context ↑ (+7)
  VERDICT: PASS (threshold: 70, minimum delta: 10)
═══════════════════════════════════════════════════════

  === WHAT WENT WELL ===
  Correctness (28/30, baseline: 12/30)
    The agent correctly implemented all required behaviors.
  Skill Adherence (22/25, baseline: 5/25)
    Followed the service object pattern and hard gates.

  === WHAT WENT WRONG ===
  Test Coverage (13/15, baseline: 3/15)
    Tests exist but edge cases are missing.
    Advice: Are there meaningful tests? Do they test the right things?

What each column means:

• BASELINE: The agent's score without the skill. This is the "unaided" performance.
• CONTEXT: The agent's score with the skill. This is the "aided" performance.
• DELTA: CONTEXT - BASELINE. How much the skill helped.
• TOTAL: Sum of all dimension scores. Max possible is 100.
• TREND: Comparison against the previous run of the same eval + skill (from .skill-bench-trends.json). Shows whether scores are improving over time.
• VERDICT: PASS only if CONTEXT >= pass_threshold AND DELTA >= minimum_delta.
• Tokens / Est. Cost: The header shows total tokens used across the run and an estimated USD cost as Tokens: N | Est. Cost: $X.XXXX. The cost is approximate — it comes from a built-in per-model price table (Services::CostCalculator) and shows — when the model isn't in that table. JSON output (--format json) exposes the same data as top-level tokens and cost fields.

Iteration timeline:

Each run (baseline and context) shows the ReAct loop steps the agent took: thinking, calling tools, and observing results. This helps you understand how the agent worked through the task. Observations are truncated to keep the output readable. If the timeline is empty, the agent finished in a single LLM call without using tools.

Feedback sections:

• WHAT WENT WELL — Dimensions where the context score is ≥ 80% of the max, with the judge's reasoning. These are the strengths of your skill.
• WHAT WENT WRONG — Dimensions where the context score is < 80% of the max, with the judge's reasoning and the baseline score for comparison. These are where your skill needs work.
• ADVICE — Each low-scoring dimension shows its description from criteria.json as actionable guidance. If the description is empty, no advice line appears.

Verdict Decision Matrix

Your eval result depends on both conditions. Here is every scenario:

Context Score	Delta	Verdict	Why
87	+55	PASS	Context >= 70 and delta >= 10. The skill helped a lot.
87	-2	FAIL	Context >= 70 but delta < 10. The skill made things worse.
65	+15	FAIL	Context < 70 even though delta >= 10. Absolute score too low.
65	+5	FAIL	Context < 70 and delta < 10. Both conditions failed.

Negative delta means the skill hurt performance. If baseline=89 and context=87, your skill confused the agent or added noise. This is the most common "unexpected FAIL" — the skill reads well to humans but backfires with the LLM.

FAIL example — skill made things worse:

  DIMENSION                BASELINE   CONTEXT    DELTA
  ──────────────────────── ───────── ───────── ───────
  Correctness (30)                28        25      -3
  Skill Adherence (25)            23        22      -1
  Code Quality (20)               18        18      +0
  Test Coverage (15)              12        13      +1
  Documentation (10)               8         9      +1
  ──────────────────────── ───────── ───────── ───────
  TOTAL                           89/100    87/100    -2

  VERDICT: FAIL (threshold: 70, minimum delta: 10)

Why this FAILs: Context score (87) is above the threshold (70), but the delta is negative (-2). The agent scored 89 without the skill and only 87 with it. The skill actively hurt performance. Common causes:

• Skill is too long or contradictory — the agent ignores the task to follow the skill
• Skill prescribes patterns that conflict with the task requirements
• Skill adds boilerplate that the judge penalizes (over-engineering)

Fix: Remove rules that don't directly improve the dimension with the lowest delta. Shorter skills usually beat longer ones.

---

Reliability & Security

• Allowlist-Gated Execution: The agent can only run commands you add to allowed_commands; with an empty allowlist it can run nothing. Commands run inside a temporary git sandbox directory (a copy of the eval files) on the host — container isolation is not yet shipped, so host execution is disabled by default and must be explicitly opted into with allow_host_execution: true.
• Command Blocklist: Dangerous commands (bash, sh, python, curl, etc.) are always blocked, even if listed in allowed_commands.
• Path Validation: Eval paths are validated to prevent directory traversal attacks.
• Atomic History Writes: Benchmark history uses file locking to prevent corruption from concurrent writes.
• URL Sanitization: All provider URL parameters are CGI-escaped to prevent injection.
• YAML Safety: Config loading uses permitted_classes: [] to prevent symbol DoS attacks.
• Traceability: Every thought and tool call is logged with full backtrace for post-mortem analysis.
• Robust Error Recovery: Handles provider outages and rate limits gracefully with standardized error logging.
• XML-Safe Output: JUnit XML output is properly escaped to prevent injection attacks.
• Test Coverage: 700+ tests covering core engine, CLI commands, and all provider clients. Run bundle exec rake test to see the current count.

Testing

The project uses Minitest with WebMock for HTTP stubbing.

# Run all tests
bundle exec rake test

# Run with coverage
bundle exec rake test COVERAGE=true

# Run specific test file
bundle exec ruby -Itest test/integration_test.rb

Test Structure:

• test/agent/ — Agent runtime tests
• test/agent_eval/ — Agent evaluation tests
• test/cli/ — CLI command tests
• test/clients/ — Provider client tests
• test/evaluator/ — Core evaluation engine tests
• test/history_recorder/ — Benchmark history persistence tests
• test/models/ — Domain model tests
• test/registry/ — Skill/eval registry tests
• test/services/ — Service layer tests
• test/skills/ — Skill loading tests
• test/tools/ — Agent tool tests
• Plus several top-level test/*_test.rb files (e.g. integration_test.rb, evaluation_runner_test.rb, trend_tracker_test.rb).

---

Security

Threat Model

Ruby Skill Bench is designed with security as a primary concern. The system executes AI agents in isolated environments and must protect against various attack vectors:

• Path Traversal: Preventing agents from accessing files outside the sandbox
• Command Injection: Preventing execution of arbitrary shell commands
• Resource Exhaustion: Preventing denial-of-service through resource consumption
• Information Leakage: Protecting sensitive data like API keys

Security Features

Path Traversal Protection

• Symlink Validation: All symlinks are validated to ensure they don't escape the sandbox
• TOCTOU Mitigation: Path validation is re-checked after directory creation operations
• Path Normalization: All paths are normalized and validated against working directory boundaries
• Character Validation: Paths are validated against strict character patterns

Command Execution Security

• Command Allowlist: Only explicitly allowed commands can be executed
• Dangerous Commands Blocklist: Dangerous commands (bash, curl, sudo, etc.) are always blocked
• Shell Tokenization: Commands are tokenized before execution to prevent shell injection
• Fail-Closed Host Execution: Container isolation is not yet active, so commands run on the host inside a temporary sandbox directory. To match this reality, run_command refuses to execute unless allow_host_execution: true is set; it is disabled by default.

The allowlist is the only real authorization control — and it only checks the base command. run_command authorizes by the first token of the command (rake, find, git, …); it does not inspect arguments. Shell tokenization stops metacharacter injection, but it does not sandbox what an allowlisted binary can do. Because many common tools are general-purpose execution wrappers, allowlisting any one of them is equivalent to granting arbitrary host code execution — for example rake -e '...', rspec -e, make (arbitrary recipes), find . -exec ..., or git (hooks, -c core.fsmonitor=..., ! ... aliases). Combined with the fail-closed model above (run_command refuses to run on the host unless allow_host_execution is explicitly enabled — see HOST_EXECUTION_REFUSED in run_command.rb), the practical guidance is: keep allowed_commands as minimal as possible — empty for untrusted skills — and treat every entry as if you were handing the skill a shell.

An optional, default-off command_argument_constraints setting can refuse commands whose arguments contain configured substrings/flags (for example blocking -e or -exec). It is a defense-in-depth speed bump, not a sandbox, and is unset by default; the allowlist remains the control that matters.

Docker Security Hardening (Planned — Not Yet Active)

Status: The container isolation model described below is planned, not shipped. No Docker build context is packaged, so containers are never launched today — run_command runs on the host gated by the allowlist and allow_host_execution. The settings below document the intended hardened model for when container isolation lands.

When container isolation is enabled in a future release, containers are intended to launch with hardened security settings:

• Non-root User: Containers run as a non-root user
• Privilege Prevention: --security-opt no-new-privileges prevents privilege escalation
• Capability Dropping: All Linux capabilities are dropped except minimal needed ones
• Network Isolation: --network none disables network access
• Read-only Root: Container filesystem is read-only (except for mounted volumes)

Resource Limits

• File Size Limits: Individual files in context hydration are limited to 50KB
• Total Context Size: Total context size is limited to 1MB to prevent memory exhaustion
• Execution Timeout: Commands are limited to a configurable timeout (default: 30 seconds)
• Max Iterations: Agent loops are limited to prevent infinite loops

API Key Security

• Environment Variables: API keys are loaded from environment variables, not hardcoded
• Configuration Hierarchy: Keys can be set in skill-bench.json or environment variables
• No Logging: API keys are never logged or exposed in error messages
• Provider-Specific Keys: Each provider uses its own API key configuration

Best Practices for Users

1. Never Commit API Keys: Never commit skill-bench.json with API keys to version control
2. Use Environment Variables: Prefer environment variables for sensitive configuration
3. Minimal Command Allowlist: Only allow commands necessary for your evals
4. Regular Updates: Keep dependencies updated to patch security vulnerabilities
5. Review Changes: Review skill files before execution to ensure they don't contain malicious code

Reporting Security Issues

To report a security vulnerability, please follow the process in SECURITY.md. Do not open a public issue — use GitHub's private vulnerability reporting (Security tab) or email the maintainer at ismael.marin@gmail.com.

---

Troubleshooting

Common Issues and Solutions

Configuration Issues

Problem: "Config load failed, using mock provider"

• Solution: Ensure your skill-bench.json file is properly formatted JSON and contains required fields
• Check: Verify the file exists in your project root or home directory

Problem: "API Key not set for [Provider]"

• Solution: Set the appropriate environment variable (e.g., SKILL_BENCH_OPENAI_API_KEY) or add it to your skill-bench.json
• Check: Run env | grep SKILL_BENCH to verify environment variables are set

Problem: "No allowed commands configured"

• Solution: Add allowed_commands array to your skill-bench.json with the commands you want to allow
• Check: Ensure commands are in the allowlist and not in the dangerous commands list

Execution Issues

Problem: "Command execution timed out"

• Solution: Increase max_execution_time in your skill-bench.json or simplify the task
• Check: Verify the command isn't hanging or waiting for input

Problem: "Command execution refused: no sandbox isolation is active and 'allow_host_execution' is not enabled"

• Cause: Container isolation is not yet shipped, so commands would run on the host. SkillBench fails closed by default rather than run un-isolated.
• Solution: Set "allow_host_execution": true in skill-bench.json to permit allowed commands to run directly on the host (inside the temporary sandbox directory). Enable it only when you accept that trade-off.

Problem: "Context hydration failed"

• Solution: Verify the source path exists and is a directory
• Check: Ensure the path is within the base directory and file sizes are under limits

Network Issues

Problem: "Network Error: Connection refused"

• Solution: Check your internet connection and API provider status
• Check: Verify the base URL in your configuration is correct

Problem: "API Request failed: 429"

• Solution: This is a rate limit error. The system will retry automatically
• Check: Reduce request frequency or check your API quota

Test Failures

Problem: Tests fail with "WebMock::NetConnectNotAllowedError"

• Solution: This occurs when tests try to make real HTTP requests. Ensure test stubs are properly configured
• Check: Verify WebMock is properly stubbing the expected URLs

Problem: "E2E sibling repositories not present"

• Solution: This is expected if you don't have the agent-mcp-runtime repository cloned
• Check: These tests will be skipped and won't affect the overall test results

Debug Mode

For detailed debugging, you can enable verbose logging:

# Set environment variable for verbose logging
export SKILL_BENCH_DEBUG=true
skill-bench run my-eval --skill=my-skill

Getting Help

If you encounter issues not covered here:

1. Check the GitHub Issues for similar problems
2. Create a new issue with detailed information about your environment and the problem
3. Include Ruby version, SkillBench version, and error messages
4. Provide steps to reproduce the issue

---

CI/CD Integration

Batch Runs

Run every eval at once instead of one at a time:

# Every eval under the default evals/ directory
skill-bench run --all --skill=my-service

# Or point at a specific directory
skill-bench run --evals-dir path/to/evals --skill=my-service

A batch run exits 0 only when every eval passes and non-zero if any fail, so the process exit code is itself a CI gate. Two formats are built for batch consumption:

• --summary emits an aggregate JSON gate — passed / failed / total counts, summed tokens and cost, and the worst_delta eval (the smallest context-minus-baseline delta in the batch). Archive it as a single machine-readable artifact:

  skill-bench run --all --skill=my-service --summary
  

• --format junit aggregates the batch into one JUnit document with one <testcase> per eval (a <failure> child for each failing eval), so test reporters show per-eval results:

  skill-bench run --all --skill=my-service --format junit > junit.xml
  

GitHub Action

Downstream repos can gate a skill change on every push or PR with the bundled composite action. Add a step that references igmarin/ruby-skill-bench@v1:

# .github/workflows/skill-bench.yml
name: skill-bench
on: [pull_request]

jobs:
  skill-bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: igmarin/ruby-skill-bench@v1
        with:
          evals-dir: evals          # directory scanned for evals (default: evals)
          skill: skills/my-service  # skill applied to every eval (default: "")
          format: junit             # human | json | junit | html (default: junit)
          ruby-version: "3.3"       # Ruby for ruby/setup-ruby (default: 3.3)
          args: --summary           # extra flags appended verbatim (e.g. --summary, --pack NAME)

The action installs the gem and runs skill-bench run --all --evals-dir <evals-dir> --format <format> (adding --skill when set and appending args verbatim). The run step's exit code is the gate. For a full copy-paste workflow template, see examples/ci/.

The gem's own repository CI (.github/workflows/ci.yml) runs the test suite — rubocop, reek, and minitest against Ruby 3.3 and 3.4, on push and pull requests — and is separate from the reusable action above.

To preview the machine-readable output locally:

skill-bench run my-eval --skill=my-skill --format json

---

License

The gem is available as open source under the terms of the MIT License.