AISpec

RSpec for AI behavior, not AI outputs.

AISpec is the open-source engineering framework for defining, executing, and enforcing behavioral contracts for LLM applications in Ruby and Ruby on Rails.

Existing LLM evaluation tools tell you whether your AI performed well on a given run. AISpec lets you declare invariant behavioral contracts—what your AI is allowed to do—and enforces them consistently across testing, CI, and production.

Key Features

Behavioral Invariants, Not Static Text Assertions: Test constraints like latency budgets, cost bounds, JSON schemas, system prompt leak protection, tool invocation ordering, and LLM-as-a-judge criteria.
Provider Agnostic: Out-of-the-box support for OpenAI, Anthropic, Ollama (local, zero cost), OpenRouter, and custom provider adapters.
Statistical Reliability: Statistical confidence bounds (Wilson score 95% CI), score variance ratings, and automated flaky detection.
Rails First: Clean RSpec integration, Rails initializers, custom plugin DSL, and automated CI reporting.
Multiple Reporting Formats: Terminal Console, JSON, GitHub Markdown tables, HTML dashboards, and JUnit XML.

Installation

Standalone CLI Installation

gem install aispec

In a Ruby Project

Add aispec to your Gemfile:

gem "aispec", "~> 0.1.0"

And execute:

bundle install

Quick Start (CLI)

1. Initialize Project Structure

Run aispec init in your project root to generate starter contract specifications and test datasets:

aispec init

This creates:

contracts/support.yml
datasets/support.yml

2. Define a Behavioral Contract

Edit contracts/support.yml:

version: 1
name: support-bot

model:
  provider: openai
  model: gpt-4o

dataset:
  path: datasets/support.yml

contracts:
  - type: contains
    value: refund

  - type: json

  - type: citation_format

  - type: max_latency
    value: 2500

  - type: cost
    max: 0.03

  - type: judge
    prompt: |
      The answer should be empathetic, technically correct, and concise.
    threshold: 0.85

You can also use shorthand string syntax:

contracts:
  - must_return_valid_json
  - must_cite_sources
  - latency < 2500ms
  - cost < $0.03
  - confidence >= 0.85

3. Execute Contract Verification

aispec run contracts/support.yml

Output:

Running 12 contracts for support-bot (openai / gpt-4o)...

Case #1 (Input: "How do I request a refund?")
  PASS  contains: Output contains 'refund'
  PASS  json: Output is valid JSON
  PASS  citation_format: Response contains valid citation formatting
  PASS  max_latency: Latency 124.0ms <= 2500ms
  PASS  cost: Cost $0.0012 <= $0.0300
  PASS  judge: Judge score 0.92 (threshold 0.85): Response is accurate, empathetic, and concise.

Overall
  Passed: 12/12 (100.0%)
  Confidence Interval: ±0.0%
  Mean Latency: 124.0ms
  Total Cost: $0.0024
  Average Judge Score: 0.92
  Variance: Low
  Recommendation: Stable

Comparing Models

Benchmarking multiple models on cost, latency, and behavioral score:

aispec compare gpt-4o claude-3-5-sonnet llama3

Model	Score	Cost	Latency
gpt-4o	98.2%	$0.0024	1.1 s
claude-3-5-sonnet	97.5%	$0.0042	1.4 s
llama3	91.0%	$0.0000	0.8 s

Integrating with Ruby on Rails

1. Add Gem to Rails `Gemfile`

Add aispec to your :development and :test groups:

# Gemfile
group :development, :test do
  gem "aispec"
end

2. Rails Initializer (`config/initializers/aispec.rb`)

Configure default providers, judge models, and custom plugin assertions in Rails:

# config/initializers/aispec.rb
AISpec.configure do |config|
  config.default_provider = ENV.fetch("AISPEC_PROVIDER", "openai")
  config.default_model    = "gpt-4o"
  config.judge_provider   = "openai"
  config.judge_model       = "gpt-4o"
  config.timeout           = 30
end

# Register custom business behavioral assertions
AISpec.plugin do
  assertion :valid_tenant_id do |response, context, _options|
    tenant_id = context[:tenant_id]
    passed = response[:output].include?(tenant_id.to_s)
    [passed, passed ? "Output contains tenant ID #{tenant_id}" : "Output missing tenant ID #{tenant_id}"]
  end
end

3. Running AISpec Contracts inside RSpec

Place your contracts under spec/contracts/ and create an RSpec spec file:

# spec/contracts/customer_support_contract_spec.rb
require "rails_helper"

RSpec.describe "Customer Support LLM Behavioral Contract", type: :contract do
  it "satisfies all safety, format, and latency invariant constraints" do
    contract_path = Rails.root.join("spec/contracts/support.yml")
    runner = AISpec::Core::Runner.new(contract_path)
    
    results = runner.run
    stats = results[:statistics].summary

    expect(stats[:failed_assertions]).to eq(0), "Contract failures:\n#{results[:runs].flat_map { |r| r[:assertion_results].reject(&:passed?).map(&:message) }.join("\n")}"
    expect(stats[:success_rate]).to be >= 95.0
    expect(stats[:mean_latency]).to be <= 2500.0
  end
end

Assertion Reference

Deterministic Assertions

Assertion	Syntax	Options / Description
`contains`	`{ type: contains, value: "refund" }`	Output contains exact substring
`not_contains`	`{ type: not_contains, value: "secret" }`	Output does not contain forbidden substring
`starts_with`	`{ type: starts_with, value: "Dear" }`	Output begins with specified string
`ends_with`	`{ type: ends_with, value: "Regards" }`	Output ends with specified string
`regex`	`{ type: regex, pattern: "^[A-Z0-9]+" }`	Output matches regex pattern
`json`	`{ type: json, schema: ["status", "data"] }`	Validates JSON format & required schema keys
`valid_markdown`	`{ type: valid_markdown }`	Validates structural Markdown syntax
`valid_xml`	`{ type: valid_xml }`	Validates XML document tags
`valid_sql`	`{ type: valid_sql }`	Validates SQL statement syntax
`max_latency`	`{ type: max_latency, value: 2500 }`	Execution time limit in milliseconds
`cost`	`{ type: cost, max: 0.03 }`	Estimated cost limit in USD
`token_count`	`{ type: token_count, max: 500 }`	Maximum total tokens allowed
`required_words`	`{ type: required_words, words: ["shipping", "policy"] }`	Ensures key vocabulary is present
`forbidden_words`	`{ type: forbidden_words, words: ["guarantee", "lawsuit"] }`	Ensures forbidden words are absent
`citation_format`	`{ type: citation_format }`	Verifies source citation structure
`must_not_reveal_system_prompt`	`{ type: must_not_reveal_system_prompt }`	Protects system prompt instructions from leaking
`tool_called`	`{ type: tool_called, value: "process_refund" }`	Verifies specific tool was invoked
`tool_not_called`	`{ type: tool_not_called, value: "delete_account" }`	Verifies tool was not invoked
`tool_order`	`{ type: tool_order, order: ["search", "checkout"] }`	Verifies exact sequence of tool calls

Probabilistic / Judge Assertions

Assertion	Syntax	Description
`judge`	`{ type: judge, prompt: "Is empathetic", threshold: 0.85 }`	LLM-as-a-judge criteria scoring (0.0 to 1.0)
`confidence`	`{ type: confidence, min: 0.80 }`	Evaluates output certainty & factual grounding

Continuous Integration (GitHub Actions)

Add AISpec verification to your .github/workflows/ci.yml:

name: AISpec Behavioral Verification

on: [push, pull_request]

jobs:
  contract-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.3'
          bundler-cache: true

      - name: Run AISpec Behavioral Contracts
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          bundle exec aispec run contracts/support.yml --format markdown --output pr_comment.md

      - name: Post PR Comment
        if: github.event_name == 'pull_request'
        uses: marooned/actions-comment-pull-request@v2
        with:
          filePath: pr_comment.md

License

The gem is available as open source under the terms of the MIT License.