ask-eval

LLM evaluation framework for Ruby. Minitest-native assertions for testing LLM outputs: deterministic checks plus LLM-as-judge for faithfulness, hallucination, bias, toxicity, and correctness. Includes session-level evaluation for agents, regression recording, and CI-native reporters.

Installation

gem "ask-eval"

Quick Start

require "ask/eval"
require "ask/eval/dsl"

class MyEvalTest < Minitest::Test
  include Ask::Eval::DSL

  test "response is faithful to context" do
    response = my_rag_app.query("What's the return policy?")
    assert_faithful response, context: [my_docs]
  end

  test "response contains expected info" do
    response = my_app.generate_email("Order confirmation")
    assert_contains response, "Thank you for your order"
    assert_regex response, /order #\d{5}/
  end
end

Include Ask::Eval::DSL in test classes, or require ask/eval/minitest in test_helper.rb to get the assertions in every test automatically.

Assertions

Deterministic: assert_contains, assert_not_contains, assert_regex, assert_json, assert_max_tokens, assert_starts_with, assert_ends_with, assert_equals, assert_min_length, assert_max_length, assert_url, assert_email.

LLM-as-judge: assert_faithful(output, context:), assert_not_hallucinating(output, context:), refute_bias, refute_toxicity, assert_correctness(output, expected:).

Judges need a model. Pass one per assertion (model:) or configure a default:

Ask::Eval.configure do |c|
  c.default_judge = "openai/gpt-4o-mini"  # any callable, Ask::Provider, or model string
end

Agent Evaluation

Evaluate an Ask::Agent::Session with the eval_session DSL:

test "agent behavior" do
  eval_session(model: "gpt-4o", tools: [Bash]) do |r|
    r.run("Check health")
    assert_tool_called "bash"
    assert_cost_under 0.01
  end
end

eval_session yields an Ask::Eval::SessionEval exposing run(prompt), tool_called?(name), tool_names, total_cost, and last_response.

Interactions are recorded on first run and replayed instead of calling the LLM when ASK_EVAL_MODE=replay is set, so regression tests run without a model or API keys.

Reporters and Custom Judges

Reporters consume Ask::Eval::Runner results: Ask::Eval::Reporters::Console (dev), JUnit (Jenkins, CircleCI, GitLab CI), and GitHub (::warning and ::error annotations for pull requests).

Create your own judge by subclassing Ask::Eval::Judge and implementing call, system_prompt, and user_message; no registration needed.

Full documentation

The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs. https://ask-rb.github.io/ask-docs/production/evaluation covers ask-eval in depth, including custom judges, cost tracking, and CI integration. API reference: https://ask-rb.github.io/ask-docs/reference/api.

Development

bundle install bundle exec rake test

License

MIT