Class: Ask::Eval::Experiment

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/eval/experiment.rb

Overview

Runs a Dataset against a single variant and compares runs.

An experiment executes every dataset item through a runner (a callable that turns an input into the agent's output), optionally scores each output, and collects per-item results. Running the same dataset with two different runners (a changed prompt, a different model) and comparing the experiments is the A/B-testing loop of eval-driven development.

Examples:

run_a = dataset.experiment(runner: ->(input) { agent(input, prompt: old) },
                           scorer: judge_scorer)
run_b = dataset.experiment(runner: ->(input) { agent(input, prompt: new) },
                           scorer: judge_scorer)
run_a.run
run_a.summary          # => {total:, passed:, failed:, avg_score:, ...}
run_a.compare(run_b)   # => per-item side-by-side + aggregate verdict

Defined Under Namespace

Classes: Result

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(dataset, runner:, scorer: nil) ⇒ Experiment

Returns a new instance of Experiment.

Parameters:

  • dataset (Ask::Eval::Dataset)
  • runner (Proc) —

    called with the item's input; returns the agent's output string

  • scorer (Proc, nil) (defaults to: nil) —

    called with (input:, output:, expected:); returns a score (0..1) or nil to skip scoring



37
38
39
40
41
42
 
# File 'lib/ask/eval/experiment.rb', line 37

def initialize(dataset, runner:, scorer: nil)
  @dataset = dataset
  @runner = runner
  @scorer = scorer
  @results = nil
end

Instance Attribute Details

#dataset ⇒ Ask::Eval::Dataset (readonly)

Returns:



45
46
47
 
# File 'lib/ask/eval/experiment.rb', line 45

def dataset
  @dataset
end

Instance Method Details

#compare(other) ⇒ Hash

Side-by-side comparison with another experiment over the same (or overlapping) dataset items.

Parameters:

Returns:

  • (Hash) —

    [{item_id:, input:, a:, b:, delta:], a: summary, b: summary, verdict: "a"|"b"|"tie"}



97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
 
# File 'lib/ask/eval/experiment.rb', line 97

def compare(other)
  a = results
  b = other.results
  ha = a.to_h { |r| [r.item.id, r] }
  hb = b.to_h { |r| [r.item.id, r] }

  deltas = (ha.keys & hb.keys).map do |id|
    ra = ha[id]
    rb = hb[id]
    {
      item_id: id,
      input: ra.item.input,
      a: { output: ra.output, score: ra.score, error: ra.error },
      b: { output: rb.output, score: rb.score, error: rb.error },
      delta: (ra.score && rb.score) ? (rb.score - ra.score).round(3) : nil
    }
  end

  { deltas: deltas, a: summary, b: other.summary, verdict: verdict(summary, other.summary) }
end

#results ⇒ Array<Result>

Returns per-item results (runs first if needed).

Returns:

  • (Array<Result>) —

    per-item results (runs first if needed)



74
75
76
 
# File 'lib/ask/eval/experiment.rb', line 74

def results
  @results || run.results
end

#run ⇒ self

Execute every dataset item through the runner. A runner error is recorded on that item and the run continues.

Returns:

  • (self) 


51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/ask/eval/experiment.rb', line 51

def run
  @results = @dataset.items.map do |item|
    start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
    output = nil
    error = nil
    begin
      output = @runner.call(item.input).to_s
    rescue StandardError => e
      error = e.message
    end

    score = nil
    if error.nil? && @scorer
      score = @scorer.call(input: item.input, output: output, expected: item.expected)
    end

    duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000).round
    Result.new(item: item, output: output, score: score, error: error, duration_ms: duration_ms)
  end
  self
end

#summary ⇒ Hash

Returns aggregate statistics.

Returns:

  • (Hash) —

    aggregate statistics



79
80
81
82
83
84
85
86
87
88
89
 
# File 'lib/ask/eval/experiment.rb', line 79

def summary
  rs = results
  scores = rs.filter_map(&:score)
  {
    total: rs.size,
    passed: rs.count(&:passed?),
    failed: rs.count { |r| !r.passed? },
    avg_score: scores.empty? ? nil : (scores.sum.fdiv(scores.size)).round(3),
    total_duration_ms: rs.sum(&:duration_ms)
  }
end