Class: LLMExperiment::MetricsReport

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_experiment/metrics_report.rb

Overview

Aggregates trial metrics and compares the conditions pairwise.

Never pool across agents or apps. Claude and Codex do not count tool calls the same way, and two subject apps are different sizes of haystack. A number averaged over those is not a measurement of anything. So the rows are grouped by [app, agent] and each cell is reported on its own.

Constant Summary collapse

METRICS = 
%w[
  wall_seconds
  output_tokens
  input_tokens
  cache_creation_input_tokens
  total_tool_calls
  context_before_first_tool_call
  tool_calls_to_first_defect_read
].freeze

Instance Method Summary collapse

Constructor Details

#initialize(experiment:) ⇒ MetricsReport

Returns a new instance of MetricsReport.



23
24
25
 
# File 'lib/llm_experiment/metrics_report.rb', line 23

def initialize(experiment:)
  @experiment = experiment
end

Instance Method Details

#condition_pairs ⇒ Object 



50
 
# File 'lib/llm_experiment/metrics_report.rb', line 50

def condition_pairs = @experiment.conditions.combination(2).to_a

#conditions ⇒ Object 



48
 
# File 'lib/llm_experiment/metrics_report.rb', line 48

def conditions = @experiment.conditions

#files ⇒ Object

One tree, never both. results-raw/ holds every trial; results/ holds the published subset, and for a public app those are the same files copied. A glob of "results*" matched both, so every published trial was counted twice -- which does not merely inflate n, it duplicates each observation, halves the apparent variance and pushes the exact p-values toward significance. Prefer the raw tree, which is always complete, and fall back to the published one so a fresh clone without results-raw/ still computes.



34
35
36
37
38
39
 
# File 'lib/llm_experiment/metrics_report.rb', line 34

def files
  @files ||= begin
    raw = Dir.glob(File.join(@experiment.results_raw_dir, "**", "metrics.json"))
    raw.empty? ? Dir.glob(File.join(@experiment.results_dir, "**", "metrics.json")) : raw
  end
end

#render(io = $stdout) ⇒ Object

Raises:



52
53
54
55
56
57
58
59
60
 
# File 'lib/llm_experiment/metrics_report.rb', line 52

def render(io = $stdout)
  raise Error, "no metrics.json found in #{@experiment.root}. Run trials, then `llmx parse --all`" if files.empty?

  io.puts "#{rows.size} trial(s) from #{files.size} file(s)"
  report_undeclared_conditions(io)
  io.puts
  cells.each { |(app, agent), cell| render_cell(io, app, agent, cell) }
  render_fix_verified(io)
end

#rows ⇒ Object

A trial whose task never reproduced measured nothing: the agent was asked to fix something that was not broken.



43
44
45
46
 
# File 'lib/llm_experiment/metrics_report.rb', line 43

def rows
  @rows ||= files.sort.map { |f| JSON.parse(File.read(f)) }
                 .reject { |r| r["task_reproduces"] == false }
end