Class: LLMExperiment::CLI::MetricsCommand

Inherits:
Command
  • Object
show all
Defined in:
lib/llm_experiment/cli/metrics_command.rb

Constant Summary collapse

NAME = 
"metrics"
SUMMARY = 
"medians and exact Mann-Whitney U, never pooled"

Instance Method Summary collapse

Methods inherited from Command

#call, #initialize, summary

Constructor Details

This class inherits a constructor from LLMExperiment::CLI::Command

Instance Method Details

#add_options(parser) ⇒ Object 



9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
# File 'lib/llm_experiment/cli/metrics_command.rb', line 9

def add_options(parser)
  parser.banner = <<~BANNER
    Usage: llmx metrics [options]

    Reads every metrics.json written by `llmx parse` and reports one cell
    per app and agent. Nothing is pooled across those: two agents do not
    count tool calls the same way, and two apps are different sizes of
    haystack, so a number averaged over them measures nothing.

    p-values are exact. With a handful of tasks per cell the normal
    approximation to the Mann-Whitney U is not trustworthy, so every
    split of the pooled values is enumerated instead.

  BANNER
end

#run(_argv) ⇒ Object 



25
26
27
 
# File 'lib/llm_experiment/cli/metrics_command.rb', line 25

def run(_argv)
  MetricsReport.new(experiment: experiment).render
end