Class: LLMExperiment::Trial

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_experiment/trial.rb

Overview

One trial: one task, one agent, one condition, in a fresh container.

On caps: the trial runs to completion under a wall-clock timeout rather than being killed at the first edit. Stopping at the first edit would measure the cost of locating the defect but throw away whether the fix was any good, and an analysis cap can always be applied afterwards from the event log, whereas a killed trial cannot be un-killed. The transcript records the index of the first defect-file read and of the first edit, so a "cost to locate" figure survives without destroying anything.

Results land in results-raw/, which is gitignored. Nothing reaches results/ until the sanitize gate has looked at it.

Constant Summary collapse

LOGIN_CHECKS =

Both agents read the mounted credential store, so this asks a dead login the same question a trial would.

{
  "claude" => ["claude auth status 2>&1", '"loggedIn": false'],
  "codex" => ["codex login status 2>&1", "Not logged in"]
}.freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(experiment:, task_id:, agent:, condition:, container: nil, timeout: nil, shell: Shell) ⇒ Trial

Returns a new instance of Trial.



30
31
32
33
34
35
36
37
38
39
 
# File 'lib/llm_experiment/trial.rb', line 30

def initialize(experiment:, task_id:, agent:, condition:, container: nil,
               timeout: nil, shell: Shell)
  @experiment = experiment
  @task_id = task_id.to_s
  @agent = agent.to_s
  @condition = condition.to_s
  @container = container
  @timeout = timeout
  @shell = shell
end

Instance Attribute Details

#agent ⇒ Object (readonly)

Returns the value of attribute agent.



28
29
30
 
# File 'lib/llm_experiment/trial.rb', line 28

def agent
  @agent
end

#condition ⇒ Object (readonly)

Returns the value of attribute condition.



28
29
30
 
# File 'lib/llm_experiment/trial.rb', line 28

def condition
  @condition
end

#experiment ⇒ Object (readonly)

Returns the value of attribute experiment.



28
29
30
 
# File 'lib/llm_experiment/trial.rb', line 28

def experiment
  @experiment
end

#task_id ⇒ Object (readonly)

Returns the value of attribute task_id.



28
29
30
 
# File 'lib/llm_experiment/trial.rb', line 28

def task_id
  @task_id
end

Instance Method Details

#app ⇒ Object 



42
 
# File 'lib/llm_experiment/trial.rb', line 42

def app = @app ||= experiment.app(task.app)

#env ⇒ Object

Pure: reads configuration and the prompt file, touches nothing else. The container never sees anything that is not in here.



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/llm_experiment/trial.rb', line 48

def env
  validate!
  values = {
    "LLMX_TASK_ID" => task.id,
    "LLMX_APP" => app.key,
    "LLMX_AGENT" => agent,
    "LLMX_CONDITION" => condition,
    "LLMX_BRANCH" => task.branch,
    "LLMX_TEST_FILE" => task.test_file,
    "LLMX_IMPL_FILES" => task.impl_files.join(","),
    "LLMX_DB_PREPARE" => app.db_prepare,
    "LLMX_DATABASE" => app.database,
    "LLMX_TEST_COMMAND" => app.test_command,
    "LLMX_SUITE_RAN_PATTERN" => app.suite_ran_pattern,
    "LLMX_TIMEOUT_SECONDS" => timeout_seconds.to_s,
    # Base64 keeps the prompt out of shell quoting entirely: newlines,
    # backticks and quotes in a prompt must not become shell syntax.
    # pack("m0") is strict_encode64 without needing the base64 gem, which
    # Ruby 3.4 unbundled -- and the runner decodes it with unpack1("m0").
    "LLMX_PROMPT_B64" => [prompt].pack("m0")
  }
  model = experiment.agents.dig(agent, "model")
  values["LLMX_MODEL"] = model if model
  values
end

#prompt_path ⇒ Object 



44
 
# File 'lib/llm_experiment/trial.rb', line 44

def prompt_path = experiment.prompt_path(task, condition)

#run(dry_run: false) ⇒ Object 



74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/llm_experiment/trial.rb', line 74

def run(dry_run: false)
  environment = env
  return dry_run_report(environment) if dry_run

  preflight!
  out_dir = prepare_results_dir
  argv = run_argv(out_dir, environment)

  puts "trial #{task_id} / #{agent} / #{condition} -> #{out_dir}"
  started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  ok = @shell.sh(*argv, allow_failure: true,
                        log_as: run_argv(out_dir, redacted(environment)).shelljoin)
  elapsed = (Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).round(1)

  report(record_meta(out_dir, elapsed, ok), elapsed)
  out_dir
end

#task ⇒ Object 



41
 
# File 'lib/llm_experiment/trial.rb', line 41

def task = @task ||= experiment.task(task_id)

#timeout_seconds ⇒ Object 



43
 
# File 'lib/llm_experiment/trial.rb', line 43

def timeout_seconds = (@timeout || experiment.timeout_seconds).to_i