Class: LLMExperiment::Grid

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_experiment/grid.rb

Overview

The grid of trials: every task x every agent x every condition.

Order is load-bearing.

Blocked by task, because the comparison is within a task: all of one task's cells run close together, so a slow patch of machine time lands on every condition of that task rather than on one of them.

Condition order rotates with the task index, so no condition always runs first and none always runs last.

Sequential on purpose. Two trials at once share the machine and contaminate wall_seconds — that happened in the source experiment and made a set of timings unusable. Never add parallelism here.

Resumable, because a cell that already has a meta.json is done and is skipped, and because one bad cell must not cost the other 89: a failure is recorded and the run carries on.

Defined Under Namespace

Classes: Result

Instance Method Summary collapse

Constructor Details

#initialize(experiment:, container: nil) ⇒ Grid

Returns a new instance of Grid.



28
29
30
31
 
# File 'lib/llm_experiment/grid.rb', line 28

def initialize(experiment:, container: nil)
  @experiment = experiment
  @container = container
end

Instance Method Details

#done?(task:, agent:, condition:) ⇒ Boolean

A cell is done once it has at least one run that got as far as writing meta.json. Anything earlier than that is an aborted attempt, not a result.

Returns:

  • (Boolean) 


48
49
50
 
# File 'lib/llm_experiment/grid.rb', line 48

def done?(task:, agent:, condition:)
  Dir.glob(File.join(@experiment.results_raw_dir, task, agent, condition, "*", "meta.json")).any?
end

#label(cell) ⇒ Object 



85
 
# File 'lib/llm_experiment/grid.rb', line 85

def label(cell) = format("%s / %s / %s", cell[:task], cell[:agent], cell[:condition])

#pending(**filters) ⇒ Object 



52
53
54
 
# File 'lib/llm_experiment/grid.rb', line 52

def pending(**filters)
  plan(**filters).reject { |cell| done?(**cell) }
end

#plan(app: nil, agent: nil, task: nil) ⇒ Object 



33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/llm_experiment/grid.rb', line 33

def plan(app: nil, agent: nil, task: nil)
  tasks = filtered_tasks(app: app, task: task)
  agents = filtered_agents(agent)
  conditions = @experiment.conditions

  tasks.each_with_index.flat_map do |subject, index|
    ordered = conditions.rotate(index % conditions.size)
    agents.flat_map do |name|
      ordered.map { |condition| { task: subject.id, agent: name, condition: condition } }
    end
  end
end

#run(app: nil, agent: nil, task: nil, redo_done: false, dry_run: false) ⇒ Object 



60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
 
# File 'lib/llm_experiment/grid.rb', line 60

def run(app: nil, agent: nil, task: nil, redo_done: false, dry_run: false)
  filters = { app: app, agent: agent, task: task }
  cells = redo_done ? plan(**filters) : pending(**filters)
  failures = []

  # One trial at a time. See the note on this class.
  cells.each_with_index do |cell, index|
    yield(index + 1, cells.size, cell) if block_given?
    # A grid dry run is about the order and the count. To inspect one
    # command and its prompt, dry-run that single trial instead.
    next if dry_run

    begin
      trial(cell).run
    rescue StandardError => e
      # Record and carry on: the grid is resumable, so a failed cell is
      # simply re-run later, and aborting here would waste every cell after it.
      failures << cell.merge(error: e.message)
      Shell.log "FAILED: #{label(cell)}: #{e.message.lines.first.to_s.strip}"
    end
  end

  Result.new(total: cells.size, failures: failures)
end

#status(**filters) ⇒ Object 



56
57
58
 
# File 'lib/llm_experiment/grid.rb', line 56

def status(**filters)
  plan(**filters).map { |cell| cell.merge(done: done?(**cell)) }
end