Class: LLMExperiment::Transcript::Parser

Inherits:
Object
  • Object
show all
Defined in:
lib/llm_experiment/transcript/parser.rb

Overview

Reads one trial directory and writes back what can be measured from it: events.jsonl (the transcript with unparsable lines dropped) and metrics.json (the summary the stats read).

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(experiment:) ⇒ Parser

Returns a new instance of Parser.



20
21
22
 
# File 'lib/llm_experiment/transcript/parser.rb', line 20

def initialize(experiment:)
  @experiment = experiment
end

Class Method Details

.detect_agent(events) ⇒ Object

Claude emits stream-json; Codex emits its own JSONL. Detect rather than configure, so a transcript can be parsed without knowing who produced it.



13
14
15
16
17
18
 
# File 'lib/llm_experiment/transcript/parser.rb', line 13

def self.detect_agent(events)
  return "claude" if events.any? { |e| e["type"] == "system" && e["subtype"] == "init" }
  return "codex" if events.any? { |e| e["type"].to_s.start_with?("item.", "turn.", "thread.") }

  "unknown"
end

Instance Method Details

#parse(dir) ⇒ Object

Returns the summary it wrote, or nil for a directory with no transcript.



31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/llm_experiment/transcript/parser.rb', line 31

def parse(dir)
  transcript = File.join(dir, "transcript.jsonl")
  unless File.exist?(transcript)
    Shell.log "skip #{dir}: no transcript.jsonl"
    return nil
  end

  meta = read_meta(dir)
  bad = 0
  events = File.readlines(transcript).filter_map do |line|
    JSON.parse(line)
  rescue JSON::ParserError
    bad += 1
    nil
  end

  summary = summarize(events, meta)
  summary["unparsable_lines"] = bad
  summary["raw_events"] = events.size
  summary.merge!(meta_fields(meta))
  summary["trial_dir"] = dir.sub("#{@experiment.root}/", "")

  File.write(File.join(dir, "events.jsonl"), "#{events.map(&:to_json).join("\n")}\n")
  File.write(File.join(dir, "metrics.json"), JSON.pretty_generate(summary))
  summary
end

#trial_dirs ⇒ Object

Every trial that got as far as writing a transcript.



25
26
27
28
 
# File 'lib/llm_experiment/transcript/parser.rb', line 25

def trial_dirs
  Dir.glob(File.join(@experiment.results_raw_dir, "**", "transcript.jsonl"))
     .map { |p| File.dirname(p) }.sort
end