Module: LLMExperiment::Transcript

Defined in:
lib/llm_experiment/transcript.rb,
lib/llm_experiment/transcript/codex.rb,
lib/llm_experiment/transcript/claude.rb,
lib/llm_experiment/transcript/parser.rb,
lib/llm_experiment/transcript/hermeticity.rb

Overview

Turning a trial's raw JSONL transcript into measurements.

The headline number an experiment like this is after is context growth BEFORE the first tool call. If a mention makes a CLI attach the file itself, the first request carries the file's tokens and no read is needed to get them; if the mention is inert text, the first request looks like the bare one and the file arrives later through a normal read. Those two shapes are distinguishable only because the per-agent summarizers keep per-turn token counts rather than only the totals.

Defined Under Namespace

Modules: Claude, Codex, Hermeticity Classes: Parser

Class Method Summary collapse

Class Method Details

.references_impl?(blob, impl_files) ⇒ Boolean

Does this tool call refer to the defect file?

Matching on the full relative path rather than the basename: a basename would also fire on an unrelated grep for a common word, and the point of the metric is how many calls it took to arrive at the file, not how many mentioned a similar-sounding name. The whole serialised input is searched, so a Read with a file_path and a shell cat <path> both count.

Returns:

  • (Boolean) 


23
24
25
 
# File 'lib/llm_experiment/transcript.rb', line 23

def references_impl?(blob, impl_files)
  impl_files.any? { |f| blob.include?(f) }
end