Module: LLMExperiment::Transcript::Codex

Defined in:
lib/llm_experiment/transcript/codex.rb

Overview

Codex emits item.completed envelopes and one turn.completed carrying usage. Item types seen in practice: command_execution (command, aggregated_output, exit_code, status), file_change, agent_message, reasoning, error.

One asymmetry to be honest about: Codex reports tokens per TURN, and a turn contains all of its tool calls. So the context carried before the first tool call - the auto-attachment signature this kind of experiment is chasing - is directly observable for Claude and not for Codex. For Codex the same effect can only show up in the totals, which is weaker but not nothing. It is left nil rather than filled with a number that does not mean what the column says.

Constant Summary collapse

EDIT_ITEMS = 
%w[file_change patch_apply].freeze
TOOL_ITEMS = 
(EDIT_ITEMS + %w[command_execution mcp_tool_call web_search]).freeze

Class Method Summary collapse

Class Method Details

.changed_paths(changes) ⇒ Object

A file_change item carries changes as a path => diff map.

This diverges from the source harness, which wrote Array(item["changes"]).flat_map { |c| c.is_a?(Hash) ? c.keys : [c.to_s] }. Array() turns a Hash into [key, value] pairs before the is_a?(Hash) test can ever see one, so that branch was unreachable and every path came out as the string '["app/x.rb", "@@ -1 +1 @@"]' -- the diff text included. Nothing published reads this field today, which is exactly why it could stay wrong unnoticed.



87
88
89
90
91
92
93
 
# File 'lib/llm_experiment/transcript/codex.rb', line 87

def changed_paths(changes)
  case changes
  when Hash then changes.keys
  when Array then changes.map(&:to_s)
  else []
  end
end

.summarize(events, impl_files:) ⇒ Object 



25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/llm_experiment/transcript/codex.rb', line 25

def summarize(events, impl_files:)
  tool_calls = []
  turns = []
  errors = []

  events.each do |event|
    case event["type"]
    when "item.completed"
      item = event["item"] || {}
      errors << item["message"] if item["type"] == "error"
      next unless TOOL_ITEMS.include?(item["type"])

      command = Array(item["command"]).join(" ")
      paths = command.scan(%r{[\w./-]+\.(?:rb|erb|yml|yaml)}).uniq
      paths |= changed_paths(item["changes"])

      tool_calls << { "name" => item["type"], "paths" => paths, "command" => command,
                      "exit_code" => item["exit_code"], "blob" => item.to_json }
    when "turn.completed"
      u = event["usage"] || {}
      turns << {
        "input_tokens" => u["input_tokens"].to_i,
        "output_tokens" => u["output_tokens"].to_i,
        "cache_read_input_tokens" => u["cached_input_tokens"].to_i,
        "cache_creation_input_tokens" => u["cache_write_input_tokens"].to_i,
        "reasoning_output_tokens" => u["reasoning_output_tokens"].to_i,
        "context_tokens" => u["input_tokens"].to_i
      }
    end
  end

  first_defect_read = tool_calls.index { |t| Transcript.references_impl?(t["blob"], impl_files) }
  first_edit = tool_calls.index { |t| EDIT_ITEMS.include?(t["name"]) }

  {
    "agent" => "codex",
    "input_tokens" => turns.sum { |t| t["input_tokens"] },
    "output_tokens" => turns.sum { |t| t["output_tokens"] },
    "cache_read_input_tokens" => turns.sum { |t| t["cache_read_input_tokens"] },
    "cache_creation_input_tokens" => turns.sum { |t| t["cache_creation_input_tokens"] },
    "thinking_tokens" => turns.sum { |t| t["reasoning_output_tokens"] },
    "total_tool_calls" => tool_calls.size,
    "tool_call_names" => tool_calls.map { |t| t["name"] },
    "tool_calls_to_first_defect_read" => first_defect_read,
    "tool_calls_to_first_edit" => first_edit,
    "read_before_edit" => first_defect_read && first_edit ? first_defect_read < first_edit : nil,
    # Only meaningful if Codex ever reports more than one turn; see the note above.
    "context_before_first_tool_call" => turns.size > 1 ? turns.first["context_tokens"] : nil,
    "codex_errors" => errors,
    "turns" => turns
  }
end