Class: Braintrust::Eval::Trace

Inherits:
Object
  • Object
show all
Defined in:
lib/braintrust/eval/trace.rb

Overview

Read-only trace data accessor for scorers.

Per-case throwaway object — no global cache, no shared state. Accepts lazy (lambda) or eager (Array) span sources.

BTQL span shape (string keys from JSON):

"span_attributes" => {"type" => "llm", "name" => "Chat Completion"}
"input"  => [{"role" => "user", "content" => "..."}]        # flat message array
"output" => [{"message" => {"role" => "assistant", ...}}]    # flat choices array

Examples:

Lazy loading from BTQL

trace = Trace.new(spans: -> { btql.trace_spans(...) })
trace.spans              # triggers BTQL query on first access
trace.spans              # returns memoized result

Eager loading

trace = Trace.new(spans: [span1, span2])
trace.spans              # returns array directly

Instance Method Summary collapse

Constructor Details

#initialize(spans:) ⇒ Trace

Returns a new instance of Trace.

Parameters:

  • spans (Proc, Array) —

    Span source — a lambda (lazy) or Array (eager).



25
26
27
28
29
 
# File 'lib/braintrust/eval/trace.rb', line 25

def initialize(spans:)
  @spans_source = spans
  @spans_resolved = false
  @spans_memo = nil
end

Instance Method Details

#spans(span_type: nil) ⇒ Array<Hash>

Resolve and return spans, optionally filtered by type.

The type lives at span_attributes.type in BTQL rows (e.g. “llm”, “eval”, “task”).

Parameters:

  • span_type (String, Array<String>, nil) (defaults to: nil) —

    Filter to spans matching this type. Accepts a single type string or an array of types (returns the union). Returns all spans when nil.

Returns:

  • (Array<Hash>) —

    Matching spans.



39
40
41
42
43
44
45
46
47
 
# File 'lib/braintrust/eval/trace.rb', line 39

def spans(span_type: nil)
  resolved = resolve_spans
  if span_type
    types = Array(span_type)
    resolved.select { |s| types.include?(span_type_for(s)) }
  else
    resolved
  end
end

#thread ⇒ Array<Hash>

Convenience method: extract a chronological message thread from LLM spans.

Walks LLM spans, collects input messages (deduplicated) and output messages (always included). Returns a flat chronological array.

BTQL LLM span format:

input:  flat array of messages  [{"role" => "user", "content" => "..."}]
output: flat array of choices   [{"message" => {"role" => "assistant", ...}}]

Returns:

  • (Array<Hash>) —

    Ordered message list.



59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/braintrust/eval/trace.rb', line 59

def thread
  llm_spans = spans(span_type: "llm")
  return [] if llm_spans.empty?

  seen = Set.new
  messages = []

  llm_spans.each do |span|
    # Input: flat message array or {messages: [...]} wrapper
    input = span["input"] || span[:input]
    input_messages = extract_input_messages(input)
    input_messages&.each do |msg|
      key = msg.hash
      unless seen.include?(key)
        seen.add(key)
        messages << msg
      end
    end

    # Output: flat choices array or {choices: [...]} wrapper
    output = span["output"] || span[:output]
    extract_output_messages(output)&.each do |msg|
      messages << msg
    end
  end

  messages
end