Class: Brute::Eval::Transcript

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/eval/transcript.rb

Overview

What one turn did.

Every observation comes off the agent's own hooks, so a transcript is what the run itself reported: the tool calls in the order they were made, with the result each came back with, what the model finally said, what the provider charged for it, and how the call failed when it did. Nothing in the agent knows it is being watched.

transcript = Brute::Eval::Transcript.new
transcript.subscribe(agent)
agent.start("what changed?")

transcript.called?("search", "query" => /fed/i)
transcript.before?("read", "write")

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ Transcript

Returns a new instance of Transcript.



27
28
29
30
31
32
33
34
 
# File 'lib/brute/eval/transcript.rb', line 27

def initialize
  @calls = []
  @usage = Hash.new(0)
  @failures = []
  @published = []
  @env = {}
  @seconds = 0.0
end

Instance Attribute Details

#calls ⇒ Object (readonly)

Returns the value of attribute calls.



24
25
26
 
# File 'lib/brute/eval/transcript.rb', line 24

def calls
  @calls
end

#error ⇒ Object

Returns the value of attribute error.



25
26
27
 
# File 'lib/brute/eval/transcript.rb', line 25

def error
  @error
end

#failures ⇒ Object (readonly)

Returns the value of attribute failures.



24
25
26
 
# File 'lib/brute/eval/transcript.rb', line 24

def failures
  @failures
end

#published ⇒ Object

Returns the value of attribute published.



25
26
27
 
# File 'lib/brute/eval/transcript.rb', line 25

def published
  @published
end

#seconds ⇒ Object

Returns the value of attribute seconds.



25
26
27
 
# File 'lib/brute/eval/transcript.rb', line 25

def seconds
  @seconds
end

#usage ⇒ Object (readonly)

Returns the value of attribute usage.



24
25
26
 
# File 'lib/brute/eval/transcript.rb', line 24

def usage
  @usage
end

Instance Method Details

#before?(first, second) ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
80
81
82
83
84
85
86
 
# File 'lib/brute/eval/transcript.rb', line 77

def before?(first, second)
  at = names.index(first.to_s)
  then_at = names.index(second.to_s)

  if at.nil?
    false
  else
    then_at.nil? || at < then_at
  end
end

#called?(name, arguments = {}) ⇒ Boolean

A call the turn made, matched on name and on whatever arguments the case cares about -- ===, so a case says "query" => /fed/i as readily as "count" => 3.

Returns:

  • (Boolean) 


70
71
72
73
74
75
 
# File 'lib/brute/eval/transcript.rb', line 70

def called?(name, arguments = {})
  @calls.any? { |call|
    call[:name] == name.to_s &&
      arguments.all? { |key, wanted| wanted === call[:arguments][key.to_s] }
  }
end

#counts ⇒ Object 



51
 
# File 'lib/brute/eval/transcript.rb', line 51

def counts = names.tally

#errors ⇒ Object 



57
 
# File 'lib/brute/eval/transcript.rb', line 57

def errors = @calls.count { |call| call[:result].to_s.start_with?("Error") }

#iterations ⇒ Object 



53
 
# File 'lib/brute/eval/transcript.rb', line 53

def iterations = @env[:current_iteration] || 0

#names ⇒ Object 



49
 
# File 'lib/brute/eval/transcript.rb', line 49

def names = @calls.map { |call| call[:name] }

#reply ⇒ Object 



59
60
61
62
63
64
65
 
# File 'lib/brute/eval/transcript.rb', line 59

def reply
  if @env[:messages].nil?
    ""
  else
    @env.extend(Brute::Env).reply&.content.to_s
  end
end

#subscribe(agent) ⇒ Object

The call env the tool pipeline hands its subscribers is one mutable hash per call, so the entry kept here at :before_tool carries the result the tool answered with by the time anyone reads it.



39
40
41
42
43
44
45
46
47
 
# File 'lib/brute/eval/transcript.rb', line 39

def subscribe(agent)
  agent
    .on(:before_tool) { |_env, call| @calls << call }
    .on(:after_llm) { |env| (env[:metadata][:last_llm_usage]) }
    .on(:faraday_error) { |_env, error| @failures << describe(error) }
    .on(:open_router_server_error) { |_env, error| @failures << describe(error) }
    .on(:standard_error) { |_env, error| @failures << describe(error) }
    .on(:turn_end) { |env| @env = env }
end

#tokens ⇒ Object 



55
 
# File 'lib/brute/eval/transcript.rb', line 55

def tokens = @usage[:total]