Class: Brute::Eval::Case

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/eval/case.rb

Overview

One evaluation case: what the agent is told, the world it is told it in, and what must be true of the turn afterwards.

Brute::Eval::Case.new(
"searches for what it cannot know",
said:     "what did the Bank of England do yesterday?",
stubs:    { "search" => RATE_DECISION },
calls:    { "search" => { "query" => /bank|rate/i } },
mentions: %w[4.25],
budget:   Brute::Eval::Budget.new(tool_calls: 2),
)

The expectations are deliberately about what the turn DID, not about prose: a call that was made, a call that was not, the order two calls came in, a word the answer has to contain, a budget it has to stay inside. What none of those can say goes in the block, which is handed the transcript.

said reaches the agent through the world -- an inbox on disk, a queue, whatever that deployment's world does with it -- unless via: :start hands it straight to the turn. files and conversation are the world's to lay out and mean nothing to a world that keeps no state.

Defined Under Namespace

Classes: Verdict

Constant Summary collapse

ABSENCE =

A system prompt that tells an agent to say plainly when it found nothing is graded on this. It is a crude reading -- a judge would do it properly -- and it is English, so a deployment whose agents answer in another language passes its own absence:.

/\b(no|not|none|nothing|cannot|can't|couldn't|didn't|don't|doesn't|isn't|aren't|unable|unfortunately|missing|without)\b/i

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, said: nil, via: :world, files: {}, conversation: [], stubs: {}, calls: {}, never: [], order: [], mentions: [], absent: false, absence: ABSENCE, silent: false, budget: Budget.new, runs: 1, &check) ⇒ Case

Returns a new instance of Case.



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/brute/eval/case.rb', line 57

def initialize(
  name,
  said: nil,
  via: :world,
  files: {},
  conversation: [],
  stubs: {},
  calls: {},
  never: [],
  order: [],
  mentions: [],
  absent: false,
  absence: ABSENCE,
  silent: false,
  budget: Budget.new,
  runs: 1,
  &check
)
  @name = name
  @said = said
  @via = via
  @files = files
  @conversation = conversation
  @stubs = stubs
  @calls = calls
  @never = never
  @order = order
  @mentions = mentions
  @absent = absent
  @absence = absence
  @silent = silent
  @budget = budget
  @runs = runs
  @check = check
end

Instance Attribute Details

#budget ⇒ Object (readonly)

Returns the value of attribute budget.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def budget
  @budget
end

#conversation ⇒ Object (readonly)

Returns the value of attribute conversation.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def conversation
  @conversation
end

#files ⇒ Object (readonly)

Returns the value of attribute files.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def files
  @files
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def name
  @name
end

#runs ⇒ Object (readonly)

Returns the value of attribute runs.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def runs
  @runs
end

#said ⇒ Object (readonly)

Returns the value of attribute said.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def said
  @said
end

#stubs ⇒ Object (readonly)

Returns the value of attribute stubs.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def stubs
  @stubs
end

#via ⇒ Object (readonly)

Returns the value of attribute via.



55
56
57
 
# File 'lib/brute/eval/case.rb', line 55

def via
  @via
end

Instance Method Details

#verdict(transcript) ⇒ Object 



93
94
95
96
97
98
99
100
101
102
103
104
 
# File 'lib/brute/eval/case.rb', line 93

def verdict(transcript)
  Verdict.new(
    [
      answered(transcript),
      called(transcript),
      ordered(transcript),
      said_it(transcript),
      afforded(transcript),
      checked(transcript),
    ].flatten.compact.uniq,
  )
end