Class: Brute::Eval::Case

Inherits:
Object
  • Object
show all
Defined in:
lib/brute/eval/case.rb

Overview

One evaluation case: what the agent is told, the world it is told it in, and what must be true of the turn afterwards.

Brute::Eval::Case.new(
"searches for what it cannot know",
said:     "what did the Bank of England do yesterday?",
stubs:    { "search" => RATE_DECISION },
calls:    { "search" => { "query" => /bank|rate/i } },
mentions: %w[4.25],
budget:   Brute::Eval::Budget.new(tool_calls: 2),
)

The expectations are deliberately about what the turn DID, not about prose: a call that was made, a call that was not, the order two calls came in, a word the answer has to contain, a budget it has to stay inside. What none of those can say goes in the block, which is handed the transcript.

said reaches the agent through the world -- an inbox on disk, a queue, whatever that deployment's world does with it -- unless via: :start hands it straight to the turn. files and conversation are the world's to lay out and mean nothing to a world that keeps no state.

Defined Under Namespace

Classes: Verdict

Constant Summary collapse

ABSENCE =

A system prompt that tells an agent to say plainly when it found nothing is graded on this. It is a crude reading -- a judge would do it properly -- and it is English, so a deployment whose agents answer in another language passes its own absence:.

/\b(no|not|none|nothing|cannot|can't|couldn't|didn't|don't|doesn't|isn't|aren't|unable|unfortunately|missing|without)\b/i

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, said: nil, via: :world, files: {}, conversation: [], stubs: {}, calls: {}, never: [], order: [], mentions: [], absent: false, absence: ABSENCE, silent: false, budget: Budget.new, runs: 1, &check) ⇒ Case

Returns a new instance of Case.



52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
 
# File 'lib/brute/eval/case.rb', line 52

def initialize(
  name,
  said: nil,
  via: :world,
  files: {},
  conversation: [],
  stubs: {},
  calls: {},
  never: [],
  order: [],
  mentions: [],
  absent: false,
  absence: ABSENCE,
  silent: false,
  budget: Budget.new,
  runs: 1,
  &check
)
  @name = name
  @said = said
  @via = via
  @files = files
  @conversation = conversation
  @stubs = stubs
  @calls = calls
  @never = never
  @order = order
  @mentions = mentions
  @absent = absent
  @absence = absence
  @silent = silent
  @budget = budget
  @runs = runs
  @check = check
end

Instance Attribute Details

#budget ⇒ Object (readonly)

Returns the value of attribute budget.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def budget
  @budget
end

#conversation ⇒ Object (readonly)

Returns the value of attribute conversation.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def conversation
  @conversation
end

#files ⇒ Object (readonly)

Returns the value of attribute files.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def files
  @files
end

#name ⇒ Object (readonly)

Returns the value of attribute name.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def name
  @name
end

#runs ⇒ Object (readonly)

Returns the value of attribute runs.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def runs
  @runs
end

#said ⇒ Object (readonly)

Returns the value of attribute said.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def said
  @said
end

#stubs ⇒ Object (readonly)

Returns the value of attribute stubs.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def stubs
  @stubs
end

#via ⇒ Object (readonly)

Returns the value of attribute via.



50
51
52
 
# File 'lib/brute/eval/case.rb', line 50

def via
  @via
end

Instance Method Details

#verdict(transcript) ⇒ Object 



88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/brute/eval/case.rb', line 88

def verdict(transcript)
  Verdict.new(
    [
      answered(transcript),
      called(transcript),
      ordered(transcript),
      said_it(transcript),
      afforded(transcript),
      checked(transcript),
    ].flatten.compact.uniq
  )
end