Class: Ask::Eval::SessionEval

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/eval/session_eval.rb

Overview

Evaluation wrapper around an Agent::Session for testing.

Tracks tool calls, cost, and responses. Integrates with the Recorder for regression testing.

Examples:

eval = Ask::Eval::SessionEval.new(session)
response = eval.run("Check health")
eval.tool_called?("bash")     # => true/false
eval.total_cost               # => 0.0012

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(session, recorder: nil) ⇒ SessionEval

Returns a new instance of SessionEval.

Parameters:

  • session (Ask::Agent::Session) —

    a configured session

  • recorder (Recorder, nil) (defaults to: nil) —

    optional recorder for regression testing



27
28
29
30
31
32
33
 
# File 'lib/ask/eval/session_eval.rb', line 27

def initialize(session, recorder: nil)
  @session = session
  @recorder = recorder
  @tool_calls_made = []
  @total_cost = 0.0
  @last_response = nil
end

Instance Attribute Details

#session ⇒ Ask::Agent::Session (readonly)

Returns the wrapped session.

Returns:

  • (Ask::Agent::Session) —

    the wrapped session



17
18
19
 
# File 'lib/ask/eval/session_eval.rb', line 17

def session
  @session
end

#tool_calls_made ⇒ Array<String> (readonly)

Returns tool names that were called.

Returns:

  • (Array<String>) —

    tool names that were called



20
21
22
 
# File 'lib/ask/eval/session_eval.rb', line 20

def tool_calls_made
  @tool_calls_made
end

#total_cost ⇒ Float (readonly)

Returns total cost of all LLM calls.

Returns:

  • (Float) —

    total cost of all LLM calls



23
24
25
 
# File 'lib/ask/eval/session_eval.rb', line 23

def total_cost
  @total_cost
end

Instance Method Details

#last_response ⇒ String?

Returns the last response text.

Returns:

  • (String, nil) —

    the last response text



46
47
48
 
# File 'lib/ask/eval/session_eval.rb', line 46

def last_response
  @last_response
end

#run(prompt) ⇒ String

Run the session with a prompt and track results.

Parameters:

  • prompt (String) —

    the user prompt

Returns:

  • (String) —

    the session response



38
39
40
41
42
43
 
# File 'lib/ask/eval/session_eval.rb', line 38

def run(prompt)
  @last_response = @session.run(prompt)
  @tool_calls_made = @session.tool_calls_made || []
  @total_cost = @session.total_cost.to_f
  @last_response
end

#running? ⇒ Boolean

Returns whether the agent is still running.

Returns:

  • (Boolean) —

    whether the agent is still running



66
67
68
 
# File 'lib/ask/eval/session_eval.rb', line 66

def running?
  @session.running?
end

#tool_called?(tool_name) ⇒ Boolean

Returns whether the tool was called.

Parameters:

  • tool_name (String) —

    tool name to check

Returns:

  • (Boolean) —

    whether the tool was called



52
53
54
55
56
 
# File 'lib/ask/eval/session_eval.rb', line 52

def tool_called?(tool_name)
  @tool_calls_made.any? { |tc|
    tc.respond_to?(:name) ? tc.name == tool_name : tc.to_s == tool_name
  }
end

#tool_names ⇒ Array<String>

Returns names of tools that were called.

Returns:

  • (Array<String>) —

    names of tools that were called



59
60
61
62
63
 
# File 'lib/ask/eval/session_eval.rb', line 59

def tool_names
  @tool_calls_made.map { |tc|
    tc.respond_to?(:name) ? tc.name : tc.to_s
  }
end