Module: Riffer::Evals::Profile::AgentClassMethods

Defined in:
lib/riffer/evals/profile.rb

Instance Method Summary collapse

Instance Method Details

#run_eval(input:, context: nil, tool_context: nil) ⇒ Object

Runs evaluations against the agent.

: (input: String, ?context: Hash[Symbol, untyped]?, ?tool_context: Hash[Symbol, untyped]?) -> Riffer::Evals::RunResult

Raises:



85
86
87
88
89
90
91
92
93
94
95
96
97
98
 
# File 'lib/riffer/evals/profile.rb', line 85

def run_eval(input:, context: nil, tool_context: nil)
  profile = @eval_profile
  raise Riffer::ArgumentError, "No eval profile configured" unless profile

  metrics = profile.eval_metrics
  raise Riffer::ArgumentError, "No metrics configured in eval profile" if metrics.empty?

  # Generate output from agent
  response = generate(input, tool_context: tool_context)

  # Run evaluations
  runner = Riffer::Evals::Runner.new(metrics: metrics)
  runner.run(input: input, output: response.content, context: context)
end