Class: ActionAgent::Evaluation

Inherits:
ApplicationRecord show all
Defined in:
app/models/action_agent/evaluation.rb

Overview

An evaluation definition for an agent: a named set of criteria scored against the agent's recorded behavior — its recent generations (solid_agent's agent_generations dataset) and its telemetry traces.

Criteria are stored as an array of { "key", "type", "config" } hashes. Rule-based criterion types score each sampled generation deterministically; telemetry criterion types score aggregates over the agent's traces (error rate, latency); the llm_judge type asks a judge model to score each sample and requires a configured provider.

Constant Summary collapse

JUDGE_KINDS =

judge_defined: the judge model authors the KPI criteria itself from the agent's instructions + sample interactions on the first run, then scores against them (criteria stay persisted/editable so scores are comparable across runs and models).

%w[rules llm judge_defined].freeze
RULE_CRITERION_TYPES = 
%w[
  response_present min_length max_latency_ms token_budget contains not_contains
].freeze
TELEMETRY_CRITERION_TYPES =

Scored from the agent's telemetry traces (aggregate, not per-sample).

%w[trace_error_rate trace_latency].freeze
CRITERION_TYPES = 
(RULE_CRITERION_TYPES + TELEMETRY_CRITERION_TYPES + %w[llm_judge]).freeze

Instance Method Summary collapse

Methods inherited from ApplicationRecord

for_owner, owner_association

Instance Method Details

#compare_models ⇒ Object

Candidate models for per-cohort comparison scoring (config, optional).



46
47
48
 
# File 'app/models/action_agent/evaluation.rb', line 46

def compare_models
  Array(config["compare_models"]).map(&:to_s).reject(&:blank?)
end

#judge_defined? ⇒ Boolean

Returns:

  • (Boolean) 


41
42
43
 
# File 'app/models/action_agent/evaluation.rb', line 41

def judge_defined?
  judge_kind == "judge_defined"
end

#latest_run ⇒ Object 



37
38
39
 
# File 'app/models/action_agent/evaluation.rb', line 37

def latest_run
  evaluation_runs.order(created_at: :desc).first
end

#llm_criteria ⇒ Object 



54
55
56
 
# File 'app/models/action_agent/evaluation.rb', line 54

def llm_criteria
  criteria.select { |c| c["type"] == "llm_judge" }
end

#run! ⇒ Object 



50
51
52
 
# File 'app/models/action_agent/evaluation.rb', line 50

def run!
  EvaluationRunnerService.call(self)
end