Class: ActionAgent::AgentScorecard

Inherits:
Object
  • Object
show all
Defined in:
app/services/action_agent/agent_scorecard.rb

Overview

Per-agent scorecard stats for the dashboard's agent cards, computed with grouped queries (no per-agent N+1) over two sources:

  • agent_runs — executions the platform itself ran, and
  • active_agent_telemetry_traces — executions reported by an SDK in the customer's own app, attributed to an Agent by AgentRegistrar.

Agents discovered by observation (status: :observed) only ever have the second kind, so a runs-only scorecard reported 0 for every tile while the Traces view showed real traffic.

A platform run writes BOTH an AgentRun and a trace sharing a trace_id, so traces are counted only when no AgentRun claims the same trace_id — otherwise every platform execution would count twice.

Constant Summary collapse

WINDOW = 
30.days

Class Method Summary collapse

Class Method Details

.for_agents(agents) ⇒ Hash{Integer => Hash}

Returns agent_id => stats.

Parameters:

  • agents (Enumerable<Agent>)

Returns:

  • (Hash{Integer => Hash}) —

    agent_id => stats



23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'app/services/action_agent/agent_scorecard.rb', line 23

def self.for_agents(agents)
  ids = agents.map(&:id)
  return {} if ids.empty?

  window_start = WINDOW.ago
  windowed = AgentRun.where(agent_id: ids, created_at: window_start..)

  run_counts = windowed.group(:agent_id).count
  completed_counts = windowed.where(status: :complete).group(:agent_id).count
  avg_durations = windowed.where.not(duration_ms: nil).group(:agent_id).average(:duration_ms)
  token_sums = windowed.group(:agent_id).sum("COALESCE(total_tokens, 0)")
  last_runs = AgentRun.where(agent_id: ids).group(:agent_id).maximum(:created_at)
  eval_runs = latest_evaluation_runs(ids)

  trace_stats = telemetry_stats(ids, window_start)
  trace_last = unclaimed_traces(ids, nil).group(:agent_id).maximum(:timestamp)
  costs = estimated_costs(ids, windowed, window_start)

  ids.index_with do |id|
    runs = run_counts[id].to_i
    stats = trace_stats[id] || {}
    traced = stats[:count].to_i
    total = runs + traced
    succeeded = completed_counts[id].to_i + stats[:ok].to_i
    eval_run = eval_runs[id]

    {
      window_days: (WINDOW / 1.day).to_i,
      runs: total,
      # Which sources contributed, so the UI can say where numbers came from.
      run_sources: run_sources(runs, traced),
      success_rate: total.positive? ? (succeeded * 100.0 / total).round(1) : nil,
      avg_duration_ms: blended_duration(avg_durations[id], runs, stats[:avg_duration], traced),
      tokens: token_sums[id].to_i + stats[:tokens].to_i,
      cost: costs[id],
      eval_score: eval_run&.average_score,
      eval_samples_passed: eval_run&.samples_passed,
      eval_samples_evaluated: eval_run&.samples_evaluated,
      last_run_at: [ last_runs[id], trace_last[id] ].compact.max&.iso8601
    }
  end
end