Class: Insika::ToolUsageReport

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/tool_usage_report.rb

Overview

The tool AUDIT the per-session trace cannot answer: tool_traces records every call, but one session at a time, and nothing aggregates — so "which tools does this agent carry and never use?" had no surface at all. This report is that surface. Read-only by design: it names the candidates, the OPERATOR removes (a tool the report flags may still be the one a rare but critical flow needs).

Attribution rides the task record: a session does not stamp its agent, but every task carries the agent in its command payload — tasks → sessions → tool_traces is the same read the Studio does. Three findings per agent:

never_called — in `tools_allow`, zero calls in any stored trace. Dead
             weight: it costs schema tokens on every request and buys
             nothing (the pilot's `search_orders` is the known example).
error_rate   — called inside the window with > 30% conventional errors
             (the trace's own `ok` flag). Either the tool is broken or
             the model cannot hold its contract; both are operator work.
stale        — called at some point, but not once inside the window.

Bounded and honest: it reads only what the stores kept (the trace caps at 200 entries/session), so a count here is "at least", never an exact total.

Defined Under Namespace

Classes: Report, Row

Constant Summary collapse

WINDOW_DAYS = 
14
ERROR_RATE_THRESHOLD = 
0.30

Instance Method Summary collapse

Constructor Details

#initialize(task_store:, tool_trace_store:, profile_source:, now: nil) ⇒ ToolUsageReport

Returns a new instance of ToolUsageReport.



55
56
57
58
59
60
 
# File 'lib/insika/tool_usage_report.rb', line 55

def initialize(task_store:, tool_trace_store:, profile_source:, now: nil)
  @task_store = task_store
  @tool_trace_store = tool_trace_store
  @profile_source = profile_source
  @now = now
end

Instance Method Details

#generate(days: WINDOW_DAYS, agent: nil) ⇒ Object

-> Report. agent: narrows to one agent (must still be a stored profile).



63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/insika/tool_usage_report.rb', line 63

def generate(days: WINDOW_DAYS, agent: nil)
  now = @now || Time.now.utc
  cutoff = now - (days * 24 * 60 * 60)
  profiles = @profile_source.all_raw
  profiles = profiles.select { |r| r["id"].to_s == agent.to_s } if agent
  sessions = sessions_by_agent

  rows = profiles.flat_map do |record|
    id = record["id"].to_s
    stats = tool_stats(sessions[id] || [], cutoff)
    never_called_rows(id, record, stats) +
      error_rate_rows(id, stats, days) +
      stale_rows(id, stats)
  end

  Report.new(generated_at: now.iso8601, days: days,
             agents: profiles.map { |r| r["id"].to_s }.sort,
             rows: rows.sort_by { |r| [r.agent, r.kind, r.tool] }.freeze)
end