Class: Insika::Refinement::EvidenceCollector

Inherits:
Object
  • Object
show all
Includes:
Coercion
Defined in:
lib/insika/refinement/evidence_collector.rb

Overview

Reads a window of an agent's real traffic and emits RANKED FINDINGS — "here is what broke, how often, and in which conversations". No model runs here and nothing is written to the agent: this is the evidence half of the loop, and it is deliberately useful on its own.

It reads ONLY durable data the engine already records:

TaskStore       — the turn, its Command (which carries the agent) and the
                executions (a failed turn keeps its error)
SessionStore    — the transcript (repetition, canned safe replies)
ToolTraceStore  — per-session tool calls with ok/args/result (already masked
                and clipped by the store itself)

Two signals of are NOT computed here, and that is a finding about the engine rather than about an agent: guardrail decisions and edge-limit hits are emitted as EVENTS and never persisted, so the only durable footprint they leave is the canned safe reply in the transcript — which is exactly what the safe_reply finding matches. Attributing one to a specific rule needs a write path that does not exist yet.

The agent of a turn comes from the Task's Command payload, not from the Session: a Session does not stamp which agent produced it.

Defined Under Namespace

Classes: Finding, Report

Constant Summary collapse

DEFAULT_WINDOW =

distinct sessions, most recent first

200
DEFAULT_MAX_FINDINGS = 
20
MAX_PROVENANCE =

session ids kept per finding

5
SNIPPET_CHARS = 
160
REPETITION_JACCARD = 
0.6
REPETITION_MIN_WORDS =

"oi"/"sim" repeated is not a defect

3
INJECTED_FRAGMENT_RE =

LEGACY FALLBACK, for transcripts written before messages carried an origin.

A message the engine wrote itself — an injected context fragment, a delegation result delivered as a new turn — is persisted with role: user like any other, because it is what the model saw. Counting those as the customer repeating themselves turned the first production run into 219 false positives, every one of them the engine reading its own <store_cep_required> back.

MessageOrigin is the structural answer and is preferred whenever a message carries it. This regex stays for everything written before that field existed (the pilot's database is full of it) and for consumers that have not started declaring origin — it is a guess, and it only ever runs on messages that made no claim about themselves.

/\A<[a-z][a-z0-9_:.-]*>/i
SEVERITY =

Weight of a finding kind when ranking (count × severity).

{
  tool_error: 3, task_failed: 3, safe_reply: 2, repetition: 2, tool_unused: 1
}.freeze

Constants included from Coercion

Coercion::TRUTHY

Instance Method Summary collapse

Methods included from Coercion

blank?, deep_stringify, presence, present?, truthy?, utf8

Constructor Details

#initialize(task_store:, session_store:, tool_trace_store:, profiles:, settings_store: nil) ⇒ EvidenceCollector

Returns a new instance of EvidenceCollector.



70
71
72
73
74
75
76
77
 
# File 'lib/insika/refinement/evidence_collector.rb', line 70

def initialize(task_store:, session_store:, tool_trace_store:, profiles:,
               settings_store: nil)
  @task_store = task_store
  @session_store = session_store
  @tool_trace_store = tool_trace_store
  @profiles = ProfileSource.coerce(profiles)
  @settings_store = settings_store
end

Instance Method Details

#collect(agent_id:, last_sessions: DEFAULT_WINDOW, since: nil, max_findings: DEFAULT_MAX_FINDINGS, exclude_sessions: []) ⇒ Object

-> Report. since (ISO8601) wins over last_sessions when both are given: an incremental run ("what happened since the last one") is the common case.

exclude_sessions drops sessions whose id starts with any of the given prefixes. It defaults to NOTHING — a report must not decide on its own what counts as real traffic — but a deployment that replays load tests or debug conversations into the same store needs it: on the pilot, loadtest- sessions outnumbered real ones and drowned every genuine finding.



87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
 
# File 'lib/insika/refinement/evidence_collector.rb', line 87

def collect(agent_id:, last_sessions: DEFAULT_WINDOW, since: nil,
            max_findings: DEFAULT_MAX_FINDINGS, exclude_sessions: [])
  agent = agent_id.to_s
  profile = @profiles[agent] ||
            (raise Insika::NotFoundError, "agent '#{agent}' not configured")

  tasks, excluded = window_tasks(agent, last_sessions: last_sessions, since: since,
                                        exclude_sessions: Array(exclude_sessions))
  session_ids = tasks.filter_map { |t| presence(t.session_id) }.uniq
  traces = session_ids.to_h { |sid| [sid, @tool_trace_store.for_session(sid)] }

  findings = [
    *tool_error_findings(traces),
    *task_failed_findings(tasks),
    *repetition_findings(session_ids),
    *safe_reply_findings(session_ids, profile),
    # An EMPTY window says nothing about a tool being unused — it says the agent
    # did not run. Without this guard every incremental run over quiet traffic
    # would report the whole tool list as "never called".
    *(tasks.empty? ? [] : tool_unused_findings(profile, traces))
  ]

  Report.new(
    agent_id: agent,
    window: since ? { "since" => since.to_s } : { "last_sessions" => last_sessions },
    findings: rank(findings).first(max_findings),
    sessions_seen: session_ids.size, turns_seen: tasks.size, excluded: excluded
  )
end