Class: Insika::Tools::RunPersonaEval

Inherits:
RubyLLM::Tool
  • Object
show all
Defined in:
lib/insika/tools/run_persona_eval.rb

Overview

run_persona_eval — a QA agent's own probe: pick an authored SIMULATED persona case (Evals::GoldenStore) and run it, in-process, against the case's declared target agent — the same Simulator + Judge machinery insika evals:simulate drives over HTTP, minus the CLI and the network hop.

SAFETY is DERIVED, never a flag: the target's reachable side-effect tools are computed from the live registry (Evals::EvalProfile), the same way the CLI derives them. A read-only target needs no swap: Simulator::Safety's own "side_effect_tools.empty?" branch allows it directly. A target that DOES reach a side-effect tool gets the REAL swap (Evals::EvalProfile.registry — own overlay, wired here): every one of those tools resolves to a Simulator::DryRunTool for the duration of this ONE simulated conversation, run through a THROWAWAY Executor+Bus built fresh per call (shadow_runtime) — sharing every OTHER collaborator of the real graph (guardrails, policy, context assembly, skills/prompts, the real session/task/checkpoint stores), so the target is tested as faithfully as --staging ever was, minus the one write. Needs the real graph: (see initialize) — a caller that does not have one (an old-style double) falls back to refusing outright.

BUDGET: the persona model + judge model calls are the cost of running the eval, charged to the CALLING agent's turn (never the target's — the target's own turns are billed normally, through the ordinary edge limiter, exactly as if a customer had sent those messages). A hard cap on the calling agent skips the run — visibly, in the tool result — before a cent is spent.

TENANT ISOLATION: a persona case belongs to a tenant (Golden#tenant, "platform" by default); this tool only ever lists/runs cases in the CALLING agent's own tenant (calling_tenant, read off turn_context, never the model). One QA agent per store (the C3.2 plan) is what makes this meaningful -- without it, "qa-store-a" could enumerate and run "qa-store-b"'s persona and read its knows in the transcript.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(golden_store:, profiles:, tool_registry:, runtime:, settings_store:, graph: nil, budget_ledger: nil, event_stream: nil, llm: nil) ⇒ RunPersonaEval

golden_store: Evals::GoldenStore — where authored persona cases live, scoped to the CALLING agent's own tenant (never the model's — see calling_tenant). A case authored for another tenant is invisible here, not merely undocumented. profiles: ProfileSource — resolves both the target agent (the case's agent:) and the CALLING agent (turn_context, for the budget check). tool_registry: the deployment's EFFECTIVE registry — what Evals::EvalProfile derives the target's side-effect tools from (the same registry a real turn resolves tools on). runtime: anything answering #chat(message, session_id:, agent:) (raises Insika::Error on failure) — Evals::GraphTransport's contract. Used AS-IS for a read-only target; a target with a reachable side-effect tool needs graph: instead (this alone cannot swap anything). graph: the real Wiring::Graph::Result — ONLY consulted to build the throwaway swapped-registry Executor+Bus (shadow_runtime) when the target has a reachable side-effect tool. nil (an old-style double) = that case refuses outright, same as before this existed. settings_store: where the platform utility_model (persona) and the judge panel (evals.judges) are configured. budget_ledger: WS2 counters — read before the run (skip on a hard cap), written after (persona + judge spend only). llm: this graph's own RubyLLM::Context, if it has one (a DSL-built graph's own credentials) — the persona/judge calls' preferred source, ahead of runtime.llm (kept for the existing double-based specs) and the process-wide RubyLLM constant.



81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/insika/tools/run_persona_eval.rb', line 81

def initialize(golden_store:, profiles:, tool_registry:, runtime:, settings_store:,
               graph: nil, budget_ledger: nil, event_stream: nil, llm: nil)
  @golden_store = golden_store
  @profiles = profiles
  @tool_registry = tool_registry
  @runtime = runtime
  @graph = graph
  @settings_store = settings_store
  @budget_ledger = budget_ledger
  @event_stream = event_stream
  @llm = llm
  super()
end

Instance Attribute Details

#turn_context ⇒ Object

Per-turn bindings, deposited by the Executor (ToolAssembly's turn_context= seam — same as save_artifact/data-tools). Only the CALLING agent's id + declared tenant are used here (the budget check); never a tenant/agent the model types.



99
100
101
 
# File 'lib/insika/tools/run_persona_eval.rb', line 99

def turn_context
  @turn_context
end

Instance Method Details

#description ⇒ Object

The runnable case ids, named so the model cannot guess one that does not exist (the Subagent tool's lesson — see AgentEnum).



107
108
109
110
111
112
 
# File 'lib/insika/tools/run_persona_eval.rb', line 107

def description
  ids = case_ids
  return super if ids.empty?

  "#{super} Cases you may run: #{ids.join(', ')}."
end

#execute(case_id:) ⇒ Object 



118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
 
# File 'lib/insika/tools/run_persona_eval.rb', line 118

def execute(case_id:)
  golden = @golden_store.find(case_id.to_s)
  # The SAME error, whether the case does not exist or exists under another
  # tenant — a QA agent must not be able to tell the two apart (that
  # distinction is itself a leak: "case exists, just not yours").
  unless golden&.simulated? && golden.tenant == calling_tenant
    return { error: "unknown or invalid persona case '#{case_id}'" }
  end

  target = @profiles[golden.agent]
  return { error: "persona case '#{case_id}' targets unknown agent '#{golden.agent}'" } unless target

  derived = Insika::Evals::EvalProfile.side_effect_tools(target, @tool_registry)
  return refuse_side_effects(golden.agent, derived) if !derived.empty? && @graph.nil?

  skip = budget_skip
  return skip if skip

  meter = []
  judge = build_judge(meter)
  return { error: "no judge configured for this case — Studio -> Settings -> Evals" } unless judge

  persona_ask = build_persona_ask(meter)
  return persona_ask if persona_ask.is_a?(Hash) # {error:}

  run_and_score(golden, derived, persona_ask, judge, meter)
rescue Insika::Evals::Simulator::UnsafeTarget => e
  { error: e.message }
end

#name ⇒ Object 



50
 
# File 'lib/insika/tools/run_persona_eval.rb', line 50

def name = "run_persona_eval"

#params_schema ⇒ Object 



114
115
116
 
# File 'lib/insika/tools/run_persona_eval.rb', line 114

def params_schema
  Insika::Tools::AgentEnum.inject(super, case_ids, path: %i[case_id])
end