Class: Insika::Evals::Simulator

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/evals/simulator.rb

Overview

SIMULATED USERS. Two models talking: the target agent (reached through the same Transport seam as the Runner) and the simulated customer (a cheap model — the platform utility_model — playing a Persona).

Pure over its seams, like the Runner: the Transport is injected (HttpTransport for a remote deployment, GraphTransport for the own graph, A2ATransport for an agent that only speaks A2A) and the persona model is an injected ask (prompt -> raw text), so the loop is unit-testable offline.

Termination is recorded, never guessed: the persona's max_turns, the persona emitting <<goal_met>> (its goal is served) or <<gave_up>> (it abandons), or an errored agent turn. "Gave up at turn 3" is a finding, not a detail.

SAFETY (rule fixed in the spec): a simulated conversation must not write for real. The Safety gate refuses the run unless the target is declared staging or the run uses an eval profile where the agent's side-effect tools are swapped for fakes — and the swap list is DERIVED from the tool registry (the engine marks side_effect on tools; see EvalProfile), never hand-maintained. Every run is simulated: true, so a report never mixes a generated conversation with real traffic.

Defined Under Namespace

Classes: DryRunTool, Safety, SimulatedRun, UnsafeTarget

Constant Summary collapse

STOP_GOAL_MET = 
"<<goal_met>>"
STOP_GAVE_UP = 
"<<gave_up>>"
STOPS = 
{ STOP_GOAL_MET => :goal_met, STOP_GAVE_UP => :gave_up }.freeze

Instance Method Summary collapse

Constructor Details

#initialize(transport:, ask:, safety:) ⇒ Simulator

transport: the target agent seam (TurnOutcome per turn, like the Runner). ask: ->(prompt) { text } — the simulated customer (the cheap model). safety: a Safety — the gate evaluated before every run.



92
93
94
95
96
 
# File 'lib/insika/evals/simulator.rb', line 92

def initialize(transport:, ask:, safety:)
  @transport = transport
  @ask = ask
  @safety = safety
end

Instance Method Details

#run(persona:, agent:, conv:) ⇒ Object

Runs one simulated conversation. -> SimulatedRun.

Raises:



99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
# File 'lib/insika/evals/simulator.rb', line 99

def run(persona:, agent:, conv:)
  reason = @safety.refusal
  raise UnsafeTarget, reason if reason

  transcript = []
  message = persona.opens_with
  stop = :max_turns

  1.upto(persona.max_turns) do |turn|
    transcript << { role: "user", text: message }
    outcome = @transport.turn(agent: agent, conv: conv, message: message)
    if outcome.result.error
      transcript << { role: "assistant", text: "", tools: [] }
      return SimulatedRun.new(transcript: transcript, stop: :error, turns: turn,
                              error: outcome.result.error)
    end
    transcript << { role: "assistant", text: outcome.result.output_text,
                    tools: outcome.result.tool_names }
    break if turn == persona.max_turns # the persona's budget is spent

    reply = @ask.call(persona.prompt(transcript)).to_s
    marker, text = strip_stop(reply)
    if marker
      transcript << { role: "user", text: text } unless text.empty?
      return SimulatedRun.new(transcript: transcript, stop: marker, turns: turn)
    end
    if text.empty?
      return SimulatedRun.new(transcript: transcript, stop: :error, turns: turn,
                              error: "the persona produced an empty message")
    end

    message = text
  end

  SimulatedRun.new(transcript: transcript, stop: stop, turns: persona.max_turns)
end