Module: Insika::Evals::EvalProfile

Defined in:
lib/insika/evals/simulator.rb

Overview

The DERIVED eval profile: which of an agent's reachable tools can write for real, computed from the tool registry — the engine marks side_effect on tools (a data-tool's non-GET method, the MCP ingestor's tools/call), so the swap list is a fact of the deployment, never a hand-maintained list.

Defined Under Namespace

Classes: OverlayRegistry

Class Method Summary collapse

Class Method Details

.registry(base, side_effect_tools:, dry_run: nil) ⇒ Object

A registry overlay that answers the swapped names with a DRY-RUN tool and delegates everything else to the base — the "side_effect -> fake" half of the eval profile, derived from the base registry (nothing hand-maintained). The overlay is a drop-in for the Executor's registry: same entries/resolve/side_effect? surface.



177
178
179
180
181
 
# File 'lib/insika/evals/simulator.rb', line 177

def registry(base, side_effect_tools:, dry_run: nil)
  swapped = Array(side_effect_tools).map(&:to_s)
  fake = dry_run || ->(name) { Simulator::DryRunTool.new(name) }
  OverlayRegistry.new(base: base, swapped: swapped, fake: fake)
end

.safe?(profile, registry) ⇒ Boolean

-> bool: can a simulated run touch this agent without a staging declaration (no reachable side-effect tool)?

Returns:

  • (Boolean) 


168
169
170
 
# File 'lib/insika/evals/simulator.rb', line 168

def safe?(profile, registry)
  side_effect_tools(profile, registry).empty?
end

.side_effect_tools(profile, registry) ⇒ Object

profile + registry (answers #names and #side_effect?) -> [tool names] the agent can reach that are marked side-effect, sorted.



158
159
160
161
162
163
164
 
# File 'lib/insika/evals/simulator.rb', line 158

def side_effect_tools(profile, registry)
  allowed = if profile.tools_allow.nil? then Array(registry.names)
            else Array(profile.tools_allow).map(&:to_s)
            end
  denied = Array(profile.tools_deny).map(&:to_s)
  (allowed - denied).select { |name| registry.side_effect?(name) }.sort
end