Module: Insika::Evals::Assertions

Defined in:
lib/insika/evals/assertions.rb

Overview

Deterministic (Fase A) evaluation — cheap, zero-token, zero-flakiness. It's the layer that catches the gross regressions (a tool stopped being called, a secret leaked, the turn errored). Subjective scoring is the LLM-judge in Fase B.

Constant Summary collapse

PII_DETECTORS =

Named negative detectors for must_not now live in the runtime (D4). Kept as an alias so any external reference to Evals::Assertions::PII_DETECTORS still resolves; the values ARE the runtime's, never a fork.

Insika::Safety::Detectors::PII
POLICIES =

HOW MUCH THE AGENT SHOULD ASK BEFORE ACTING (RFC-0014 §3.3). Declared per case because it is a per-STORE decision, not a universal rule: sometimes the agent should establish the objective before searching ("energia, treino ou sono?" — Ocean Drop does this well), and sometimes asking again is the failure and it should just search. A global assertion would be wrong half the time; the judge is TOLD the policy (Judge#build_prompt) and this layer checks the half that needs no reader.

Each rule is stated as the CUSTOMER-VISIBLE fact it checks. RFC-0014 phrased this as "questions before the first tool call", written before P19: text a model emits before calling a tool never reaches the customer now (it rides :intermediate), and the eval is a client of /v1/responses, so what it can observe per turn is the published answer plus the tools that turn called. That is also the honest scope — a question nobody received is not a question.

{
  # "UMA PERGUNTA POR VEZ" — the rule Insika broke twice under a 28 KB prompt.
  "ask_once" => "at most one question per reply",
  # Establish the objective before acting on a vague opener.
  "investigate_first" => "asks before calling a tool, on the first turn",
  # Act on the first plausible reading; refine after.
  "act_fast" => "calls a tool on the first turn instead of asking"
}.freeze

Class Method Summary collapse

Class Method Details

.act_fast(turn) ⇒ Object 



220
221
222
223
224
225
226
227
 
# File 'lib/insika/evals/assertions.rb', line 220

def act_fast(turn)
  return [false, "no turn to check"] if turn.nil?

  tools = turn.tool_names
  return [true, "acted: called #{tools.join(', ')}"] unless tools.empty?

  [false, "asked instead of acting: #{turn.output_text.to_s.strip[0, 160].inspect}"]
end

.ask_once(turns) ⇒ Object

Every reply asks at most one question. Reported with the offending turn and the reply itself — "2 questions" alone sends the reader digging.



200
201
202
203
204
205
206
207
 
# File 'lib/insika/evals/assertions.rb', line 200

def ask_once(turns)
  offender = turns.each_with_index.find { |t, _| count_questions(t.output_text) > 1 }
  return [true, "at most one question per reply"] unless offender

  turn, i = offender
  [false, "turn #{i + 1} asked #{count_questions(turn.output_text)} questions: " \
          "#{turn.output_text.to_s.strip[0, 160].inspect}"]
end

.count_questions(text) ⇒ Object

Questions in ONE reply. Deliberately crude and deliberately documented: a run of "?" counts once ("já pensou??" is one question), and URLs are dropped first so a tracking link's query string is not read as the agent asking something. It is a policy signal, not grammar — and it already caught a real violation ("é pra você ou tá pensando em presentear alguém? E qual seu tamanho?").



234
235
236
 
# File 'lib/insika/evals/assertions.rb', line 234

def count_questions(text)
  text.to_s.gsub(%r{https?://\S+}, " ").scan(/\?+/).size
end

.detect(name, text) ⇒ Object

Runs a named detector over the text. "pii_leak" = union of all PII detectors; otherwise a single named pattern. Delegates to the runtime's single source (D4) — which itself fails loud on an unknown name (a typo'd assertion must not silently pass).



242
243
244
 
# File 'lib/insika/evals/assertions.rb', line 242

def detect(name, text)
  Insika::Safety::Detectors.detect(name, text)
end

.evaluate(golden, result, turns: nil) ⇒ Object

Golden + TurnResult -> CaseResult. A turn that failed to run yields a single failing check (there's nothing to assert on a turn that never produced output).

turns is every turn of the conversation, in order; result is the last one (what the tool/content assertions have always run on). The policy checks need all of them — "one question per reply" is a rule about every reply, and the violation that motivated this was on the FIRST turn. Defaults to the single result so existing callers keep working.



137
138
139
140
141
142
143
144
145
146
147
148
149
150
 
# File 'lib/insika/evals/assertions.rb', line 137

def evaluate(golden, result, turns: nil)
  if result.error
    return CaseResult.new(id: golden.id, agent: golden.agent, error: result.error, rubric: nil, judge: nil,
                          checks: [Check.new(name: "turn", pass: false, detail: "turn error: #{result.error}")])
  end

  # NOT `Array(turns)`: TurnResult is a Struct, so Array() would explode a single
  # one into its members and hand the policy checks three strings.
  conversation = turns.nil? || turns.empty? ? [result] : turns
  checks = tool_checks(golden, result) + must_not_checks(golden, result) +
           policy_checks(golden, conversation)
  CaseResult.new(id: golden.id, agent: golden.agent, error: nil, checks: checks,
                 rubric: golden.rubric, judge: nil)
end

.investigate_first(turn) ⇒ Object 



209
210
211
212
213
214
215
216
217
218
 
# File 'lib/insika/evals/assertions.rb', line 209

def investigate_first(turn)
  return [false, "no turn to check"] if turn.nil?

  tools = turn.tool_names
  return [false, "called #{tools.join(', ')} before asking anything"] unless tools.empty?
  return [false, "answered without asking: #{turn.output_text.to_s.strip[0, 160].inspect}"] if
    count_questions(turn.output_text).zero?

  [true, "asked before acting"]
end

.must_not_checks(golden, result) ⇒ Object

must_not detectors. "tool_error" is special (inspects statuses); the rest are content detectors over the output text.



167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/insika/evals/assertions.rb', line 167

def must_not_checks(golden, result)
  golden.must_not.map do |name|
    if name == "tool_error"
      bad = result.errored_tools
      Check.new(name: "must_not:tool_error", pass: bad.empty?,
                detail: bad.empty? ? "no tool errors" : "errored: #{bad.map { |t| t['name'] || t[:name] }.join(', ')}")
    else
      hit = detect(name, result.output_text.to_s)
      Check.new(name: "must_not:#{name}", pass: hit.nil?,
                detail: hit ? "matched #{hit.inspect}" : "clean")
    end
  end
end

.ok_status?(status) ⇒ Boolean

A tool status counts as success when it's blank/"ok"/"success" or a 2xx code.

Returns:

  • (Boolean) 


119
120
121
122
123
124
125
126
127
 
# File 'lib/insika/evals/assertions.rb', line 119

def ok_status?(status)
  return true if status.nil?

  s = status.to_s.strip.downcase
  return true if s.empty? || %w[ok success succeeded done].include?(s)

  code = Integer(s, exception: false)
  code ? code.between?(200, 299) : false
end

.policy_checks(golden, turns) ⇒ Object

The declared policy, checked deterministically over the conversation. No policy -> no check (and nothing to explain in the report).



183
184
185
186
187
188
189
190
191
192
193
194
195
196
 
# File 'lib/insika/evals/assertions.rb', line 183

def policy_checks(golden, turns)
  name = golden.policy
  return [] if name.nil?

  # Exhaustive on purpose: a policy added to POLICIES without a rule here would
  # otherwise fall into whichever branch was last and check the wrong thing.
  pass, detail = case name
                 when "ask_once" then ask_once(turns)
                 when "investigate_first" then investigate_first(turns.first)
                 when "act_fast" then act_fast(turns.first)
                 else raise ArgumentError, "policy #{name.inspect} has no rule"
                 end
  [Check.new(name: "policy:#{name}", pass: pass, detail: detail)]
end

.skip(golden, reason) ⇒ Object

The case did not run and MUST NOT read as either a pass or a failure.



113
114
115
116
 
# File 'lib/insika/evals/assertions.rb', line 113

def skip(golden, reason)
  CaseResult.new(id: golden.id, agent: golden.agent, checks: [], error: nil,
                 rubric: nil, judge: nil, skipped: reason)
end

.tool_checks(golden, result) ⇒ Object

Each REQUIRED expected tool must appear in the turn's tool calls. Optional ("name?") tools are informational — present or not, they never fail.



154
155
156
157
158
159
160
161
162
163
 
# File 'lib/insika/evals/assertions.rb', line 154

def tool_checks(golden, result)
  names = result.tool_names
  golden.tools_called.filter_map do |t|
    next if t[:optional]

    present = names.include?(t[:name])
    Check.new(name: "tool:#{t[:name]}", pass: present,
              detail: present ? "called" : "expected but not called (saw: #{names.join(', ')})")
  end
end

.unmet_requirements(golden, available) ⇒ Object

WHAT THIS DEPLOYMENT LACKS for the case to be worth running (RFC-0014 §3.2). -> [reason]; empty = run it.

available is the deployment's answer for this agent:

{ "tools" => [names] | nil, "capabilities" => [names] }

tools nil means an OPEN allowlist — the agent may call every registered tool, so no tool requirement can be judged missing and the case runs. That is the deliberate reading: "I could not rule it out" must not become a skip, or a permissive agent would quietly stop being tested.



99
100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/insika/evals/assertions.rb', line 99

def unmet_requirements(golden, available)
  tools = available["tools"]
  declared = Array(available["capabilities"]).map(&:to_s)

  missing_tools = tools.nil? ? [] : golden.required_tools - Array(tools).map(&:to_s)
  missing_caps = golden.required_capabilities - declared

  reasons = []
  reasons << "tool not available: #{missing_tools.join(', ')}" unless missing_tools.empty?
  reasons << "capability not declared: #{missing_caps.join(', ')}" unless missing_caps.empty?
  reasons
end