Class: Insika::Safety::Moderator

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/safety/moderator.rb

Overview

LLM content moderator. The subtle tier the regex can't reach: social engineering (a fabricated prior promise), veiled hostility, tone.

Pure over an injected ask callable (prompt -> raw model text), exactly like the eval Judge — so it is unit-testable with no LLM. The real ask (RubyLLM on the utility_model, temp 0) is built by the Safety::Factory at wiring time.

FAIL-OPEN by construction: the deterministic layer already ran and caught the gross cases, so an unparseable/failed moderator reply must NOT block a legitimate customer. Blocking is the high-stakes direction (RFC: a false positive turns away a real buyer). But fail-open is not a fake negative a moderator that could not answer returns the third state unavailable — it does not block, and it is NOT recorded as a clean allow, so a degraded tier is distinguishable from a healthy one in the audit stream.

Defined Under Namespace

Classes: Verdict

Constant Summary collapse

CATEGORIES = 
%w[injection abuse sexual self_harm off_topic safe].freeze
ACTIONS = 
%w[allow refuse escalate unavailable].freeze

Instance Method Summary collapse

Constructor Details

#initialize(ask:) ⇒ Moderator

ask: ->(prompt) { "" }.



31
32
33
 
# File 'lib/insika/safety/moderator.rb', line 31

def initialize(ask:)
  @ask = ask
end

Instance Method Details

#classify(message, context: nil) ⇒ Object

Classifies a user message. Returns a Verdict; on ANY failure -> unavailable (fail-open: never blocks, but never masquerades as a real allow either — context is optional free text (e.g. the agent's domain) woven into the prompt.



39
40
41
42
43
44
 
# File 'lib/insika/safety/moderator.rb', line 39

def classify(message, context: nil)
  raw = @ask.call(build_prompt(message.to_s, context)).to_s
  parse(raw)
rescue StandardError
  Verdict.new(category: "safe", action: "unavailable", reason: "moderator error (fail-open)")
end