Module: Protege::Subscribers::Alerter

Extended by:
Alerter
Included in:
Alerter
Defined in:
lib/protege/subscribers/alerter.rb

Overview

The operational-alert subscriber — emails the platform admin when an inference run fails. A peer of Subscribers::Tracing / HookDispatcher / IntrospectionBroadcaster, installed once at boot. Subscribes to InferenceFailedEvent and, when a host has opted in via config.failure_alerts, sends a Protege::AlertMailer notice carrying the error and the run's correlation id.

Opt-in and best-effort by design:

  • Off until config.failure_alerts[:to] is set, and it will not send without a :from — there is no deliverable default sender (a real :from must be a DKIM-signable address at a registered EmailDomain), so a :to with no :from is a misconfiguration that logs and sends nothing.
  • InferenceFailedEvent fires once per failed run, and the job layer retries a transient failure up to three times — so a single failing message emits the event several times, all sharing one correlation id. Delivery is deduped on that id so the retry storm collapses to one alert, and a global per-window cap guards against a systemic outage flooding the admin.
  • The event is emitted from inside the Harness's rescue, right before it re-raises, and ActiveSupport::Notifications propagates subscriber exceptions — so this module rescues everything internally. An alert failure must never mask the inference error that triggered it.

Constant Summary collapse

DEDUP_TTL =

How long a run's correlation id suppresses duplicate alerts (covers the retry backoff window).

1.hour
GLOBAL_KEY =

Cache key + ceiling + window for the systemic-outage flood guard.

'protege:alert:global'
GLOBAL_CAP = 
10
GLOBAL_TTL = 
1.hour

Instance Method Summary collapse

Instance Method Details

#install! ⇒ void

This method returns an undefined value.

Subscribe the alerter to inference failures — the subscriber's install! contract. Idempotent; called once at boot from the engine's after_initialize.



40
41
42
43
44
 
# File 'lib/protege/subscribers/alerter.rb', line 40

def install!
  return unless @handles.empty?

  @handles = [InferenceFailedEvent.subscribe { |event| alert(event) }]
end

#reset! ⇒ void

This method returns an undefined value.

Detach every subscription — the subscriber's reset! contract, for boot re-wiring and tests.



49
50
51
52
 
# File 'lib/protege/subscribers/alerter.rb', line 49

def reset!
  @handles.each { |handle| ActiveSupport::Notifications.unsubscribe(handle) }
  @handles = []
end