Anomonitor

Rails engine gem that watches Sidekiq, Delayed Job, Solid Queue, and custom job tables for anomalies — threshold breaches, growth spikes, and schema drift — then notifies via webhook and shows an ops dashboard.

Install

# Gemfile
gem "anomonitor", github: "flori-s/anomonitor"

Requires Ruby >= 3.0 and Rails >= 6.1.

bundle install
rails generate anomonitor:install
rails anomonitor:install:migrations
rails db:migrate

Mount the engine:

# config/routes.rb
mount Anomonitor::Engine => "/anomonitor"

Configure

config/initializers/anomonitor.rb:

Anomonitor.configure do |c|
  c.webhook_url = ENV.fetch("ANOMONITOR_WEBHOOK_URL")
  c.dashboard_base_url = ENV["ANOMONITOR_DASHBOARD_BASE_URL"] # e.g. "https://ops.example.com"
  c.poll_interval = 60
  c.cooldown = 15 * 60
  c.retention_days = 7
  c.schema_drift_interval = 15 * 60

  c.collectors.sidekiq = true
  c.collectors.delayed_job = true
  c.collectors.solid_queue = true

  c.alert :queue_depth, max: 1_000
  c.alert :failed, max: 50
  c.alert :latency, max: 120
  c.alert :growth_spike, window: 5 * 60, multiplier: 3.0
end

Per-queue Sidekiq (tagged queue_depth / latency points):

c.alert :queue_depth, max: 250, match: { queue: true }
c.alert :latency, max: 60, match: { queue: "mailers" }

match values: true = tag present, nil/false = tag absent, string = exact.

Mutes

Anomonitor.config.mute(metric: "extra_tables", tenant: "acme", duration: 24.hours, reason: "migration")

Or use /anomonitor/mutes in the dashboard.

Digests

c.digest_interval = 3600       # batch detections into anomaly.digest
c.notifier_rate_limit = 30     # max delivers per minute

Dashboard auth

The engine is open by default. Protect it:

Anomonitor.configure do |c|
  c.authenticate = -> {
    authenticate_or_request_with_http_basic("Anomonitor") do |user, pass|
      ActiveSupport::SecurityUtils.secure_compare(user, ENV.fetch("ANOMONITOR_USER")) &&
        ActiveSupport::SecurityUtils.secure_compare(pass, ENV.fetch("ANOMONITOR_PASSWORD"))
    end
  }
end

Or reuse host auth:

c.authenticate = -> { redirect_to main_app.root_path unless current_user&.admin? }

You can also constrain the mount in routes (authenticate :user, ->(u) { u.admin? } do ... end).

Multi-tenancy + schema drift

For Apartment / schema-per-tenant apps, set tenants so Delayed Job is collected per tenant and enable schema drift (PostgreSQL / information_schema):

Anomonitor.configure do |c|
  c.tenants = -> { CustomerTenant.pluck(:name) }
  c.exclude_tenants = %w[public]
  # optional — defaults to Apartment::Tenant.switch when Apartment is loaded
  c.tenant_switch = ->(name, &block) { Apartment::Tenant.switch(name, &block) }

  c.collectors.delayed_job = true
  c.collectors.schema_drift = true
  c.schema_drift_interval = 15 * 60 # heavier than queue polls; default 15m
  c.schema_drift_exclude = %w[
    schema_migrations
    ar_internal_metadata
    anomonitor_*
    a*
    conv_*
  ]

  c.alert :queue_depth, max: 1_000
  c.alert :missing_tables, max: 0
  c.alert :extra_tables, max: 0
  c.alert :missing_columns, max: 0
  c.alert :extra_columns, max: 0
end

schema_drift_exclude accepts exact names or File.fnmatch globs (a* matches all a-prefixed tables; a_* only matches a_…).

Schema drift webhooks are sticky: one anomaly.detected per tenant/metric/item-set, silence until clear or the set changes, then anomaly.resolved when it clears. Queue threshold and growth alerts still use c.cooldown (default 15 minutes).

Custom tables

Status-based queues:

c.table :customer_jobs do |t|
  t.model = "Job"
  t.timestamp = :created_at
  t.status = :status
  t.active = %w[pending running]
end

Cross-tenant Delayed Job index (host provides an AR model, e.g. SuppJob → public.supp_jobs):

c.table :supp_jobs do |t|
  t.model = "SuppJob"
  t.style = :delayed_job   # failed_at / locked_at / run_at / locked_by
  t.tenant = :tenant       # emit per-tenant metrics
end

Webhooks

Built-in: set ANOMONITOR_WEBHOOK_URL. Set ANOMONITOR_DASHBOARD_BASE_URL (or c.dashboard_base_url) so Slack/JSON links are absolute.

Custom notifier (e.g. your app’s Webhook::Broadcast proxy) — replaces the built-in HTTP client:

Anomonitor.configure do |c|
  c.notifier = ->(anomaly, event:) {
    Webhook::Broadcast.new(
      urls: [{ url: ENV.fetch("ANOMONITOR_DEST_URL"), headers: [
        { name: "Content-Type", value: "application/json" }
      ] }],
      message: Anomonitor::Notifiers.payload(anomaly, event: event)
    ).call
  }
end

Callables may return true/false, or a Hash with :accepted (Broadcast’s return value). Objects with #deliver(anomaly, event:) also work; pass an Array to fan out to several notifiers.

Slack Incoming Webhooks (hooks.slack.com) get a formatted text payload when using the built-in notifier. Other built-in URLs receive JSON:

{
  "gem": "anomonitor",
  "event": "anomaly.detected",
  "severity": "high",
  "rule": "growth_spike",
  "source": "delayed_job",
  "metric": "queue_depth",
  "value": 4200,
  "threshold": 1000,
  "sampled_at": "2026-08-07T09:00:00Z",
  "resolved_at": null,
  "dashboard_url": "https://ops.example.com/anomonitor/anomalies/123",
  "tags": { "tenant": "acme" }
}

Events: anomaly.detected and anomaly.resolved (sticky schema drift clear).

Polling: thread vs cron

Anomonitor can collect on a background thread or via an external scheduler.

poll_mode	Behavior
:thread (default)	Starts an in-process poller after boot (poll_interval seconds). Avoid with multi-worker Puma/Unicorn unless only one worker should run it. Prefer :cron in multi-worker setups.
:cron	No background thread. Schedule rails anomonitor:poll yourself.

# In-process (default)
c.poll_mode = :thread

# Cron / systemd / Kubernetes CronJob / whenever
c.poll_mode = :cron

Cron examples:

* * * * * cd /app && bin/rails anomonitor:poll

# config/schedule.rb (whenever)
every 1.minute do
  rake "anomonitor:poll"
end

c.auto_start = false is still supported and is equivalent to poll_mode = :cron.

Samples and resolved anomalies older than retention_days are pruned (open schema-drift rows are kept until resolved).

Dashboard

Open /anomonitor for:

• Current metric cards and mini history bars
• Jobs — per-collector health plus a read-only live job browser (filter by source, status, tenant, queue, search; expandable detail)
• Open anomalies (filter open / resolved / all) + resolve/ack, reopen, retry webhook
• Mutes — snooze by metric/rule/source/tenant
• Overview tenant filter
• Poller health (last run, collector status)

Metrics export

• JSON: /anomonitor/metrics.json
• Prometheus text: /anomonitor/metrics.prom

Retry failed webhooks

rails anomonitor:retry_webhooks

c.poll_lock = true (default) skips overlapping ticks via Postgres advisory lock or a tmp file lock. Prefer poll_mode = :cron in multi-worker setups.

Manual poll

rails anomonitor:poll

With Apartment, the rake task switches to the first exclude_tenants entry (default public) before collecting.

Development

bundle install
bundle exec rake test

License

MIT