Wide Events

Gem Version CI

One wide telemetry event per Rails request or job execution, in a database you own.

Wide Events collects everything your app knows about each unit of work (route, user, account, build SHA, query counts, cache hits, feature flags, phase timings, errors) into one flat, high-cardinality event on the OpenTelemetry root span you already export. Every request becomes one row, and every question becomes one query against storage you run. ClickHouse with HyperDX is a proven pairing (both open source); any OTLP backend works. If you'd rather not run tracing, the log sink emits one JSON event per request.

That format matters more now that agents build with you. Telemetry stops being something humans only glance at and becomes something software queries repeatedly: an agent can instrument a feature, deploy it, and verify it in production. One row per request is a dense way to feed production behavior back into a context window, and owning the storage avoids metered per-event and per-query observability fees.

Install

# Gemfile
gem "wide_events"
gem "useragent"  # optional: parses user_agent.browser/os/platform from the raw UA string

bin/rails generate wide_events:install   # initializer + attribute registry + AGENTS.md section + test helper wiring
bin/rails generate wide_events:skills    # agent skills into .claude/skills/

See an event in 60 seconds

Wide events are off unless OTEL_EXPORTER_OTLP_ENDPOINT is set (the development log tells you so at boot). To see one immediately, no tracing required, enable the log sink in config/initializers/wide_events.rb:

WideEvent.configure do |config|
  config.enabled = true
  config.sink = :log
end

Hit any route, then:

grep '"main":true' log/development.log

One JSON object per request: route, status, timings, query counts, everything.

Production setup with OpenTelemetry

The default sink writes the event onto the current OTel root span, which is the span OTel's Rack instrumentation opens at the top of the middleware stack. That means production needs an OpenTelemetry SDK configured; the gem detects it and wires itself after your initializers run. Minimal setup:

# Gemfile
gem "opentelemetry-sdk"
gem "opentelemetry-exporter-otlp"
gem "opentelemetry-instrumentation-rails"
gem "opentelemetry-instrumentation-rack"
gem "opentelemetry-instrumentation-pg"        # enables stats.postgres_query_*
gem "opentelemetry-instrumentation-net_http"  # enables stats.http_call_*

# config/initializers/opentelemetry.rb
return if ENV["OTEL_EXPORTER_OTLP_ENDPOINT"].to_s.empty?

require "opentelemetry/sdk"
require "opentelemetry/exporter/otlp"
require "opentelemetry/instrumentation/rails"
require "opentelemetry/instrumentation/rack"
require "opentelemetry/instrumentation/pg"
require "opentelemetry/instrumentation/net/http"

OpenTelemetry::SDK.configure(&:use_all)

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318   # your collector or backend
OTEL_SERVICE_NAME=myapp
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

With the endpoint set, wide events enable themselves: the railtie inserts the middleware directly below ActionDispatch::Executor (the executor clears the per-request store, so placement matters), instruments ActiveJob::Base, and registers the span counter and notification subscribers via WideEvent.install! after your initializers run. Without Rails, call WideEvent.install! yourself after configuring the SDK.

What you get per event

  • http.request.id, http.response.status_code, http.route.controller, request body size, parsed user_agent.*
  • db.duration_ms and view.duration_ms (Rails' own measurements)
  • stats.postgres_query_count / _duration_ms and stats.http_call_count / _duration_ms, rolled up from OTel child spans (scope map is configurable)
  • cache.<prefix> hit/miss booleans, capped per event
  • job.class, job.queue, job.queue_latency_ms, job.executions, job.scheduled (Solid Queue recurring detection built in, detector pluggable)
  • error, exception.type, exception.message, uptime_sec, main: true

A request and the jobs it enqueues are separate units of work. Synchronous enqueue work contributes to the request's total duration; each job execution emits its own event carrying job.queue_latency_ms (time spent waiting in the queue), so queue pressure is visible per job rather than folded into request latency.

Adding your own attributes

WideEvent.set("report.id" => report.id, "report.format" => "pdf")
WideEvent.phase("pdf_render") { render_pdf }         # -> pdf_render.duration_ms

Attribute writes are safe no-ops outside a unit of work; phase still executes and returns the wrapped block. Telemetry failures never raise into app code, so a telemetry bug cannot fail a request or job.

Errors come in two shapes, and the difference is the point:

# Handled: you rescued it, you name it
rescue Vendor::ApiError => e
  WideEvent.error!(slug: "err-vendor-sync-failed", exception: e, expected: true)

# Unhandled: any exception that escapes gets error: true with NO slug, automatically

Filtering for error = true, a missing exception.slug, and a present exception.type isolates exception paths that escaped without a named handled failure. Treat the results as a prioritized investigation queue; add a rescue and error! only when handling the exception is intentional.

The registry is a schema, not a wiki page

Every attribute is declared in config/wide_event/registry.yml (the gem's own attributes are pre-registered; globs like feature_flag.* cover dynamic families):

report.format:
  type: string
  set_by: ReportsController#create
  pii: none
  notes: pdf or csv

The workflow, end to end:

  1. Set the attribute in code, declare it in registry.yml, same change.
  2. Assert it in a test with assert_wide_event.
  3. Strict mode (config.strict = Rails.env.test?, the generated default) records any undeclared attribute the suite sees; assert_registered_wide_event_attributes fails on them.
  4. CI runs bin/rails wide_events:registry:check to validate the file.
  5. bin/rails wide_events:registry:docs generates docs/wide-events-registry.md from it, so documentation can't drift from reality.
# .github/workflows/ci.yml (host app)
- run: bin/rails wide_events:registry:check
- run: bin/rails test

Testing

The install generator wires this into test/test_helper.rb (or tells you how):

require "wide_event/test_helper"

class ActiveSupport::TestCase
  include WideEvent::TestHelper
  teardown { assert_registered_wide_event_attributes }
end

test "report rendering is instrumented" do
  assert_wide_event("pdf_render.duration_ms", "report.format" => "pdf") do
    Report.new(format: "pdf").render
  end
end

test "job emits one wide event" do
  events = capture_wide_events { ExportJob.perform_now }
  assert_equal 1, events.length
end

The teardown pattern makes the registry check work with parallel test workers: violation tracking is per process, and checking (then resetting) after every test means the test that set an undeclared attribute is the one that fails, in whichever worker it ran.

When events are emitted

Environment Typical setup Result
Production / staging OTEL_EXPORTER_OTLP_ENDPOINT set Enabled automatically, :otel sink, events on root spans
Development config.enabled = true, config.sink = :log One JSON line per request in the log
Test Enabled via the test helper's capture, or strict mode for registry tracking capture_wide_events collects them

WideEvent.install! runs once at boot, only if enabled at that point. Flipping config.enabled at runtime gates the middleware and job hook (they check per request), but the span counter and subscribers only register at boot, so treat enablement as a boot-time decision.

Built for the agent loop

bin/rails generate wide_events:skills installs two agent skills:

  • instrumenting-wide-events: the write path. Naming conventions, when to use set vs phase vs error!, the registry workflow, PII rules, test assertions.
  • debugging-with-wide-events: the read path. Symptom-to-query workflow against ClickHouse/HyperDX or JSON logs, plus the standing queries worth running.

Skills land in .claude/skills/, which Claude Code and Claude-compatible agents read. Other agents (Cursor, etc.) read the pointer the install generator appends to AGENTS.md; the SKILL.md files are plain markdown, so copy or symlink them wherever your tooling looks (.cursor/rules/, docs, a system prompt).

Configuration

WideEvent.configure do |config|
  config.enabled = ENV["OTEL_EXPORTER_OTLP_ENDPOINT"].present?  # default
  config.sink = :otel               # :otel, :log, or any object responding to flush(attrs)
  config.logger = nil               # :log sink target; defaults to Rails.logger
  config.strict = Rails.env.test?   # track attributes against the registry
  config.max_cache_attrs = 10
  config.span_scopes = {            # OTel instrumentation scope -> stats.* name
    "OpenTelemetry::Instrumentation::PG" => "postgres_query",
    "OpenTelemetry::Instrumentation::Net::HTTP" => "http_call"
  }
  # Optional overrides; defaults detect Solid Queue recurring executions and report via OpenTelemetry.handle_error:
  # config.scheduled_job_detector = ->(job) { MyApp.scheduled_job?(job) }
  # config.error_handler = ->(exception, message) { Rails.error.report(exception, context: { message: message }) }
end

Log sink notes: events go to config.logger (or Rails.logger) at info level as single-line JSON. Point it at a dedicated logger (config.logger = Logger.new("log/wide_events.log")) to keep them out of your main log, and leave the sink off in the test environment unless you want one JSON line per test request; capture_wide_events is the intended test-side tap.

Querying: ClickHouse + HyperDX

docs/clickhouse-hyperdx.md has a local quickstart (one container) and a query cookbook. The flavor:

SELECT SpanAttributes['http.route.controller'] AS controller,
       count() AS requests,
       round(quantile(0.5)(Duration/1e6)) AS p50_ms,
       round(avg(toFloat64OrZero(SpanAttributes['stats.postgres_query_count']))) AS avg_queries
FROM otel_traces
WHERE SpanAttributes['main'] = 'true'
GROUP BY controller ORDER BY p50_ms DESC LIMIT 15

In production

Wide Events was extracted from a multi-tenant Rails app subject to healthcare privacy rules, where it runs in staging and production today: one row per request and job execution, exported over OTLP into self-hosted ClickStack. A request that runs a hybrid search and drafts a reply with an LLM still lands as one event, carrying the account, the feature flags evaluated, search quality (semantic hits kept, top cosine similarity), the model, tokens, latency, and cost, the Postgres query count, and an error slug for anything rescued along the way. docs/production-example.md shows a full anonymized event, the instrumentation patterns behind it, and two standing queries taken from that deployment.

Conventions

Flat keys, dot namespaces, snake_case leaves. Durations end in _duration_ms, counts in _count, booleans read as assertions, timestamps serialize to RFC 3339. Apply your deployment's privacy policy to every identifier: opaque IDs are preferable to names or emails but should still be registered as pii: opaque_id. Exclude request params and free text, or flag them as pii: review for explicit handling.

Development

bin/setup                 # bundle install + appraisal gemfiles
bundle exec rake test     # run the suite
bin/rubocop               # lint (rubocop-rails-omakase)

The CI matrix runs the suite across Ruby 3.2 to 4.0 and Rails 7.1 to main via Appraisal; run a specific combination locally with BUNDLE_GEMFILE=gemfiles/rails_7_1.gemfile bundle exec rake test. Releases go out with bin/release <version>.

Requirements

Ruby >= 3.2, Rails >= 7.1 (activesupport and rack are the only hard dependencies; opentelemetry-sdk and useragent are optional and detected at runtime).

License

MIT.