foam-otel (Ruby)

Foam's OpenTelemetry core for Ruby services. A thin, safe wrapper over the official OpenTelemetry libraries: foam owns the pipeline (providers, batch processors, OTLP export to the foam fleet endpoint), turns on automatic tier-1/2 instrumentation, and hands you a small set of never-throw helpers — and nothing else. Redaction is fully opt-in above two always-on exceptions: by default foam captures every value RAW (no PII preset) except (1) the credential floor — a fixed list of credential/secret header and field NAMES always masked to [REDACTED] (see "The default credential denylist" below) — and (2) the value-pattern secret layer — credential-SHAPED value spans (cloud keys, tokens, JWTs, private-key blocks, user:pass@ URLs…) masked to [REDACTED] regardless of field name (see "The value-pattern secret layer" below); you enumerate any further fields to mask via redact_keys/redact_pii_keys.

Built to docs/BASE_PACKAGE_SPEC.md. This README is the manual you integrate from.

Foam builds on the OpenTelemetry authors' work (https://opentelemetry.io) — see THIRD-PARTY-NOTICES. Licensed, not sold — see LICENSE.

Install

# Gemfile
gem "foam-otel"
# foam-otel bundles the UNIVERSAL FLOOR and auto-activates each piece when its
# target library is present (presence-checked, and gated on foam owning the
# slot it produces into): Rack + the Rails family, the HTTP clients (Net::HTTP,
# Faraday), the primary datastores (pg, mysql2, redis, mongo) with RAW
# db.statement capture, Sidekiq, the stdlib Logger bridge, hand-written Ruby
# runtime + GC metrics, session stitching (browser session.id via baggage),
# and the LLM shims (OpenAI, Anthropic, Gemini, ruby_llm — activity + raw
# content). Add only the niche long tail your app needs beyond the floor:
gem "opentelemetry-instrumentation-graphql"  # e.g. GraphQL
gem "opentelemetry-instrumentation-resque"   # e.g. Resque

Supported versions (tested in CI, spec/version_spec.rb): Ruby >= 3.1; OpenTelemetry API opentelemetry-api ~> 1.1 (1.x). The metrics and logs SDKs are pre-1.0 upstream (metrics alpha, logs development) and are pinned with pessimistic constraints; foam wraps them behind stable helpers so app code never touches the churning API (see GOTCHAS R2).

Quick start — scenario 1: a clean service (the common case)

Put init FIRST, before your app boots. In Rails, that means an initializer (and, for cluster mode, per worker — see "Puma / Unicorn" below).

# config/initializers/foam.rb  (the one config module you own)
require "foam/otel"

Foam::Otel.init(
  name: "checkout-api",                       # → service.name
  environment: ENV.fetch("APP_ENV", "development"),  # → deployment.environment.name (verbatim)
  enabled: !%w[test ci].include?(ENV["APP_ENV"]),    # silence tests/CI; export everywhere else
  token: ENV.fetch("FOAM_OTEL_TOKEN"),             # required WHEN enabled — wire it explicitly
  version: ENV["GIT_SHA"]                      # → service.version (recommended; optional)
)

Outcome: foam registers the tracer/meter/logger providers, exports OTLP to the foam fleet endpoint, and auto-activates the universal floor — the official rack/rails instrumentation plus the bundled HTTP (Net::HTTP/Faraday), datastore (pg/mysql2/redis/mongo), and Sidekiq instrumentations — each presence-gated, and gated on foam owning the traces slot (a foreign-owned tracer stands the activation down). The datastore set is activated with db_statement: :include — foam explicitly overrides the upstream :obfuscate default so RAW SQL / redis args / mongo commands land on the wire. On top of the contrib set the floor adds foam's own pieces: the stdlib Logger bridge (every host Logger line also lands as a trace-correlated OTel log record, body raw), hand-written Ruby runtime + GC metrics (GC counts + timing, heap slots, allocation, thread count, RSS — Ruby has no drop-in upstream metrics gem, so foam ships its own collector), session stitching (inbound baggage: session.id from the foam browser package is stamped onto every span and log record), and the LLM shims (below). Every inbound request, DB call, HTTP call, job, log line, and model call now flows to foam. Redaction is opt-in above the credential floor: by default every value — attributes, URLs/query strings, DB statement text, log bodies, LLM content — is captured RAW, except values whose NAME is on the always-on credential denylist (masked [REDACTED]; see "The default credential denylist" below). Pass redact_keys/redact_pii_keys to mask or erase further specific fields (see below).

Sidekiq trace shape (GOTCHAS G16): the bundled instrumentation keeps its upstream default propagation_style: :link — a performed job runs as its OWN trace whose first span LINKS back to the enqueuing span (context is always propagated across the hop; proven against a real Redis in spec/floor_sidekiq_wire_spec.rb). Prefer one continuous web→job trace? Set the standard contrib lever: OTEL_RUBY_INSTRUMENTATION_SIDEKIQ_CONFIG_OPTS='propagation_style=child'.

The LLM surface — OpenAI, Anthropic, Gemini (and ruby_llm)

The Ruby contrib registry ships no LLM instrumentation, so foam does: thin, presence-checked shims over the SDKs' public call sites, emitting the standard gen_ai.* semantic conventions — activity (model, response id, finish reasons, token usage, latency) AND prompt/response content, RAW (gen_ai.input.messages / gen_ai.output.messages / gen_ai.system_instructions). Agent tool loops ride in full: tool-call turns keep their tool names/arguments and tool-result turns keep their tool_call_id linkage inside the message content. gen_ai.operation.name (and the {operation} {model} span name) follows the semconv operation for each API shape — chat for the OpenAI/Anthropic/ruby_llm chat seams, generate_content for Gemini's generateContent, fleet-identical with the js/python Gemini paths (wire change in 1.2.1: Gemini spans previously said chat). Covered: the official openai SDK (chat.completions.create + responses.create), the official anthropic SDK (messages.create), Gemini via the gemini-ai gem (Google ships no official Ruby SDK), and ruby_llm (one seam covering all its providers). Absent SDK = inert; version-guarded; fail-to-dark — a shim failure can NEVER break the model call, and the SDK's own error re-raises identically. One model call = one gen_ai client span (the SDK's HTTP send rides the same suppression context foam's exporters use). Streaming call sites (#stream/#stream_raw on the official SDKs) are a recorded follow-up.

Session stitching — browser `session.id` on backend telemetry

The foam browser package stamps session.id on every browser signal and sends it to your API as W3C Baggage beside traceparent. This gem's floor copies session.id (and session.previous_id across a rotation) from the inbound baggage onto every span and log record foam emits — browser session → backend traces/logs, stitched with no ingest-side join. No-op when the baggage is absent. CORS note (the FDE wires this on every frontend-called API): the allowlist must include BOTH headers — Access-Control-Allow-Headers: traceparent, baggage.

The `init` options

Option	Type	Required	Default	What it does
`name:`	String	yes	—	`service.name`. Blank → raises at boot.
`environment:`	String	yes	—	`deployment.environment.name`, exported verbatim. A value outside `{production, staging, development, test}` warns but is never rewritten. Blank → raises.
`enabled:`	Boolean	yes	—	The config switch. `false` = fully inert (no SDK, no providers, no network, helpers no-op). `true` = export, in EVERY environment. No default — you write the logic.
`token:`	String	when enabled	—	`Authorization: Bearer` for export. Required (and validated) only when `enabled: true`; wired explicitly from your secret source — never an env fallback read by the gem. Fleet convention: read it from the `FOAM_OTEL_TOKEN` env var. Blank while enabled → raises at boot.
`version:`	String	no	nil	`service.version`, verbatim (git SHA recommended). Missing → warns and continues. Never detected at runtime.
`redact_keys:`	Array	no	`[]`	The ONLY field names tail-masked (shape-preserving, e.g. `********f456`). Opt-in and ADDITIVE on top of the always-on credential floor — empty (the default) means no masking beyond the floor. Matches by case-insensitive substring, including keys nested in structured values and query-string keys. A key that is also on the floor stays fully `[REDACTED]` (the floor wins; it is never downgraded to a tail).
`redact_pii_keys:`	Array	no	`[]`	Field names fully erased to `[REDACTED]` (no tail). Opt-in and additive above the floor — empty (the default) means no erasure beyond the floor. Foam ships no PII preset; this is your enumeration.
`additional_span_processors:`	Array	no	`[]`	Tenant seam: constructed SpanProcessor instances added to foam's pipeline (additive; never replace foam's export). See coexistence.
`additional_log_record_processors:`	Array	no	`[]`	Tenant seam, logs.
`additional_metric_readers:`	Array	no	`[]`	Tenant seam, metrics.
`additional_instrumentations:`	Array	no	`[]`	Constructed tier-2 instrumentation instances to register (fault-isolated: one that throws is skipped with a `[foam]` warning).
`ignored_outbound_hosts:`	Array	no	`[]`	Hosts whose outbound calls produce no spans — EXTENDS the built-in export-loop guard. For a co-resident agent's intake or a tenant exporter's endpoint. Applies to the clients whose official instrumentation supports host suppression: Net::HTTP and Excon. Faraday/HTTP (httprb)/HTTPX have no upstream `untraced_hosts` option — foam warns loudly at init when one of those is bundled (GOTCHAS G13); foam's own export loop is guarded for every client regardless.
`diagnostics:`	Boolean	no	false	Verbose `[foam]` self-reporting of init/health. Warnings and errors are always loud regardless.
`secret_heuristics:`	Boolean	no	true	The value-pattern secret layer's HEURISTIC tier (generic keyword+entropy detection, see "The value-pattern secret layer"). `false` disables ONLY the heuristics — the named provider patterns, the credential floor and the redaction-coverage contract have no off switch. Disabling logs one loud `[foam]` line at init (an explicit, audited decision for telemetry whose legitimate values collide with the heuristics).

Examples for the remaining options (each is runnable as-is):

# Redaction is OPT-IN (default: nothing is masked). List the fields to redact:
Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
                redact_keys: %w[password api_key internal_ref voucher_code], # tail-masked
                redact_pii_keys: %w[customer_email full_name])               # fully [REDACTED]

# Only if your legitimate telemetry collides with the generic secret
# heuristics (e.g. base64 content-addressed ids masked as high-entropy
# tokens): disable the HEURISTIC tier only. The named provider patterns and
# the credential floor stay on — they have no off switch. Logs one loud
# [foam] line.
Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
                secret_heuristics: false)

# Constructed tier-2 instrumentation instances (fault-isolated; one that
# throws is skipped with a [foam] warning, never a crashed boot):
Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
                additional_instrumentations: [
                  OpenTelemetry::Instrumentation::Sidekiq::Instrumentation.instance,
                  [OpenTelemetry::Instrumentation::PG::Instrumentation.instance, { db_statement: :omit }],
                ])

# Tenant seam, logs + metrics (spans are shown in scenario 3 below):
Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"),
                additional_log_record_processors: [EvalTool::LogRecordProcessor.new],
                additional_metric_readers: [EvalTool::MetricReader.new],
                ignored_outbound_hosts: ["ingest.eval-tool.example"]) # the tenant loop guard — required

# Verbose self-reporting while wiring foam up (warnings are loud regardless):
Foam::Otel.init(name: "checkout-api", environment: "development", enabled: true,
                token: ENV.fetch("FOAM_OTEL_TOKEN"), diagnostics: true)

And the unhappy path — a blank required argument fails AT BOOT, on your machine, never silently:

Foam::Otel.init(name: "", environment: "production", enabled: true, token: ENV.fetch("FOAM_OTEL_TOKEN"))
# => raises ArgumentError: Foam::Otel.init requires a non-empty name

Deliberately absent (and why): endpoint, cadence/sampling, is_production, force_export, disabled_environments, and any processor/exporter injection beyond the tenant seam. Customers identify their service and turn foam on or off; they can never drop a share of the data, inject their own pipeline pieces, or redirect export from code. The only endpoint override is the operator-level OTEL_EXPORTER_OTLP_ENDPOINT env var (below).

The default credential denylist (the always-on floor)

Fleet ruling 2026-07-26 (binding design: docs/decisions/credential-denylist-design.md; frozen fixture: contract/credential-denylist.json). This is the ONE exception to foam's raw-by-default capture, and it has no off switch — no init option, no env var, and no door-2 parameter can disable, shrink, or re-spell it.

What is masked. The VALUE of every header/attribute/field whose NAME is on the frozen list, in every signal (spans — events and links included — logs, metrics), on door 1, on all three door-2 ingest taps, and in the view handed to tenant additional_* processors:

(a) the seven credential headers, matched as header names wherever headers are captured — including the semconv http.request.header.<name> / http.response.header.<name> span-attribute forms: authorization, proxy-authorization, cookie, set-cookie, x-api-key, x-auth-token, www-authenticate;
(b) the frozen 52-entry key list of contract/credential-denylist.json (the deduplicated union of those seven, sentry-python's default denylists, and foam's documented reference roots — password, token, secret, api_key, ssn, jwt, private_key, connect.sid, phpsessid, …), matched as attribute/field names at every depth the engine walks, and per-key in URL query strings (?token=x&user=bob → token=[REDACTED]&user=bob).

The matching rule. Case-insensitive, dash/underscore-normalized EXACT name-equality — never substring: authorization masks the header/field authorization (any casing, -≡_) and nothing else — authorization_url, x_api_key_id, secretary, and session_count all ride RAW. Dotted entries match verbatim and whole (connect.sid masks; user.session does not). This is deliberately STRICTER matching than your own redact_keys lists, which keep their documented substring behavior.

The mask. The full literal [REDACTED] — no tail, no length preservation. These are credentials, not debug aids. Semconv header attributes (string arrays) mask per element, preserving arity.

Precedence. The floor runs unconditionally BEFORE your redact_keys/redact_pii_keys, which stay purely additive on top. Listing a floor name in redact_keys does NOT downgrade it to a tail mask — the floor is terminal. enabled: false stays fully inert exactly as before (nothing exports at all — the floor masks data that leaves; inert mode has none).

Everything else stays RAW — above the floor and the value layer. The floor matches NAMES only, never values. Credential-SHAPED value spans are the value-pattern secret layer's job (next section); UNSHAPED values — bodies, query values, SQL text, LLM content, and PII under unlisted names — export byte-identical raw, exactly as documented everywhere else in this README.

Why. Coverage-over-masking remains the mission, but raw Authorization headers, cookies, and API keys on the wire are a breach in waiting for every customer at once — the fleet ruling carves out exactly this list (and only this list) as non-negotiable. The list is byte-identical in every foam core and gate-checked in CI against the fleet fixture (spec/credential_floor_spec.rb).

Fleet param-name canon (the customer options above the floor, identical concept in every core — the floor itself has NO init surface anywhere):

Concept	js/otel	js/browser	python	ruby	java
secret keys (tail-mask)	`redactKeys`	`redactKeys`	`redact_keys`	`redact_keys:`	`.redactKeys(String…)`
PII keys (full `[REDACTED]`)	`redactPiiKeys`	`redactPiiKeys`	`redact_pii_keys`	`redact_pii_keys:`	`.redactPiiKeys(String…)`

Migration note — 1.3.0 (minor): always-on credential masking. As of this version foam masks, by default and in every signal, the VALUES of a fixed list of credential/secret header and field NAMES (authorization, cookie, set-cookie, proxy-authorization, x-api-key, x-auth-token, www-authenticate, and the 52-name key list in contract/credential-denylist.json — Sentry-parity plus foam's documented roots) to the literal [REDACTED]. Matching is exact name-equality, case-insensitive, dash/underscore-insensitive — never substring: authorization_url is untouched. Everything else still exports RAW exactly as before; redact_keys/redact_pii_keys are unchanged and additive. There is no off switch — if a dashboard keyed off a raw credential value (it should not have), it will now see [REDACTED]. Customers who previously received these header/field values raw stop receiving them.

The value-pattern secret layer (always-on)

The credential floor masks by NAME and is deliberately value-shape-blind — a live AWS key under the field name note would sail past it. Foam telemetry is read downstream by LLMs, so a leaked credential is exfiltratable by prompt injection; "no leakage at all" is the bar. The value-pattern secret layer is the second always-on control (fleet ruling 2026-07-27, docs/decisions/security-fixes-design.md):

What it masks. Credential-SHAPED value spans, regardless of the field name they hide under: AWS access-key ids and keyword-anchored AWS secrets, GCP API keys, Azure client/storage secrets, GitHub tokens and fine-grained PATs, GitLab tokens, Slack tokens + webhooks, Stripe secret keys (publishable pk_ keys are deliberately NOT masked), JWTs, PEM private-key blocks (-----BEGIN … PRIVATE KEY — a truncated block masks to the end of the value, fail-closed; PUBLIC KEY/CERTIFICATE blocks never match) and PuTTY PPK keys, scheme://user:pass@ URI credentials (the whole userinfo, username included), Bearer/Basic auth values, and Anthropic/OpenAI API keys. The matched span becomes the literal [REDACTED] — no tail, no prefix, zero reconstructable bytes; surrounding text keeps its diagnostic value (conn to postgres://[REDACTED]@db:5432 refused).

The heuristic tier additionally catches the long tail: a keyword-anchored generic pass (password = hunter2SecretXyz99 inside a log body masks its value — gitleaks-derived, entropy-gated, with placeholder/ stopword/key-context false-positive suppression) and a whole-value high-entropy base64 token pass (detect-secrets-derived; pure hex and UUIDs are deliberately exempt — trace ids and digests are telemetry's ambient vocabulary). Only this tier is disableable: secret_heuristics: false (one loud [foam] line; the named patterns above are not configurable).

Where it runs. Every value-bearing string on every export path, both doors: span/event/link attribute values, status.message, exception messages, log bodies (string and structured leaves), SQL (db.statement), LLM content (BEFORE serialization and again at the boundary), URL query AND fragment pair values, baggage-derived attributes, exemplar filtered_attributes, and resource identity strings (once, at init). Metric datapoint attributes (metric labels) are the one exempt surface — low-cardinality by construction; a value mask there would break your aggregation dimensions.

Bounded by design. Compile-once rules, a literal-anchor pre-filter (no anchor in the value → zero regexes run), a 256 KiB value cap (an oversized value with an anchor hit anywhere masks WHOLE, fail-closed; without one it passes untouched), a match-flood cap, a 256-pair query/fragment tokenizer cap, and a per-Regexp timeout on Ruby ≥ 3.2. ANY scanner fault masks the value rather than shipping it raw.

Provenance. The ruleset derives from MIT/Apache-licensed OSS secret scanners (gitleaks, detect-secrets, secretlint) — see THIRD-PARTY-NOTICES. Rules are frozen module constants, byte-consistent across the foam cores, changed only by fleet ruling.

Migration note — 1.4.0 (minor): value-pattern secret masking + redaction coverage fix + TLS pin. foam now masks credential-SHAPED values (AWS/GCP/Azure keys, GitHub/GitLab tokens, Slack, Stripe, Anthropic/OpenAI keys, JWTs, private-key blocks, user:pass@ URI credentials, bearer/basic tokens) to [REDACTED] in every exported string value — regardless of field name — plus a generic keyword+entropy heuristic you can disable with secret_heuristics: false (the named patterns and the credential floor cannot be disabled). Redaction now also reaches URL FRAGMENTS (OAuth implicit-flow tokens, hash-router params), bare k=v pair lists, nested attribute containers (fail-closed), and structured LLM content BEFORE serialization. Query/fragment tokenizing is capped at 256 pairs (the overflow remainder masks whole). Metric labels are not value-scanned. TLS peer verification is now PINNED on every exporter: OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE is INERT and loudly warned about. If a dashboard keyed off a raw token value (it should not have), it will now see [REDACTED].

The helpers

All helpers never raise, and no-op silently before init and when disabled.

Lifecycle

Foam::Otel.flush       # force-flush buffered pipelines (call at handler/worker exit)
Foam::Otel.shutdown    # flush + shut down providers (normal exit is auto via at_exit)
Foam::Otel.after_fork! # restart metric-reader threads after a fork (see cluster mode)

Traces

result = Foam::Otel.span("nightly-sync") do |span|
  # ... work ...  the span always ends; a raised error is recorded + re-raised
end

Foam::Otel.set_attribute("user.tier", "pro")           # on the active span; false if none recording
Foam::Otel.set_attributes("a" => 1, "b" => 2)
Foam::Otel.add_event("cache.miss")
Foam::Otel.record_exception(e)                          # records on active span; never notifies a tracker
ctx = Foam::Otel.get_trace_context                      # {trace_id:, span_id:} hex, or nil — never fabricated

Metrics (one helper per synchronous instrument; names pass through verbatim)

Foam::Otel.increment_counter("checkout.completed", by: 1, attributes: { "plan" => "pro" })
Foam::Otel.record_histogram("checkout.duration_ms", 428)
Foam::Otel.add_up_down_counter("queue.depth", by: -1)
Foam::Otel.set_metric("cache.size_bytes", 10_485_760)   # gauge (last value wins)

⚠️ Metric attributes are a loaded gun: an UNBOUNDED value (user id, raw path, full URL) creates one metric stream per value — a memory leak that the SDK's cardinality cap then turns into silently flat dashboards. Use bounded, low-cardinality attribute values only.

Logs

Foam::Otel.log(:info, "checkout finished", attributes: { "order.id" => id })
# severity is a level symbol (:trace :debug :info :warn :error :fatal) or an
# OTel severity number (1..24). Trace-correlated when a span is active.

Redaction

Foam::Otel.redact(user_supplied_token)  # tail-masks a scalar you know is sensitive; [REDACTED] for structures

Passthroughs (the real OTel objects on foam's pipeline — links, span kinds, observable instruments, baggage all reachable here)

Foam::Otel.get_tracer("my.lib")
Foam::Otel.get_meter("my.lib")
Foam::Otel.get_logger("my.lib")

Coexistence — foam arriving in a monitored service

init classifies each signal's global slot independently and does the right, non-destructive thing. The classifier's output is information for you, never a decision it makes for you.

Scenario 2 — beside a proprietary APM agent

Disjoint pipelines: foam runs its own beside the agent, touching nothing. Add the agent's intake host so foam's outbound instrumentation doesn't shadow its egress.

Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FOAM_OTEL_TOKEN,
                ignored_outbound_hosts: ["agent-intake.vendor.example"])

Scenario 3 — a scoped tenant rides foam's pipeline (LLM-eval, AI-observability)

Attach the tenant's processor via the seam, and add its exporter host to the ignore list (its exports would otherwise be shadow-spanned into a loop).

Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FOAM_OTEL_TOKEN,
                additional_span_processors: [EvalTool::SpanProcessor.new(project: "prod")],
                ignored_outbound_hosts: ["ingest.eval-tool.example"])

Ruby note: when you configure redact_keys/redact_pii_keys, foam's HELPERS apply the key pass at capture, so a tenant processor receives foam-helper data (span/log helper attributes, log bodies, metric labels) ALREADY masked. The Ruby SDK freezes span attributes at finish, so historically a tenant processor saw third-party instrumentation attributes UNMASKED — as of 1.3.0 that gap is closed: tenant span/log processors are handed a READ-ONLY MASKED VIEW (the credential floor plus your configured keys, the same pass foam's own export applies — GOTCHAS F1). A tenant never sees a raw floor value. With no keys configured (the default) only the credential floor is masked on either path.

Loop-guard note: ignored_outbound_hosts suppresses spans for tenant/agent egress made through Net::HTTP or Excon. A tenant exporter using Faraday/HTTP (httprb)/HTTPX cannot be host-suppressed today (no upstream untraced_hosts option — foam warns at init; GOTCHAS G13).

Scenario 4 — a foreign OTel SDK already owns a signal (door 2)

If another OTel SDK registered (say) the tracer before foam, foam does NOT displace it. init warns once, naming the owner, and foam is dark for that signal — your tier-3 telemetry for it rides the foreign SDK to ITS backend. To ALSO send that pipeline to foam, add ONE LINE per claimed signal in THEIR setup — the ingest entries (see "Door 2 — the ingest entries" below). YOU place the tap: foam never attaches itself to a pipeline it does not own (their data moves only when a human says so), and the placement line doubles as the off-switch — gate it on the same condition you would give enabled:.

# config/initializers/opentelemetry.rb — THEIR setup, plus the foam lines
require "opentelemetry/sdk"
require "foam/otel"

OpenTelemetry::SDK.configure do |c|
  c.service_name = "their-service"
  c.use_all
  # REQUIRED loop step (see "Door 2 — the required loop step" below):
  c.use "OpenTelemetry::Instrumentation::Net::HTTP", { untraced_hosts: ["otel.api.foam.ai"] }
  c.add_span_processor(their_vendor_processor)           # their export, untouched
  if ENV["APP_ENV"] == "production"                       # your consent AND your off-switch
    c.add_span_processor(Foam::Otel.create_ingest_span_processor(
      token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
  end
end

Outcome: their pipeline keeps working exactly as before — their data, their resource, their export, byte-identical — and foam ALSO receives the signal, carrying THEIR resource identity plus foam's export-time stamp, the always-on credential floor applied to foam's copy, and any per-tap redact_keys/redact_pii_keys you set (opt-in — none by default; details in the Door 2 section).

Unhappy paths:

Foam::Otel.create_ingest_span_processor(token: "", environment: "production")
# => raises ArgumentError: [foam] Foam::Otel.create_ingest_span_processor requires a non-empty token

OTEL_SDK_DISABLED=true at construction → the factory returns an inert no-op instance (their setup line keeps working, nothing is sent) and warns: [foam] OTEL_SDK_DISABLED=true — this foam ingest entry is disabled by the operator kill switch and will send nothing.
Tap wired onto FOAM'S OWN pipeline (self-ingest wiring bug) → one loud warning and the tap STANDS DOWN — door 1 already exports that data, so nothing is lost and nothing is doubled.
init() already exports a signal AND a tap sits on a separate foreign pipeline for the same signal → one loud warning; BOTH keep flowing (different pipelines; foam.ingest.tier distinguishes the copies).

When the TRACES slot is foreign-owned, foam also skips its automatic tier-1/2 instrumentation activation (it never adds span producers or Rack middleware to a pipeline it does not own) and warns once; additional_instrumentations you pass explicitly are still installed. The same holds for the process-wide propagator: a propagator the incumbent installed first is KEPT (foam installs its W3C set only over the API default), with one warning.

Scenario 5 — per-signal composition

Signals are independent. A foreign SDK owning only traces still gets foam for metrics and logs on the full contract:

Foam::Otel.init(name: "checkout-api", environment: env, enabled: true, token: FOAM_OTEL_TOKEN)
# → warns: traces foreign-owned (foam dark for traces); metrics + logs → foam, full contract.

# The doors COMPOSE: add the traces tap in THEIR setup and foam receives
# traces too — the tap rides THEIR tracer, and trace ids correlate with
# foam-exported logs/metrics through the shared context:
their_provider.add_span_processor(Foam::Otel.create_ingest_span_processor(
  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
# (no double-claim warning: init() serves metrics+logs, the tap serves traces)

Scenario 6 — tests / CI (fully inert, no token needed)

Foam::Otel.init(name: "checkout-api", environment: "test", enabled: false)

Scenario 7 — serverless / short-lived (flush at the end)

Foam::Otel.init(name: "orders-fn", environment: "production", enabled: true, token: FOAM_OTEL_TOKEN)

def handler(event:, context:)
  run(event)
ensure
  Foam::Otel.flush   # the platform freezes the process before the batch timer fires
end

Puma / Unicorn (cluster mode) — CRITICAL

Exporter threads do not survive fork. Foam auto-recovers metric readers via a Process._fork hook, but the cleanest pattern is to init PER WORKER so each worker gets its own pipeline and a unique service.instance.id:

# config/puma.rb
on_worker_boot do
  Foam::Otel.init(name: "checkout-api", environment: ENV["APP_ENV"], enabled: true, token: ENV["FOAM_OTEL_TOKEN"])
  # or, if you init in an initializer: Foam::Otel.after_fork!
end

Door 2 — the ingest entries

When a foreign OTel SDK owns a signal (scenario 4), the ingest entries are how that pipeline's data ALSO reaches foam: constructed instances you add — one line per claimed signal — to the CUSTOMER'S OWN OTel setup. Additive readers only: they never mutate their data, their resource, or their export.

All three entries are SHIPPED. The traces entry rides the stable 1.x SDK; the log and metric entries ride upstream's pre-1.0 SDKs (logs "development", metrics alpha), pinned to this gem's capped ranges (GOTCHAS R2/R5) so a 0.x drift is a deliberate re-prove, never a silent change. The pre-1.0 SDK maturity is a version-pin caveat, not a deferral — the entries are available and supported today.

Version-pin posture: because the log and metric taps ride those pre-1.0 upstream SDKs, they are version-pinned and re-proven on every 0.x bump rather than assumed stable — shipped and supported either way.

The three entries

All three share one signature (token: and environment: required; the redaction kwargs are the only CUSTOMER keys that tap redacts — opt-in, none by default, scoped to that tap, additive above the always-on credential floor, which every tap applies with zero configuration; diagnostics: is tap-scoped narration). Construction validates loudly at boot and NEVER throws afterwards — a failure inside a tap can never break their pipeline. Fleet canon mapping (pinned by spec/export_surface_spec.rb): create_ingest_span_processor = createFoamIngestSpanProcessor, create_ingest_log_record_processor = createFoamIngestLogRecordProcessor, create_ingest_metric_reader = createFoamIngestMetricReader.

Traces — a SpanProcessor for THEIR TracerProvider (scenario 4 snippet):

processor = Foam::Otel.create_ingest_span_processor(
  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV"),
  redact_keys: %w[voucher_code],          # optional: extra secret keys, this tap only
  redact_pii_keys: %w[customer_email],    # optional: your PII enumeration, this tap only
  diagnostics: false)                     # optional: per-tap [foam] narration
their_tracer_provider.add_span_processor(processor)
# returns a new instance per call; no-op inert instance under OTEL_SDK_DISABLED=true;
# raises ArgumentError at construction for a blank token/environment

Logs — a LogRecordProcessor for THEIR LoggerProvider. The credential floor and the tap's redact_keys/redact_pii_keys (if any) match over record attributes and keys nested in structured bodies; free-text bodies ride RAW (no value-pattern pass):

provider = OpenTelemetry::SDK::Logs::LoggerProvider.new(resource: their_resource)
provider.add_log_record_processor(their_log_processor)
provider.add_log_record_processor(Foam::Otel.create_ingest_log_record_processor(
  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
OpenTelemetry.logger_provider = provider

Metrics — a PeriodicMetricReader for THEIR MeterProvider. Attach order is forgiving: add_metric_reader back-fills instruments created before the tap attached. Foam's reader collects on its own schedule beside theirs:

OpenTelemetry.meter_provider.add_metric_reader(Foam::Otel.create_ingest_metric_reader(
  token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))

Door 2 — the required loop step

The tap's exports to foam are outbound HTTP calls THEIR instrumentation will span, and their pipeline hands those spans back to the tap — a feedback loop in THEIR pipeline. Three layers keep it closed:

The SDK suppression context (automatic): foam's exporters send inside untraced, which the official http instrumentations respect.
REQUIRED — suppress foam's endpoint host in THEIR http instrumentation (belt for non-contrib patches and older contrib versions): add untraced_hosts: ["otel.api.foam.ai"] to their Net::HTTP/Excon config (scenario 4 snippet). When the OTEL_EXPORTER_OTLP_ENDPOINT override is active, list THAT host instead. Faraday/HTTP (httprb)/HTTPX cannot take the option, but they also never see the tap's exports — foam's exporters use Net::HTTP only.
The echo filter (structural backstop): a client-kind span naming the ACTIVE foam export host is telemetry about foam's own export call — the tap refuses to re-export its own echo, with one loud warning naming the missing step. Their pipeline keeps that span untouched.

Identity on the wire

	their pipeline	foam's copy
`service.name` / `service.version` / `service.instance.id`	theirs	theirs, untouched (the data carries THEIR identity)
all attributes / events / bodies	raw, byte-identical	the credential floor + your per-tap `redact_keys`/`redact_pii_keys` applied (default: floor only — everything off the frozen list identical to theirs)
`deployment.environment.name`	theirs (if any)	the tap's `environment:` argument (stamped at export time)
`foam.ingest.tier`	absent	`"external"` (the wire-contract marker)
`telemetry.distro.name` / `telemetry.distro.version`	absent	`"foam"` / the gem version (support reads the tap's presence and version from telemetry alone)

The stamp is merged onto FOAM'S COPY at export serialization — never onto the shared objects; it changes exactly those four keys and nothing else.

External-tier caveats (recorded, never fought)

THEIR sampler gates what the traces tap sees (unsampled spans never reach processors).
THEIR Views shape what the metrics tap receives; per-reader temporality is not a Ruby knob — aggregations default cumulative (or the operator's OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE), and cumulative totals start at attach time.
Lifecycle rides THEIR pipeline: their provider's shutdown/force_flush drives the tap's. Serverless mirror: flush THEIR provider at handler end (the platform freezes the process before batch timers fire).
Kill switch and endpoint override apply to taps exactly as to init: OTEL_SDK_DISABLED=true at construction → inert instance + one warning; OTEL_EXPORTER_OTLP_ENDPOINT honored with the loud warning; per-signal OTLP endpoint variants stay INERT + warned.

Puma / Unicorn note (door 2)

Construct taps where THEIR setup runs — with SDK.configure in an initializer under cluster preload, taps are inherited by workers. The span and log taps self-heal in the child (the stock batch processors reset on first use after a fork); the metrics tap is restarted by foam's own Process._fork hook, which the factory installs (a separate registry from init's own readers, so a per-worker re-init never orphans a tap — GOTCHAS R7). Foam::Otel.after_fork! covers both if you wire it manually.

Tests / CI

Gate the add-line on the same condition you would give enabled: — the placement line IS the switch (no foam-invented env var):

if ENV["APP_ENV"] == "production"
  c.add_span_processor(Foam::Otel.create_ingest_span_processor(
    token: ENV.fetch("FOAM_OTEL_TOKEN"), environment: ENV.fetch("APP_ENV")))
end

Failure modes — what happens and where the warning appears

Situation	Behavior
blank `name`/`environment`, or `enabled` omitted	raises `ArgumentError` at boot (on your machine)
`enabled: true` but blank `token`	raises `ArgumentError` at boot
`enabled: false`	fully inert; helpers no-op silently
missing `version`	`[foam]` warning, continues
a foreign OTel SDK owns a signal	`[foam]` warning once, naming the owner AND the matching door-2 ingest entry; foam dark for that signal
`init` called twice	`[foam]` warning; returns the first instance (providers are process-global)
ingest entry with blank `token:`/`environment:`	raises `ArgumentError` at construction (on your machine)
ingest entry under `OTEL_SDK_DISABLED=true`	inert no-op instance + one `[foam]` warning; their setup line never crashes
ingest tap wired onto foam's OWN pipeline	`[foam]` wiring-bug warning once; the tap STANDS DOWN (zero duplicates)
`init()` exports a signal AND a tap on a separate foreign pipeline	`[foam]` wiring-bug warning once; both keep flowing (`foam.ingest.tier` distinguishes them)
their instrumentation traces foam's export calls (missing loop step)	echo filter refuses foam's own echo + one `[foam]` warning naming the required `untraced_hosts` step

Environment variables

Var	Posture
`OTEL_SDK_DISABLED=true`	HONORED — full kill switch, supersedes `enabled: true` (matched trimmed + case-insensitive, so `TRUE` / `true` also kill — the fleet's one emergency switch behaves the same across languages). Door-2 ingest entries honor it too: construction returns an inert instance (read once at construction; changing it implies a restart).
`OTEL_EXPORTER_OTLP_ENDPOINT`	HONORED — the one operator-level override of the pinned fleet endpoint (for foam's own conformance rig / enterprise egress). Active → loud `[foam]` warning naming the destination. Applies to door-2 taps identically (and moves the host the required loop step must name).
`OTEL_PROPAGATORS=none`	HONORED — turns trace propagation OFF (links lost) while telemetry keeps flowing; warns. Any other value warns and is ignored (foam's propagator set is fixed: W3C tracecontext + baggage).
`OTEL_BSP_SCHEDULE_DELAY` / `OTEL_BLRP_SCHEDULE_DELAY` / `OTEL_METRIC_EXPORT_INTERVAL`	HONORED — batch cadence, read natively by the upstream SDK.
`OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`	INERT — foam wires ONE resolved endpoint into all three exporters explicitly, so per-signal endpoint vars never redirect (or split) foam's export. Set → loud `[foam]` warning that it is inert.
`OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`	INERT — as above (warns when set).
`OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`	INERT — as above (warns when set).
`OTEL_RUBY_EXPORTER_OTLP_SSL_VERIFY_NONE`	INERT — foam pins TLS peer verification (`ssl_verify_mode: VERIFY_PEER`) on every exporter constructor, both doors, so this vendored-exporter knob can never strip certificate verification off the credential-bearing export connection (1.4.0 security fix, CWE-295). Set → loud `[foam]` warning that it is inert; fix the trust chain (CA bundle) instead.
every other `OTEL_*`	INERT — `OTEL_SERVICE_NAME` / `OTEL_RESOURCE_ATTRIBUTES` never override the init-declared identity; sampler/exporter/span-limit vars are ignored.

Troubleshooting

Telemetry halved or doubled after a dependency bump → check for a second copy of the OTel API first: bundle list | grep opentelemetry-api must show ONE. Two copies mean two global registries (silent no-op or doubled data).
No data at all → set diagnostics: true and read the [foam] lines; check init runs before your app code, that enabled is true, and that a token is wired.
No door-2 data → check, in order: OTEL_SDK_DISABLED unset (a killed tap is inert and warned once at boot); the tap is on the RIGHT provider (the one their setup actually registers — a wiring-bug warning at first traffic means self-ingest stand-down); the [foam] wiring-bug/echo warnings in the boot log; construct with diagnostics: true for per-export narration; for metrics under Puma, see the Door 2 Puma note.
A downstream rejects traceparent → set OTEL_PROPAGATORS=none to stop injection while keeping telemetry flowing.

See GOTCHAS.md (the traps and how foam handles them) and RESEARCH.md (the primitive audit, instrumentation census, and redaction provenance).