Module: Foam::Otel::Redaction

Defined in:
lib/foam/otel/redaction.rb

Overview

The central redaction engine: the always-on CREDENTIAL FLOOR, the always-on VALUE-PATTERN SECRET LAYER, and the customer's opt-in key lists (holistic redesign 2026-07-26, amended by the credential-floor fleet ruling — docs/decisions/credential-denylist-design.md — and the 2026-07-27 value-pattern/coverage ruling — docs/decisions/security-fixes-design.md).

THE FLOOR (unconditional, no off switch): every attribute/field whose NAME is on the frozen CREDENTIAL_KEY_DENYLIST (constants.rb) — including the semconv http.request,response.header. forms of the CREDENTIAL_HEADER_DENYLIST — masks in FULL to the literal [REDACTED], in every signal, at every depth this engine walks, before the customer's lists run. Matching is canon-exact (lowercase + dash→underscore fold, design §1.0/§2.1): NEVER substring — "authorization" does not mask "authorization_url"; "secret" does not mask "secretary". The floor consults the module-level frozen constant directly (never Config), so no configuration path can disable, shrink, or re-spell it.

THE VALUE LAYER (unconditional Tier 1; heuristic tier has the single documented secret_heuristics: false opt-out — security-fixes-design CONTRACT V): the floor is value-shape-blind by design, so this second control masks credential-SHAPED spans inside every leaf STRING the engine walks (scan_value_secrets below) — AWS/GCP/Azure keys, GitHub/GitLab/Slack/Stripe tokens, JWTs, private-key blocks, user:pass@ URI credentials, bearer/basic values, Anthropic/OpenAI keys — to the literal [REDACTED], regardless of key name. ReDoS-bounded (compile-once, anchor pre-filter, 256 KiB cap, match-flood cap, per-Regexp timeout on Ruby >= 3.2) and FAIL-CLOSED (any scanner fault masks the whole value, never a raw pass-through). Metric datapoint attributes are the one exempt surface (design §V.7 — low-cardinality by construction; pinned by the vuln suite's off-surface case).

ABOVE the floor and the value layer, redaction stays fully opt-in: UNSHAPED values under unlisted names are captured RAW. Customer redaction runs ONLY over the keys the customer enumerates, ADDITIVE:

* redact_keys     — tail-mask matching scalar values (shape-preserving,
                  e.g. ********f456); non-scalars redact in full. A
                  value-pattern hit is TERMINAL for its span: the
                  tail mask never runs over a shaped secret (a
                  4-char reveal is exactly the fragment an LLM could
                  reassemble — design §V.0).
* redact_pii_keys — erase matching values in full ([REDACTED]).

CUSTOMER matching is SUBSTRING, case-insensitive, over the lowercased key (documented Ruby behavior, unchanged). The FLOOR deliberately does NOT ride that matcher (design §2.4 — substring would over-match): it uses its own canon-exact predicate, floor_kind, below. A floor match is TERMINAL: listing a floor name in redact_keys never downgrades it to a tail mask.

Two entry points:

* redact(value)            — the public helper: mask a value the caller
                           KNOWS is sensitive (scalar → tail mask,
                           everything else → full [REDACTED]). This is
                           unconditional — the caller opted in by
                           calling it.
* mask_span_data / mask_log_record_data / mask_metric_data — the
                           exporter-boundary pass that applies the
                           floor plus the customer's key lists, run by
                           the redacting exporters
                           (redacting_exporter.rb) on door 1 AND all
                           three door-2 taps. Runs UNCONDITIONALLY —
                           the floor has no off switch; values under
                           unlisted names pass through raw.

Ruby SDK constraint (see GOTCHAS.md): a Span freezes its attributes at finish (opentelemetry-sdk span.rb) BEFORE any SpanProcessor#on_finish runs, so the key pass cannot be a processor — it runs at the exporter boundary, rebuilding the mutable SpanData/LogRecordData Structs.

Constant Summary collapse

FLOOR_HEADER_PREFIXES =

The two semconv header-attribute prefixes — the ONLY prefixes the floor peels (design §2.2). Canonical (lowercase, no dashes), so matching the canon'd name covers any case variant of the prefix too.

["http.request.header.", "http.response.header."].freeze
FLOOR_KEY_SET =

O(1) lookup set for the frozen key list (the array stays the shipped, fixture-gated constant; this is a derived view, same object identity of entries).

CREDENTIAL_KEY_DENYLIST.to_set.freeze
SECRET_REGEXP_TIMEOUT =

---- compile-once value-layer machinery (design §V.5.1) --------------- Built at require time from the frozen constants.rb ruleset; a pattern that fails to compile fails the require LOUDLY — never a silent skip. Ruby >= 3.2 compiles each rule with a per-Regexp timeout (belt-and- braces on top of the authoring rules; a timeout fires the fail-closed whole-value mask via scan_value_secrets' rescue). No global Regexp.timeout mutation — customer-process hygiene.

0.25
SECRET_VALUE_SCAN_CAP = 
SECRET_SCAN_CAPS.fetch(:value_scan_cap)
SECRET_MAX_MATCHES = 
SECRET_SCAN_CAPS.fetch(:pattern_max_matches_per_value)
SECRET_PEM_BLOCK_CAP = 
SECRET_SCAN_CAPS.fetch(:pem_block_cap)
SECRET_H2_MIN = 
SECRET_SCAN_CAPS.fetch(:h2_token_min)
SECRET_H2_MAX = 
SECRET_SCAN_CAPS.fetch(:h2_token_max)
SECRET_QUERY_PAIR_CAP = 
SECRET_SCAN_CAPS.fetch(:query_pair_cap)
SECRET_RULES_COMPILED = 
SECRET_VALUE_RULES.map do |rule|
  rule.merge(re: build_secret_regexp(rule[:regex], rule[:ci])).freeze
end.freeze
SECRET_H1_RE = 
build_secret_regexp(SECRET_HEURISTIC_H1[:regex], true)
SECRET_H2_TOKEN_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:token_regex], false)
SECRET_H2_UUID_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:uuid_regex], false)
SECRET_H2_HEX_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:hex_regex], false)
SECRET_PLACEHOLDER_RES = 
SECRET_VALUE_PLACEHOLDERS.map { |p| build_secret_regexp(p, true) }.freeze
SECRET_ALL_ANCHORS =

Every lowercase literal anchor across Tier 1 + the linear scans + H1 — the single pre-filter deciding whether ANY regex runs (and the whole decision for oversized values).

(
  SECRET_VALUE_RULES.flat_map { |r| r[:anchors] } +
  SECRET_LINEAR_SCAN_ANCHORS +
  SECRET_HEURISTIC_H1[:anchors]
).uniq.sort.freeze

Class Method Summary collapse

Class Method Details

.apply_secret_spans(text, spans) ⇒ Object

Replace each (merged) secret span with [REDACTED], left to right, never rescanning emitted mask text (design §V.6 — zero partial reveal: no tail, no prefix, no length preservation).



428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
 
# File 'lib/foam/otel/redaction.rb', line 428

def apply_secret_spans(text, spans)
  merged = []
  spans.sort.each do |s, e|
    if !merged.empty? && s <= merged[-1][1]
      merged[-1][1] = e if e > merged[-1][1]
    else
      merged << [s, e]
    end
  end
  out = +""
  prev = 0
  merged.each do |s, e|
    next if s < prev

    out << text[prev...s] << REDACTED
    prev = e
  end
  out << text[prev..]
  out
end

.build_secret_regexp(source, case_insensitive) ⇒ Object

seconds, per execution



905
906
907
908
909
910
911
912
 
# File 'lib/foam/otel/redaction.rb', line 905

def build_secret_regexp(source, case_insensitive)
  options = case_insensitive ? Regexp::IGNORECASE : 0
  Regexp.new(source, options, timeout: SECRET_REGEXP_TIMEOUT)
rescue ArgumentError, TypeError
  # Ruby < 3.2 has no timeout: kwarg — rules 1–5 of design §V.5 carry
  # the linearity guarantee on their own there.
  Regexp.new(source, options)
end

.canon(name) ⇒ Object

---- the credential-floor predicate (design §2, self-contained) ------- canon(name): trim ASCII whitespace → ASCII-lowercase → fold '-' to '_' (design §1.0). Applied to BOTH sides of every floor comparison; the stored CREDENTIAL_KEY_DENYLIST entries are already canonical.



158
159
160
 
# File 'lib/foam/otel/redaction.rb', line 158

def canon(name)
  name.to_s.strip.downcase(:ascii).tr("-", "_")
end

.classify_key(key, config) ⇒ Object

---- key classification ---------------------------------------------- The credential floor is checked FIRST and is TERMINAL (design §4): a floor-matched name is [REDACTED] in full even when the customer also lists it in redact_keys — the tail pass must never see a floor value (a tail mask over the [REDACTED] literal would emit ******** and destroy the contract literal). Below the floor: redact_keys (:mask) wins over redact_pii_keys (:erase) when both claim a key. Customer matching is substring, case-insensitive (unchanged).



586
587
588
589
590
591
592
593
594
595
 
# File 'lib/foam/otel/redaction.rb', line 586

def classify_key(key, config)
  return :floor if floor_match?(key)
  return nil if config.nil?

  lower = key.downcase
  return :mask if match_any_substring?(lower, config.redact_keys)
  return :erase if match_any_substring?(lower, config.redact_pii_keys)

  nil
end

.decode_component(raw) ⇒ Object

Single-pass percent-decode (never recursive — a double-encoded name is decoded exactly once, matching every other core).



710
711
712
713
714
 
# File 'lib/foam/otel/redaction.rb', line 710

def decode_component(raw)
  URI.decode_www_form_component(raw)
rescue StandardError
  raw
end

.deep_mask(value, config, depth, value_scan: true) ⇒ Object

C3 (coverage contract): recurse EVERYWHERE, fail CLOSED — depth exceeded, a cycle bottoming out, or an unknown node type is replaced with [REDACTED]/a scanned stringification, never passed through raw.



855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
 
# File 'lib/foam/otel/redaction.rb', line 855

def deep_mask(value, config, depth, value_scan: true)
  return REDACTED if depth > MAX_REDACT_DEPTH

  case value
  when Hash
    value.each_with_object({}) do |(key, val), out|
      k = key.to_s
      if malformed_header_key?(k)
        out[key] = REDACTED # C2.3, same fail-closed shape as mask_one
        next
      end
      case classify_key(k, config)
      when :floor then out[key] = floor_mask(floor_kind(k), val)
      when :erase then out[key] = REDACTED
      when :mask  then out[key] = masked_value_for(val, config, value_scan)
      else out[key] = deep_mask(val, config, depth + 1, value_scan: value_scan)
      end
    end
  when Array
    value.map { |v| deep_mask(v, config, depth + 1, value_scan: value_scan) }
  when String
    value_scan ? redact_leaf_string(value, config) : value
  when Numeric, Symbol, TrueClass, FalseClass, NilClass
    value
  else
    # C3.2 unknown node types fail CLOSED: the exporter would stringify
    # the object anyway — stringify HERE and run the leaf pass over the
    # result so a credential inside a #to_s is still caught. An object
    # whose #to_s explodes becomes the mask, never the raw object.
    value_scan ? scan_unknown_object(value, config) : value
  end
rescue StandardError
  REDACTED
end

.descend(value, config = nil, depth = 0, value_scan: true) ⇒ Object

Unmatched key: a STRING leaf runs the central free-text pass (the C1 URL/query/fragment tokenizer + the value-pattern scan — an UNSHAPED value is captured RAW, byte-identical); structured values DESCEND so a floor-listed or redact_keys/redact_pii_keys key NESTED under an innocent key is honored — bounded to MAX_REDACT_DEPTH so an adversarial/cyclic structure can never drive a downstream encoder into SystemStackError (rule 9).



555
556
557
558
559
560
561
562
 
# File 'lib/foam/otel/redaction.rb', line 555

def descend(value, config = nil, depth = 0, value_scan: true)
  cfg = config || Foam::Otel.active_config
  case value
  when Hash, Array then deep_mask(value, cfg, depth, value_scan: value_scan)
  when String then value_scan ? redact_leaf_string(value, cfg) : value
  else value
  end
end

.floor_kind(name) ⇒ Object

Classify a name against the FLOOR ONLY (never the customer lists):

:header — a semconv header attribute (http.{request,response}
        .header.<name>) whose peeled suffix canon-matches the key
        list (which contains all seven headers — subset invariant,
        design §1.1; the peel checks the FULL 52-entry list, §2.2,
        so e.g. http.request.header.x_forwarded_for masks too);
:key    — the full name itself canon-matches a key-list entry
        (full-name equality: "user.session" does NOT match
        "session"; "connect.sid" matches its own verbatim entry);
nil     — no floor match (customer lists may still apply).


182
183
184
185
186
187
188
189
190
 
# File 'lib/foam/otel/redaction.rb', line 182

def floor_kind(name)
  c = canon(name)
  FLOOR_HEADER_PREFIXES.each do |prefix|
    next unless c.start_with?(prefix)

    return FLOOR_KEY_SET.include?(c[prefix.length..]) ? :header : nil
  end
  FLOOR_KEY_SET.include?(c) ? :key : nil
end

.floor_mask(kind, value) ⇒ Object

The floor mask (design §3): a semconv header attribute is a string array (one element per header instance) — mask PER ELEMENT, preserving arity and the string-array type (OTel's own sanitize_header_values shape). Everything else replaces the ENTIRE value with the single literal [REDACTED] — scalar, object, array, number, binary. Never a tail, never length-preserving: these are credentials, not debug aids.



202
203
204
205
206
 
# File 'lib/foam/otel/redaction.rb', line 202

def floor_mask(kind, value)
  return value.map { REDACTED } if kind == :header && value.is_a?(Array)

  REDACTED
end

.floor_match?(name) ⇒ Boolean

Returns:

  • (Boolean) 


192
193
194
 
# File 'lib/foam/otel/redaction.rb', line 192

def floor_match?(name)
  !floor_kind(name).nil?
end

.h1_key_context_allowlisted?(text, match) ⇒ Boolean

Returns:

  • (Boolean) 


387
388
389
390
 
# File 'lib/foam/otel/redaction.rb', line 387

def h1_key_context_allowlisted?(text, match)
  ctx = text[match.begin(0)...match.begin(1)].to_s.downcase
  SECRET_KEY_CONTEXT_ALLOWLIST.any? { |name| ctx.include?(name) }
end

.h1_spans(text) ⇒ Object

H1 — keyword-anchored generic secret (gitleaks generic-api-key, MIT). Four-layer FP suppression (design §V.3): no-digit reject, placeholder reject, stopword reject, key-context allowlist; entropy floor 3.5 (hex candidates 3.0 with the detect-secrets all-numeric penalty).



370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
 
# File 'lib/foam/otel/redaction.rb', line 370

def h1_spans(text)
  spans = []
  text.scan(SECRET_H1_RE) do
    m = Regexp.last_match
    cand = m[1]
    next if cand.nil? || cand.match?(/\A[a-zA-Z_.\-]+\z/)
    next if secret_placeholder?(cand) || secret_stopword?(cand)
    next if h1_key_context_allowlisted?(text, m)

    floor = cand.match?(/\A[0-9a-fA-F]+\z/) ? SECRET_HEURISTIC_H1[:entropy_hex] : SECRET_HEURISTIC_H1[:entropy]
    entropy = shannon_entropy(cand)
    entropy -= 1.2 / Math.log2([2, cand.length].max) if cand.match?(/\A[0-9]+\z/)
    spans << [m.begin(1), m.end(1)] if entropy >= floor
  end
  spans
end

.h2_whole_token?(text) ⇒ Boolean

H2 — standalone high-entropy token (detect-secrets Base64HighEntropyString, Apache-2.0): ONLY when the entire trimmed value is one base64/url-safe token; class-mix + entropy >= 4.5 + not-UUID + not-pure-hex (trace/span ids, git SHAs and digests are telemetry's ambient vocabulary — recorded deviation, design §V.3) + not-placeholder + no-stopword.

Returns:

  • (Boolean) 


398
399
400
401
402
403
404
405
406
407
 
# File 'lib/foam/otel/redaction.rb', line 398

def h2_whole_token?(text)
  s = text.strip
  return false unless s.length.between?(SECRET_H2_MIN, SECRET_H2_MAX)
  return false unless SECRET_H2_TOKEN_RE.match?(s)
  return false unless s.match?(/[0-9]/) && s.match?(/[A-Z]/) && s.match?(/[a-z]/)
  return false if SECRET_H2_UUID_RE.match?(s) || SECRET_H2_HEX_RE.match?(s)
  return false if secret_placeholder?(s) || secret_stopword?(s)

  shannon_entropy(s) >= SECRET_HEURISTIC_H2[:entropy]
end

.heuristics_enabled?(config) ⇒ Boolean

The heuristic tier's single opt-out (design §V.8): secret_heuristics: false narrows ONLY H1/H2 — Tier 1, the floor and Contract C are unaffected. Unknown/absent config defaults ON.

Returns:

  • (Boolean) 


452
453
454
455
456
457
458
459
 
# File 'lib/foam/otel/redaction.rb', line 452

def heuristics_enabled?(config)
  return true if config.nil?
  return true unless config.respond_to?(:secret_heuristics)

  config.secret_heuristics != false
rescue StandardError
  true
end

.malformed_header_key?(name) ⇒ Boolean

C2.3 (security-fixes-design): a semconv header attribute whose peeled suffix is PURELY NUMERIC (http.request.header.0) is malformed header capture (upstream's array-of-pairs Object.keys artifact) — the real header name is unknowable, so the value is masked WHOLE, fail-closed.

Returns:

  • (Boolean) 


212
213
214
215
216
217
218
219
220
 
# File 'lib/foam/otel/redaction.rb', line 212

def malformed_header_key?(name)
  c = canon(name)
  FLOOR_HEADER_PREFIXES.each do |prefix|
    next unless c.start_with?(prefix)

    return c[prefix.length..].match?(/\A[0-9]+\z/)
  end
  false
end

.mask_attributes(attributes, config, value_scan: true) ⇒ Object

---- the floor + value-layer + customer key pass over attributes ------ Returns a NEW attributes hash with the credential floor, the value-pattern secret layer and the customer's key lists applied. Runs UNCONDITIONALLY (neither the floor nor Tier 1 has an off switch); UNSHAPED values under unmatched names pass through raw (same objects). Never raises; a key that fails to mask redacts in full rather than leaking. value_scan: false is the ONE mandated exemption — metric datapoint attributes (design §V.7): the key passes still run there, only the value-shape scan is off.



487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
 
# File 'lib/foam/otel/redaction.rb', line 487

def mask_attributes(attributes, config, value_scan: true)
  return attributes unless attributes.is_a?(Hash)

  attributes.each_with_object({}) do |(key, value), out|
    out[key] = mask_one(key.to_s, value, config, value_scan: value_scan)
  rescue StandardError
    # Fail closed at the VALUE level, loudly (rule 15): the key name only
    # — never the value — reaches the diagnostics channel. The warning is
    # itself guarded: a poisoned key (#to_s raises) must still redact.
    begin
      Diagnostics.warn("masking failed for attribute #{key.to_s.inspect} — value redacted in full " \
                       "(fail-closed, rule 14)")
    rescue StandardError, SystemStackError
      Diagnostics.warn("masking failed for an attribute (unprintable key) — value redacted in full " \
                       "(fail-closed, rule 14)")
    end
    out[key] = REDACTED
  end
end

.mask_body(body, config) ⇒ Object

A log body is AnyValue: a free-text string rides the central leaf-string pass (C1 tokenizer + value-pattern scan — UNSHAPED text is captured RAW), a structured map/array DESCENDS so a floor-listed or customer-listed key nested inside is honored (depth-guarded). Runs unconditionally.



846
847
848
849
850
 
# File 'lib/foam/otel/redaction.rb', line 846

def mask_body(body, config)
  deep_mask(body, config, 0)
rescue StandardError
  REDACTED
end

.mask_exemplars!(point, config) ⇒ Object

Exemplars carry filtered_attributes the OTLP exporter encodes RAW, one field over from the point attributes just masked — the floor must reach them too (design §2.3). Two postures, both fail-closed:

* customer keys configured → DROP exemplars entirely (the documented
pre-floor behavior for configured redaction, unchanged);
* floor only (the default path) → keep the exemplars but run their
filtered_attributes through the same engine pass, so floor-listed
names are [REDACTED] and everything else stays raw (§9: the only
default-path wire delta is floor values becoming [REDACTED]). An
exemplar whose rebuild fails is dropped, never exported raw.


820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
 
# File 'lib/foam/otel/redaction.rb', line 820

def mask_exemplars!(point, config)
  return unless point.respond_to?(:exemplars=)

  if redaction_configured?(config)
    point.exemplars = nil
    return
  end
  exemplars = point.respond_to?(:exemplars) ? point.exemplars : nil
  return if exemplars.nil?

  point.exemplars = exemplars.filter_map do |exemplar|
    copy = exemplar.dup
    copy.filtered_attributes = mask_attributes(exemplar.filtered_attributes || {}, config)
    copy
  rescue StandardError
    nil # fail closed: a poisoned exemplar drops, never ships raw
  end
rescue StandardError
  point.exemplars = nil # fail closed at the list level too
end

.mask_log_record_data(record, config) ⇒ Object 



754
755
756
757
758
759
760
761
 
# File 'lib/foam/otel/redaction.rb', line 754

def mask_log_record_data(record, config)
  masked = record.dup
  masked.attributes = mask_attributes(record.attributes, config) if record.attributes
  masked.body = mask_body(record.body, config) if record.body
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_metric_data(metric_data, config) ⇒ Object

Door 1 masks metric attributes at capture; foreign (door-2) instrument data never passes those helpers, so the ingest metric path applies the floor + key lists here. Rebuilds every data point as a COPY (the collected points' attributes hash is the LIVE aggregation key upstream — masking in place would corrupt the customer's aggregation state). Runs UNCONDITIONALLY (the floor has no off switch).

FAIL CLOSED: returns NIL on a hard failure — the caller drops the metric loudly rather than exporting a half-masked point.



789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
 
# File 'lib/foam/otel/redaction.rb', line 789

def mask_metric_data(metric_data, config)
  masked = metric_data.dup
  points = metric_data.data_points
  if points
    masked.data_points = points.map do |point|
      copy = point.dup
      # Metric datapoint attributes are the ONE surface the value-
      # pattern layer is mandated OFF for (design §V.7: low-cardinality
      # by construction; a value-shape mask would break the customer's
      # aggregation dimensions). The floor + customer key passes still
      # apply in full.
      copy.attributes = mask_attributes(point.attributes || {}, config, value_scan: false)
      mask_exemplars!(copy, config)
      copy
    end
  end
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_one(key, value, config, value_scan: true) ⇒ Object

Ordering per value (design §V.0): floor name-match (terminal) → coverage tokenizers with per-pair name match → value-pattern scan (terminal per span) → customer keys.



510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
 
# File 'lib/foam/otel/redaction.rb', line 510

def mask_one(key, value, config, value_scan: true)
  if malformed_header_key?(key)
    # C2.3 fail-closed: a numeric header suffix means the real header
    # name (possibly Authorization) is unknowable — mask whole.
    warn_secret_scan_once(:numeric_header,
                          "malformed header-capture attribute (numeric suffix) — value redacted " \
                          "in full (fail-closed, coverage contract C2.3)")
    return REDACTED
  end
  case classify_key(key, config)
  when :floor then floor_mask(floor_kind(key), value)
  when :erase then REDACTED
  when :mask  then masked_value_for(value, config, value_scan)
  else descend(value, config, 0, value_scan: value_scan)
  end
end

.mask_span_data(span_data, config) ⇒ Object

---- exporter-boundary pass (SpanData / LogRecordData) ---------------- Rebuild the mutable Struct with the credential floor and the customer's key lists applied. A dup keeps foam's export copy independent of the span the tenant seam saw. Runs UNCONDITIONALLY (the floor has no off switch); values under unmatched names are carried through raw.

FAIL CLOSED: on a hard failure these return NIL — never the raw payload once the pass has begun. The redacting exporters drop a nil entry (with a loud [foam] warning) rather than export a half-masked record.



725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
 
# File 'lib/foam/otel/redaction.rb', line 725

def mask_span_data(span_data, config)
  masked = span_data.dup
  masked.attributes = mask_attributes(span_data.attributes, config) if span_data.attributes
  # status.message is a value-bearing string (design §V.7): a shaped
  # secret inside an exception-derived status description must not ship.
  masked.status = mask_status(span_data.status, config) if span_data.respond_to?(:status) && span_data.status
  # Link attributes ride the key pass too: links are reachable on BOTH
  # doors and the OTLP exporter encodes link.attributes RAW. Copy-not-
  # mutate, exactly as events are handled (Link freezes its attributes at
  # construction).
  if span_data.links && !span_data.links.empty?
    masked.links = span_data.links.map do |link|
      next link unless link.respond_to?(:attributes) && link.attributes

      link.class.new(link.span_context, mask_attributes(link.attributes, config))
    end
  end
  if span_data.events && !span_data.events.empty?
    masked.events = span_data.events.map do |event|
      next event unless event.respond_to?(:attributes) && event.attributes

      event.class.new(event.name, mask_attributes(event.attributes, config), event.timestamp)
    end
  end
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_status(status, config) ⇒ Object

Rebuild a span Status with the leaf-string pass applied to its description. The Status constructor is private API — the public ok/error/unset builders are used per code.



766
767
768
769
770
771
772
773
774
775
776
777
778
 
# File 'lib/foam/otel/redaction.rb', line 766

def mask_status(status, config)
  description = status.respond_to?(:description) ? status.description : nil
  return status unless description.is_a?(String) && !description.empty?

  scanned = redact_leaf_string(description, config)
  return status if scanned == description

  case status.code
  when OpenTelemetry::Trace::Status::ERROR then OpenTelemetry::Trace::Status.error(scanned)
  when OpenTelemetry::Trace::Status::OK then OpenTelemetry::Trace::Status.ok(scanned)
  else OpenTelemetry::Trace::Status.unset(scanned)
  end
end

.masked_value_for(value, config, value_scan) ⇒ Object

A redact_keys match: the value-pattern scan runs FIRST (design §V.0 — a shaped secret must never survive as a 4-char tail reveal); a V hit is terminal for its span. Otherwise: tail-mask a scalar leaf, redact anything else in full.



531
532
533
534
535
536
537
538
539
 
# File 'lib/foam/otel/redaction.rb', line 531

def masked_value_for(value, config, value_scan)
  return value.map { REDACTED } if value.is_a?(Array)

  if value_scan && value.is_a?(String)
    scanned = scan_value_secrets(value, heuristics_enabled?(config))
    return scanned unless scanned == value
  end
  scalar_or_full(value)
end

.match_any_substring?(lower, keys) ⇒ Boolean

Returns:

  • (Boolean) 


597
598
599
600
601
 
# File 'lib/foam/otel/redaction.rb', line 597

def match_any_substring?(lower, keys)
  return false if keys.nil?

  keys.any? { |k| lower.include?(k) }
end

.pem_spans(text, low) ⇒ Object

private-key-block linear scan (design §V.2 normative algorithm): every -----BEGIN … PRIVATE KEY block is a secret; mask BEGIN→END inclusive, or BEGIN→end-of-value when END is missing within PEM_BLOCK_CAP (truncated capture ⇒ fail closed). PUBLIC KEY / CERTIFICATE blocks never match. Covers GCP service-account JSON via its embedded PEM.



333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
 
# File 'lib/foam/otel/redaction.rb', line 333

def pem_spans(text, low)
  spans = []
  start = 0
  n = text.length
  while (i = low.index("-----begin", start))
    unless low[i, 40].include?("private key")
      start = i + 10
      next
    end
    limit = [n, i + SECRET_PEM_BLOCK_CAP].min
    close = -1
    probe = i + 10
    while probe < limit
      j = low.index("-----end", probe)
      break if j.nil? || j >= limit

      if low[j, 40].include?("private key") && (k = text.index("-----", j + 8)) && k < limit
        close = k + 5
        break
      end
      probe = j + 8
    end
    if close > i
      spans << [i, close]
      start = close
    else
      spans << [i, n] # truncated ⇒ mask to end of value, fail-closed
      break
    end
  end
  spans
end

.redact(value) ⇒ Object

---- public helper (section 0 helper set) ---------------------------- Mask a value the caller knows is sensitive. Scalars get the tail mask; anything structured redacts in full. Fail closed: never return the unredacted payload — empty string on hard failure.



88
89
90
91
92
93
94
95
96
97
 
# File 'lib/foam/otel/redaction.rb', line 88

def redact(value)
  case value
  when String, Numeric, Symbol then tail_mask(value.to_s)
  else REDACTED
  end
rescue StandardError, SystemStackError
  # SystemStackError is not a StandardError: a value whose #to_s recurses
  # unboundedly must still degrade to empty here, never escape (rule 9).
  ""
end

.redact_leaf_string(text, config) ⇒ Object

The central leaf-string pass (design §V.10 ruby row): TWO always-on controls compose — (1) the C1 tokenizer (per-pair name match over query AND fragment, per-pair value scan), (2) the whole-string value-pattern scan for shaped secrets outside k=v pairs (a bearer token in an exception message, a PEM block, a bare JWT). [REDACTED] emitted by (1) never re-matches in (2).



570
571
572
573
574
575
576
 
# File 'lib/foam/otel/redaction.rb', line 570

def redact_leaf_string(text, config)
  out = scan_value_secrets(redact_url_like(text, config), heuristics_enabled?(config))
  # Raw capture stands byte-identical: when neither pass changed
  # anything, the ORIGINAL object rides through (content- and
  # identity-preserving, pinned by the floor negative controls).
  out == text ? text : out
end

.redact_pair(pair, config) ⇒ Object 



688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
 
# File 'lib/foam/otel/redaction.rb', line 688

def redact_pair(pair, config)
  eq = pair.index("=")
  return pair if eq.nil?

  raw_name = pair[0...eq]
  decoded = decode_component(raw_name).downcase
  if floor_match?(decoded) ||
     match_any_substring?(decoded, config&.redact_keys) ||
     match_any_substring?(decoded, config&.redact_pii_keys)
    return "#{raw_name}=#{REDACTED}"
  end

  # C1.5: the pair VALUE gets the value-pattern scan on a percent-
  # decoded copy; ANY hit masks the whole pair value.
  raw_value = pair[(eq + 1)..]
  decoded_value = decode_component(raw_value)
  scanned = scan_value_secrets(decoded_value, heuristics_enabled?(config))
  scanned == decoded_value ? pair : "#{raw_name}=#{REDACTED}"
end

.redact_query_pairs(query, config) ⇒ Object

Per-pair redaction of a '&'-delimited segment, never re-encoding, never leaking on failure. X3/F-JS2 parity: at most QUERY_PAIR_CAP (256) pairs are processed, scanning segment-by-segment (never a whole-string split — that is the memory-amplification shape); the overflow remainder is replaced wholesale with [REDACTED] + one loud warning (fail-closed).



660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
 
# File 'lib/foam/otel/redaction.rb', line 660

def redact_query_pairs(query, config)
  return query if query.nil? || query.empty?

  out = []
  processed = 0
  length = query.length
  start = 0
  while start <= length
    if processed >= SECRET_QUERY_PAIR_CAP
      warn_secret_scan_once(:pair_cap,
                            "query/fragment carried more than #{SECRET_QUERY_PAIR_CAP} pairs — " \
                            "remainder masked (fail-closed, coverage contract C1.3)")
      out << REDACTED
      break
    end
    amp = query.index("&", start)
    pair = amp ? query[start...amp] : query[start..]
    out << redact_pair(pair, config)
    processed += 1
    break if amp.nil?

    start = amp + 1
  end
  out.join("&")
rescue StandardError
  REDACTED
end

.redact_url_like(text, config) ⇒ Object

---- URL / query / fragment redaction (coverage contract C1) ---------- ONE canonical tokenizer decodes-and-scans the query AND the fragment (security-fixes-design C1; the F-JS1/F-BR1 fragment blind spot never existed here because EVERY leaf string rides this pass). Split at the FIRST '#' → (head, fragment); split head at the FIRST '?' → (path, query). NO early return when '?' is absent — a fragment-only or bare k=v&k2=v2 string is still tokenized; a hash-router fragment (#/reset?token=x) has BOTH of its '?'-delimited segments tokenized. Pair NAMES are percent-decoded before matching (an access%5Ftoken name must match access_token), the floor AND the customer's lists both apply, and the replacement is the LITERAL [REDACTED] — never percent-encoded. Pair VALUES get the value-pattern scan (a fragment-embedded JWT must not survive as recognizable fragments).



616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
 
# File 'lib/foam/otel/redaction.rb', line 616

def redact_url_like(text, config)
  return text unless text.is_a?(String)

  hash_i = text.index("#")
  head = hash_i ? text[0...hash_i] : text
  fragment = hash_i ? text[(hash_i + 1)..] : nil

  q = head.index("?")
  new_head =
    if q.nil?
      head.include?("=") ? tokenize_segment(head, config) : head
    elsif head.include?("=")
      head[0..q] + redact_query_pairs(head[(q + 1)..], config)
    else
      head
    end
  return new_head if hash_i.nil?

  fq = fragment.index("?")
  new_fragment =
    if fq.nil?
      tokenize_segment(fragment, config)
    else
      "#{tokenize_segment(fragment[0...fq], config)}?#{tokenize_segment(fragment[(fq + 1)..], config)}"
    end
  "#{new_head}##{new_fragment}"
rescue StandardError
  REDACTED # fail closed: never the raw URL-shaped value on an engine fault
end

.redaction_configured?(config) ⇒ Boolean

True when the customer enumerated at least one key to redact — the CUSTOMER lists only. The credential floor runs regardless: this predicate gates only customer-list behavior (e.g. the exemplar-drop posture below), never the floor.

Returns:

  • (Boolean) 


144
145
146
147
148
149
150
151
152
 
# File 'lib/foam/otel/redaction.rb', line 144

def redaction_configured?(config)
  return false if config.nil?

  keys = config.redact_keys
  pii = config.redact_pii_keys
  (!keys.nil? && !keys.empty?) || (!pii.nil? && !pii.empty?)
rescue StandardError
  false
end

.reset_secret_scan_warnings! ⇒ Object

test hook (mirrors Foam::Otel.reset_for_tests!)



474
475
476
 
# File 'lib/foam/otel/redaction.rb', line 474

def reset_secret_scan_warnings!
  @secret_scan_warned = {}
end

.scalar_or_full(value) ⇒ Object 



541
542
543
544
545
546
 
# File 'lib/foam/otel/redaction.rb', line 541

def scalar_or_full(value)
  case value
  when String, Numeric then tail_mask(value.to_s)
  else REDACTED
  end
end

.scan_secrets_unguarded(text, heuristics) ⇒ Object

rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity



257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
 
# File 'lib/foam/otel/redaction.rb', line 257

def scan_secrets_unguarded(text, heuristics)
  if text.length > SECRET_VALUE_SCAN_CAP
    # Oversize: NEVER regex-scan (the X3 DoS shape). The linear anchor
    # scan decides: every Tier-1 rule carries a literal anchor, so an
    # oversized value cannot smuggle a Tier-1-shaped secret past this.
    if secret_anchor?(text.downcase)
      warn_secret_scan_once(:oversize,
                            "value over the #{SECRET_VALUE_SCAN_CAP}-char scan cap carries a secret " \
                            "anchor — masked whole (fail-closed)")
      return REDACTED
    end
    return text
  end

  low = text.downcase
  # H2 runs even with no Tier-1 anchor (a bare high-entropy token has none).
  return REDACTED if heuristics && h2_whole_token?(text)
  return text unless secret_anchor?(low)
  # PuTTY: the PPK header means Private-Lines ride inline — whole value.
  return REDACTED if low.include?("putty-user-key-file")

  spans = pem_spans(text, low)
  SECRET_RULES_COMPILED.each do |rule|
    next unless rule[:anchors].any? { |a| low.include?(a) }

    flood = false
    group = rule[:group]
    text.scan(rule[:re]) do
      m = Regexp.last_match
      cand = m[group]
      next if cand.nil? || !tier1_filter_ok?(rule, cand)

      spans << [m.begin(group), m.end(group)]
      if spans.length > SECRET_MAX_MATCHES
        flood = true
        break
      end
    end
    if flood
      warn_secret_scan_once(:flood, "value exceeded the match-flood cap — masked whole (fail-closed)")
      return REDACTED
    end
  end
  spans.concat(h1_spans(text)) if heuristics
  return text if spans.empty?
  return REDACTED if spans.length > SECRET_MAX_MATCHES

  apply_secret_spans(text, spans)
end

.scan_unknown_object(value, config) ⇒ Object 



890
891
892
893
894
 
# File 'lib/foam/otel/redaction.rb', line 890

def scan_unknown_object(value, config)
  redact_leaf_string(value.to_s, config)
rescue StandardError, SystemStackError
  REDACTED
end

.scan_value_secrets(text, heuristics = true) ⇒ Object

---- THE VALUE-PATTERN SECRET LAYER (design CONTRACT V) ---------------- scan_value_secrets(text) masks credential-SHAPED spans to [REDACTED] regardless of key name: Tier 1 (frozen SECRET_VALUE_RULES + the private-key/PuTTY linear scans) ALWAYS; the H1/H2 heuristic tier when heuristics (the config's secret_heuristics, default true).

Execution discipline (design §V.5, mandatory — the engines backtrack):

1. compile once at load (a bad pattern fails the require, loudly);
2. anchor pre-filter: no lowercase literal anchor in the value ⇒ NO
 regex runs at all (the zero-cost path for normal telemetry);
3. values over VALUE_SCAN_CAP (256 KiB) are NEVER regex-scanned —
 any anchor hit ⇒ whole value masked (fail-closed), none ⇒ pass;
4. more than PATTERN_MAX_MATCHES_PER_VALUE matches ⇒ whole value
 masked (flood cap);
5. per-Regexp timeout (Ruby >= 3.2) — a timeout fires the
 fail-closed whole-value mask via the rescue below;
6. ANY throw inside the scanner ⇒ [REDACTED], never the raw value.


239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/foam/otel/redaction.rb', line 239

def scan_value_secrets(text, heuristics = true)
  return text unless text.is_a?(String) && !text.empty?

  # Invalid encodings would raise inside String#downcase/regex match —
  # scan a scrubbed COPY; when nothing matches, the original bytes ride
  # through untouched (raw capture stands).
  work = text.valid_encoding? ? text : scrub_utf8(text)
  work = text unless work.is_a?(String)
  result = scan_secrets_unguarded(work, heuristics)
  result.equal?(work) ? text : result
rescue StandardError, SystemStackError
  # Fail closed (design §V.5.8): a scanner fault — Regexp::TimeoutError
  # included — ships the mask, NEVER the raw value.
  warn_secret_scan_once(:fault, "value-pattern scan failed for a value — masked whole (fail-closed)")
  REDACTED
end

.scrub_utf8(value) ⇒ Object

---- encoding hygiene (NOT redaction) --------------------------------- OTLP protobuf requires valid UTF-8, and the upstream Ruby exporter's encode failure drops the WHOLE batch — one invalid-encoding attribute from a helper call would silently kill every healthy span/log/metric batched beside it (the rule-15 silent-loss shape; observed live: the conformance /hostile route's bad-UTF-8 attribute dropping the /health span's batch). Foam scrubs String values to valid UTF-8 AT CAPTURE on its helper surface: invalid bytes become U+FFFD — content otherwise untouched (this is byte-validity, not masking; raw capture stands).



108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/foam/otel/redaction.rb', line 108

def scrub_utf8(value)
  case value
  when String
    if value.encoding == Encoding::UTF_8
      value.valid_encoding? ? value : value.scrub("�")
    elsif value.encoding == Encoding::ASCII_8BIT
      # binary payloads carry no declared charset — treat as UTF-8 bytes
      value.dup.force_encoding(Encoding::UTF_8).scrub("�")
    else
      value.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
    end
  when Array
    value.map { |entry| scrub_utf8(entry) }
  else
    value
  end
rescue StandardError, SystemStackError
  "" # a value that cannot be made encodable degrades, never poisons a batch
end

.secret_anchor?(low) ⇒ Boolean

The single multi-substring anchor pre-filter over the lowercase copy.

Returns:

  • (Boolean) 


309
310
311
 
# File 'lib/foam/otel/redaction.rb', line 309

def secret_anchor?(low)
  SECRET_ALL_ANCHORS.any? { |a| low.include?(a) }
end

.secret_placeholder?(cand) ⇒ Boolean

Returns:

  • (Boolean) 


409
410
411
 
# File 'lib/foam/otel/redaction.rb', line 409

def secret_placeholder?(cand)
  SECRET_PLACEHOLDER_RES.any? { |re| re.match?(cand) }
end

.secret_stopword?(cand) ⇒ Boolean

Returns:

  • (Boolean) 


413
414
415
416
 
# File 'lib/foam/otel/redaction.rb', line 413

def secret_stopword?(cand)
  low = cand.downcase
  SECRET_VALUE_STOPWORDS.any? { |w| low.include?(w) }
end

.shannon_entropy(str) ⇒ Object 



418
419
420
421
422
423
 
# File 'lib/foam/otel/redaction.rb', line 418

def shannon_entropy(str)
  return 0.0 if str.empty?

  n = str.length.to_f
  str.each_char.tally.each_value.sum { |c| p = c / n; -p * Math.log2(p) }
end

.tail_mask(str) ⇒ Object

---- the two mask shapes --------------------------------------------- A redact_keys match on a SCALAR leaf: eight mask chars, plus the last four when the value is long enough to make a tail safe. The mask body is always exactly eight chars — a secret's length is itself information (entropy-safe masking).



133
134
135
136
137
138
 
# File 'lib/foam/otel/redaction.rb', line 133

def tail_mask(str)
  s = str.to_s
  return MASK_BODY if s.length < TAIL_MIN_LENGTH

  "#{MASK_BODY}#{s[-4..]}"
end

.tier1_filter_ok?(rule, cand) ⇒ Boolean

Tier-1 post-filters (design §V.2): optional entropy floor, optional digit / class-mix requirement. NO value allowlist for Tier 1 — doc examples (AKIAIOSFODNN7EXAMPLE) are deliberately masked.

Returns:

  • (Boolean) 


316
317
318
319
320
321
322
323
324
325
326
 
# File 'lib/foam/otel/redaction.rb', line 316

def tier1_filter_ok?(rule, cand)
  entropy = rule[:entropy]
  return false if entropy && shannon_entropy(cand) < entropy

  case rule[:filter]
  when :digit then cand.match?(/[0-9]/)
  when :classmix
    (cand.match?(/[A-Z]/) && cand.match?(/[a-z]/)) || cand.match?(/[0-9]/) || cand.match?(%r{[+/=]})
  else true
  end
end

.tokenize_segment(segment, config) ⇒ Object

A segment is tokenized per-pair iff it contains '='; otherwise it still gets the whole-segment value scan (a bare fragment secret).



648
649
650
651
652
 
# File 'lib/foam/otel/redaction.rb', line 648

def tokenize_segment(segment, config)
  return redact_query_pairs(segment, config) if segment.include?("=")

  scan_value_secrets(segment, heuristics_enabled?(config))
end

.warn_secret_scan_once(key, message) ⇒ Object

Rate-limited (once per process per class) scanner warnings — loud (rule 15) but never a log flood an adversary controls.



463
464
465
466
467
468
469
470
471
 
# File 'lib/foam/otel/redaction.rb', line 463

def warn_secret_scan_once(key, message)
  @secret_scan_warned ||= {}
  return if @secret_scan_warned[key]

  @secret_scan_warned[key] = true
  Diagnostics.warn(message)
rescue StandardError
  nil
end