Module: Foam::Otel::Redaction

Defined in:
lib/foam/otel/redaction.rb

Overview

The central redaction engine: the always-on CREDENTIAL FLOOR, the always-on VALUE-PATTERN SECRET LAYER, and the customer's opt-in key lists (holistic redesign 2026-07-26, amended by the credential-floor fleet ruling — docs/decisions/credential-denylist-design.md — and the 2026-07-27 value-pattern/coverage ruling — docs/decisions/security-fixes-design.md).

THE FLOOR (unconditional, no off switch): every attribute/field whose NAME is on the frozen CREDENTIAL_KEY_DENYLIST (constants.rb) — including the semconv http.request,response.header. forms of the CREDENTIAL_HEADER_DENYLIST — masks in FULL to the literal [REDACTED], in every signal, at every depth this engine walks, before the customer's lists run. Matching is canon-exact (lowercase + dash→underscore fold, design §1.0/§2.1): NEVER substring — "authorization" does not mask "authorization_url"; "secret" does not mask "secretary". The floor consults the module-level frozen constant directly (never Config), so no configuration path can disable, shrink, or re-spell it.

THE VALUE LAYER (unconditional Tier 1; heuristic tier has the single documented secret_heuristics: false opt-out — security-fixes-design CONTRACT V): the floor is value-shape-blind by design, so this second control masks credential-SHAPED spans inside every leaf STRING the engine walks (scan_value_secrets below) — AWS/GCP/Azure keys, GitHub/GitLab/Slack/Stripe tokens, JWTs, private-key blocks, user:pass@ URI credentials, bearer/basic values, Anthropic/OpenAI keys — to the literal [REDACTED], regardless of key name. ReDoS-bounded (compile-once, anchor pre-filter, 256 KiB cap, match-flood cap, per-Regexp timeout on Ruby >= 3.2) and FAIL-CLOSED (any scanner fault masks the whole value, never a raw pass-through). Metric datapoint attributes are the one exempt surface (design §V.7 — low-cardinality by construction; pinned by the vuln suite's off-surface case).

ABOVE the floor and the value layer, redaction stays fully opt-in: UNSHAPED values under unlisted names are captured RAW. Customer redaction runs ONLY over the keys the customer enumerates, ADDITIVE:

* redact_keys     — tail-mask matching scalar values (shape-preserving,
                  e.g. ********f456); non-scalars redact in full. A
                  value-pattern hit is TERMINAL for its span: the
                  tail mask never runs over a shaped secret (a
                  4-char reveal is exactly the fragment an LLM could
                  reassemble — design §V.0).
* redact_pii_keys — erase matching values in full ([REDACTED]).

CUSTOMER matching is SUBSTRING, case-insensitive, over the lowercased key (documented Ruby behavior, unchanged). The FLOOR deliberately does NOT ride that matcher (design §2.4 — substring would over-match): it uses its own canon-exact predicate, floor_kind, below. A floor match is TERMINAL: listing a floor name in redact_keys never downgrades it to a tail mask.

Two entry points:

* redact(value)            — the public helper: mask a value the caller
                           KNOWS is sensitive (scalar → tail mask,
                           everything else → full [REDACTED]). This is
                           unconditional — the caller opted in by
                           calling it.
* mask_span_data / mask_log_record_data / mask_metric_data — the
                           exporter-boundary pass that applies the
                           floor plus the customer's key lists, run by
                           the redacting exporters
                           (redacting_exporter.rb) on door 1 AND all
                           three door-2 taps. Runs UNCONDITIONALLY —
                           the floor has no off switch; values under
                           unlisted names pass through raw.

Ruby SDK constraint (see GOTCHAS.md): a Span freezes its attributes at finish (opentelemetry-sdk span.rb) BEFORE any SpanProcessor#on_finish runs, so the key pass cannot be a processor — it runs at the exporter boundary, rebuilding the mutable SpanData/LogRecordData Structs.

Constant Summary collapse

FLOOR_HEADER_PREFIXES =

The two semconv header-attribute prefixes — the ONLY prefixes the floor peels (design §2.2). Canonical (lowercase, no dashes), so matching the canon'd name covers any case variant of the prefix too.

["http.request.header.", "http.response.header."].freeze
FLOOR_KEY_SET =

O(1) lookup set for the frozen key list (the array stays the shipped, fixture-gated constant; this is a derived view, same object identity of entries).

CREDENTIAL_KEY_DENYLIST.to_set.freeze
BODY_ATTRIBUTE_NAMES =

---- payload-capture body attributes (mandate 2026-07-28) ------------- The two value-bearing BODY names payload_capture.rb emits. A JSON-shaped string under one of these names is parsed and DEEP-masked as STRUCTURE — the credential floor and the customer's redact_keys/redact_pii_keys reach the field names inside the body exactly as they reach any nested attribute (… masks) — then re-serialized. When nothing masks, the ORIGINAL string rides through byte-identical (raw capture stands). Non-JSON bodies — and cap-TRUNCATED JSON, which no longer parses (documented limit, GOTCHAS F15) — fall back to the central leaf-string pass: urlencoded form bodies get their k=v pairs name-matched by the C1 tokenizer, and the value-pattern secret layer scans everything. Fail CLOSED: a body that parsed but cannot be re-serialized is [REDACTED], never half-masked.

%w[http.request.body http.response.body].freeze
SECRET_REGEXP_TIMEOUT =

---- compile-once value-layer machinery (design §V.5.1) --------------- Built at require time from the frozen constants.rb ruleset; a pattern that fails to compile fails the require LOUDLY — never a silent skip. Ruby >= 3.2 compiles each rule with a per-Regexp timeout (belt-and- braces on top of the authoring rules; a timeout fires the fail-closed whole-value mask via scan_value_secrets' rescue). No global Regexp.timeout mutation — customer-process hygiene.

0.25
SECRET_VALUE_SCAN_CAP = 
SECRET_SCAN_CAPS.fetch(:value_scan_cap)
SECRET_MAX_MATCHES = 
SECRET_SCAN_CAPS.fetch(:pattern_max_matches_per_value)
SECRET_PEM_BLOCK_CAP = 
SECRET_SCAN_CAPS.fetch(:pem_block_cap)
SECRET_H2_MIN = 
SECRET_SCAN_CAPS.fetch(:h2_token_min)
SECRET_H2_MAX = 
SECRET_SCAN_CAPS.fetch(:h2_token_max)
SECRET_QUERY_PAIR_CAP = 
SECRET_SCAN_CAPS.fetch(:query_pair_cap)
SECRET_RULES_COMPILED = 
SECRET_VALUE_RULES.map do |rule|
  rule.merge(re: build_secret_regexp(rule[:regex], rule[:ci])).freeze
end.freeze
SECRET_H1_RE = 
build_secret_regexp(SECRET_HEURISTIC_H1[:regex], true)
SECRET_H2_TOKEN_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:token_regex], false)
SECRET_H2_UUID_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:uuid_regex], false)
SECRET_H2_HEX_RE = 
build_secret_regexp(SECRET_HEURISTIC_H2[:hex_regex], false)
SECRET_PLACEHOLDER_RES = 
SECRET_VALUE_PLACEHOLDERS.map { |p| build_secret_regexp(p, true) }.freeze
PII_DETECT_COMPILED =

The detect tier's compiled view of the frozen fixture table (fleet ruling 2026-07-28) — same compile-once discipline (and the same per-Regexp timeout belt) as the secret rules above: a pattern that fails to compile fails the require LOUDLY, never a silent skip.

PII_DETECT_ENTITIES.transform_values do |entity|
  entity.merge(re: build_secret_regexp(entity[:regex], false)).freeze
end.freeze
SECRET_ALL_ANCHORS =

Every lowercase literal anchor across Tier 1 + the linear scans + H1 — the single pre-filter deciding whether ANY regex runs (and the whole decision for oversized values).

(
  SECRET_VALUE_RULES.flat_map { |r| r[:anchors] } +
  SECRET_LINEAR_SCAN_ANCHORS +
  SECRET_HEURISTIC_H1[:anchors]
).uniq.sort.freeze

Class Method Summary collapse

Class Method Details

.apply_detect_spans(text, spans) ⇒ Object

Replace each detected span with ITS entity's typed placeholder, left to right; overlaps resolved by earliest-start-longest-match (the secret layer's span-merge rule, fixture "span_masking"). Emitted placeholders are never rescanned — running detection twice is byte-identical (fixture "placeholder_idempotence").



568
569
570
571
572
573
574
575
576
577
578
579
580
 
# File 'lib/foam/otel/redaction.rb', line 568

def apply_detect_spans(text, spans)
  spans.sort_by! { |s, e, _| [s, -e] }
  out = +""
  prev = 0
  spans.each do |s, e, placeholder|
    next if s < prev # overlapped by an earlier-start/longer match — dropped

    out << text[prev...s] << placeholder
    prev = e
  end
  out << text[prev..]
  out
end

.apply_secret_spans(text, spans) ⇒ Object

Replace each (merged) secret span with [REDACTED], left to right, never rescanning emitted mask text (design §V.6 — zero partial reveal: no tail, no prefix, no length preservation).



428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
 
# File 'lib/foam/otel/redaction.rb', line 428

def apply_secret_spans(text, spans)
  merged = []
  spans.sort.each do |s, e|
    if !merged.empty? && s <= merged[-1][1]
      merged[-1][1] = e if e > merged[-1][1]
    else
      merged << [s, e]
    end
  end
  out = +""
  prev = 0
  merged.each do |s, e|
    next if s < prev

    out << text[prev...s] << REDACTED
    prev = e
  end
  out << text[prev..]
  out
end

.body_attribute?(name) ⇒ Boolean

EXACT match, deliberately not canon(): the only emitters spell these names verbatim (payload_capture.rb; the js cores use the same exact strings), and this predicate runs for every unmatched string attribute on the hot export path — a canon() here would allocate per attribute for nothing.

Returns:

  • (Boolean) 


676
677
678
 
# File 'lib/foam/otel/redaction.rb', line 676

def body_attribute?(name)
  BODY_ATTRIBUTE_NAMES.include?(name)
end

.build_secret_regexp(source, case_insensitive) ⇒ Object

seconds, per execution



1090
1091
1092
1093
1094
1095
1096
1097
 
# File 'lib/foam/otel/redaction.rb', line 1090

def build_secret_regexp(source, case_insensitive)
  options = case_insensitive ? Regexp::IGNORECASE : 0
  Regexp.new(source, options, timeout: SECRET_REGEXP_TIMEOUT)
rescue ArgumentError, TypeError
  # Ruby < 3.2 has no timeout: kwarg — rules 1–5 of design §V.5 carry
  # the linearity guarantee on their own there.
  Regexp.new(source, options)
end

.canon(name) ⇒ Object

---- the credential-floor predicate (design §2, self-contained) ------- canon(name): trim ASCII whitespace → ASCII-lowercase → fold '-' to '_' (design §1.0). Applied to BOTH sides of every floor comparison; the stored CREDENTIAL_KEY_DENYLIST entries are already canonical.



158
159
160
 
# File 'lib/foam/otel/redaction.rb', line 158

def canon(name)
  name.to_s.strip.downcase(:ascii).tr("-", "_")
end

.classify_key(key, config) ⇒ Object

---- key classification ---------------------------------------------- The credential floor is checked FIRST and is TERMINAL (design §4): a floor-matched name is [REDACTED] in full even when the customer also lists it in redact_keys — the tail pass must never see a floor value (a tail mask over the [REDACTED] literal would emit ******** and destroy the contract literal). Below the floor: redact_keys (:mask) wins over redact_pii_keys (:erase) when both claim a key. Customer matching is substring, case-insensitive (unchanged).



771
772
773
774
775
776
777
778
779
780
 
# File 'lib/foam/otel/redaction.rb', line 771

def classify_key(key, config)
  return :floor if floor_match?(key)
  return nil if config.nil?

  lower = key.downcase
  return :mask if match_any_substring?(lower, config.redact_keys)
  return :erase if match_any_substring?(lower, config.redact_pii_keys)

  nil
end

.decode_component(raw) ⇒ Object

Single-pass percent-decode (never recursive — a double-encoded name is decoded exactly once, matching every other core).



895
896
897
898
899
 
# File 'lib/foam/otel/redaction.rb', line 895

def decode_component(raw)
  URI.decode_www_form_component(raw)
rescue StandardError
  raw
end

.deep_mask(value, config, depth, value_scan: true) ⇒ Object

C3 (coverage contract): recurse EVERYWHERE, fail CLOSED — depth exceeded, a cycle bottoming out, or an unknown node type is replaced with [REDACTED]/a scanned stringification, never passed through raw.



1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
 
# File 'lib/foam/otel/redaction.rb', line 1040

def deep_mask(value, config, depth, value_scan: true)
  return REDACTED if depth > MAX_REDACT_DEPTH

  case value
  when Hash
    value.each_with_object({}) do |(key, val), out|
      k = key.to_s
      if malformed_header_key?(k)
        out[key] = REDACTED # C2.3, same fail-closed shape as mask_one
        next
      end
      case classify_key(k, config)
      when :floor then out[key] = floor_mask(floor_kind(k), val)
      when :erase then out[key] = REDACTED
      when :mask  then out[key] = masked_value_for(val, config, value_scan)
      else out[key] = deep_mask(val, config, depth + 1, value_scan: value_scan)
      end
    end
  when Array
    value.map { |v| deep_mask(v, config, depth + 1, value_scan: value_scan) }
  when String
    value_scan ? redact_leaf_string(value, config) : value
  when Numeric, Symbol, TrueClass, FalseClass, NilClass
    value
  else
    # C3.2 unknown node types fail CLOSED: the exporter would stringify
    # the object anyway — stringify HERE and run the leaf pass over the
    # result so a credential inside a #to_s is still caught. An object
    # whose #to_s explodes becomes the mask, never the raw object.
    value_scan ? scan_unknown_object(value, config) : value
  end
rescue StandardError
  REDACTED
end

.descend(value, config = nil, depth = 0, value_scan: true) ⇒ Object

Unmatched key: a STRING leaf runs the central free-text pass (the C1 URL/query/fragment tokenizer + the value-pattern scan — an UNSHAPED value is captured RAW, byte-identical); structured values DESCEND so a floor-listed or redact_keys/redact_pii_keys key NESTED under an innocent key is honored — bounded to MAX_REDACT_DEPTH so an adversarial/cyclic structure can never drive a downstream encoder into SystemStackError (rule 9).



736
737
738
739
740
741
742
743
 
# File 'lib/foam/otel/redaction.rb', line 736

def descend(value, config = nil, depth = 0, value_scan: true)
  cfg = config || Foam::Otel.active_config
  case value
  when Hash, Array then deep_mask(value, cfg, depth, value_scan: value_scan)
  when String then value_scan ? redact_leaf_string(value, cfg) : value
  else value
  end
end

.detect_enabled?(config) ⇒ Boolean

True only when the customer opted into at least one detect entity — the zero-cost gate for the byte-identical default path.

Returns:

  • (Boolean) 


481
482
483
484
485
486
487
488
489
 
# File 'lib/foam/otel/redaction.rb', line 481

def detect_enabled?(config)
  return false if config.nil?
  return false unless config.respond_to?(:redact_detect)

  list = config.redact_detect
  !list.nil? && !list.empty?
rescue StandardError
  false
end

.detect_filter_ok?(filter, cand) ⇒ Boolean

Returns:

  • (Boolean) 


540
541
542
543
544
545
 
# File 'lib/foam/otel/redaction.rb', line 540

def detect_filter_ok?(filter, cand)
  return true unless filter == :luhn

  digits = cand.delete("^0-9")
  digits.length.between?(13, 19) && luhn_valid?(digits)
end

.detect_pii(text, config) ⇒ Object

THE detect-tier entrypoint. Same guard shape as scan_value_secrets: scan a scrubbed COPY of an invalid-encoding value; when nothing matches, the ORIGINAL object rides through (raw capture stands, identity-preserving).



495
496
497
498
499
500
501
502
503
504
505
506
507
508
 
# File 'lib/foam/otel/redaction.rb', line 495

def detect_pii(text, config)
  return text unless text.is_a?(String) && !text.empty?

  entities = config.redact_detect
  work = text.valid_encoding? ? text : scrub_utf8(text)
  work = text unless work.is_a?(String)
  result = detect_pii_unguarded(work, entities)
  result.equal?(work) ? text : result
rescue StandardError, SystemStackError
  # Fail closed (fixture "fail_closed"): a detector fault ships the
  # mask, NEVER the raw value.
  warn_secret_scan_once(:pii_fault, "PII detection failed for a value — masked whole (fail-closed)")
  REDACTED
end

.detect_pii_unguarded(text, entities) ⇒ Object 



510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
 
# File 'lib/foam/otel/redaction.rb', line 510

def detect_pii_unguarded(text, entities)
  # Over-cap follows the secret layer's over-cap rule (fixture "caps"):
  # NEVER regex-scanned. The secret layer's anchor scan already ran and
  # decided mask-whole vs pass for oversized values; detect adds nothing.
  return text if text.length > SECRET_VALUE_SCAN_CAP

  spans = []
  PII_DETECT_COMPILED.each do |name, entity|
    next unless entities.include?(name)

    group = entity[:group]
    text.scan(entity[:re]) do
      m = Regexp.last_match
      cand = m[group]
      next if cand.nil? || !detect_filter_ok?(entity[:filter], cand)

      spans << [m.begin(group), m.end(group), entity[:placeholder]]
      if spans.length > SECRET_MAX_MATCHES
        # match flood ⇒ whole value masked (fail-closed, secret-layer parity)
        warn_secret_scan_once(:pii_flood,
                              "PII detection exceeded the match-flood cap — masked whole (fail-closed)")
        return REDACTED
      end
    end
  end
  return text if spans.empty?

  apply_detect_spans(text, spans)
end

.floor_kind(name) ⇒ Object

Classify a name against the FLOOR ONLY (never the customer lists):

:header — a semconv header attribute (http.{request,response}
        .header.<name>) whose peeled suffix canon-matches the key
        list (which contains all seven headers — subset invariant,
        design §1.1; the peel checks the FULL 52-entry list, §2.2,
        so e.g. http.request.header.x_forwarded_for masks too);
:key    — the full name itself canon-matches a key-list entry
        (full-name equality: "user.session" does NOT match
        "session"; "connect.sid" matches its own verbatim entry);
nil     — no floor match (customer lists may still apply).


182
183
184
185
186
187
188
189
190
 
# File 'lib/foam/otel/redaction.rb', line 182

def floor_kind(name)
  c = canon(name)
  FLOOR_HEADER_PREFIXES.each do |prefix|
    next unless c.start_with?(prefix)

    return FLOOR_KEY_SET.include?(c[prefix.length..]) ? :header : nil
  end
  FLOOR_KEY_SET.include?(c) ? :key : nil
end

.floor_mask(kind, value) ⇒ Object

The floor mask (design §3): a semconv header attribute is a string array (one element per header instance) — mask PER ELEMENT, preserving arity and the string-array type (OTel's own sanitize_header_values shape). Everything else replaces the ENTIRE value with the single literal [REDACTED] — scalar, object, array, number, binary. Never a tail, never length-preserving: these are credentials, not debug aids.



202
203
204
205
206
 
# File 'lib/foam/otel/redaction.rb', line 202

def floor_mask(kind, value)
  return value.map { REDACTED } if kind == :header && value.is_a?(Array)

  REDACTED
end

.floor_match?(name) ⇒ Boolean

Returns:

  • (Boolean) 


192
193
194
 
# File 'lib/foam/otel/redaction.rb', line 192

def floor_match?(name)
  !floor_kind(name).nil?
end

.h1_key_context_allowlisted?(text, match) ⇒ Boolean

Returns:

  • (Boolean) 


387
388
389
390
 
# File 'lib/foam/otel/redaction.rb', line 387

def h1_key_context_allowlisted?(text, match)
  ctx = text[match.begin(0)...match.begin(1)].to_s.downcase
  SECRET_KEY_CONTEXT_ALLOWLIST.any? { |name| ctx.include?(name) }
end

.h1_spans(text) ⇒ Object

H1 — keyword-anchored generic secret (gitleaks generic-api-key, MIT). Four-layer FP suppression (design §V.3): no-digit reject, placeholder reject, stopword reject, key-context allowlist; entropy floor 3.5 (hex candidates 3.0 with the detect-secrets all-numeric penalty).



370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
 
# File 'lib/foam/otel/redaction.rb', line 370

def h1_spans(text)
  spans = []
  text.scan(SECRET_H1_RE) do
    m = Regexp.last_match
    cand = m[1]
    next if cand.nil? || cand.match?(/\A[a-zA-Z_.\-]+\z/)
    next if secret_placeholder?(cand) || secret_stopword?(cand)
    next if h1_key_context_allowlisted?(text, m)

    floor = cand.match?(/\A[0-9a-fA-F]+\z/) ? SECRET_HEURISTIC_H1[:entropy_hex] : SECRET_HEURISTIC_H1[:entropy]
    entropy = shannon_entropy(cand)
    entropy -= 1.2 / Math.log2([2, cand.length].max) if cand.match?(/\A[0-9]+\z/)
    spans << [m.begin(1), m.end(1)] if entropy >= floor
  end
  spans
end

.h2_whole_token?(text) ⇒ Boolean

H2 — standalone high-entropy token (detect-secrets Base64HighEntropyString, Apache-2.0): ONLY when the entire trimmed value is one base64/url-safe token; class-mix + entropy >= 4.5 + not-UUID + not-pure-hex (trace/span ids, git SHAs and digests are telemetry's ambient vocabulary — recorded deviation, design §V.3) + not-placeholder + no-stopword.

Returns:

  • (Boolean) 


398
399
400
401
402
403
404
405
406
407
 
# File 'lib/foam/otel/redaction.rb', line 398

def h2_whole_token?(text)
  s = text.strip
  return false unless s.length.between?(SECRET_H2_MIN, SECRET_H2_MAX)
  return false unless SECRET_H2_TOKEN_RE.match?(s)
  return false unless s.match?(/[0-9]/) && s.match?(/[A-Z]/) && s.match?(/[a-z]/)
  return false if SECRET_H2_UUID_RE.match?(s) || SECRET_H2_HEX_RE.match?(s)
  return false if secret_placeholder?(s) || secret_stopword?(s)

  shannon_entropy(s) >= SECRET_HEURISTIC_H2[:entropy]
end

.heuristics_enabled?(config) ⇒ Boolean

The heuristic tier's single opt-out (design §V.8): secret_heuristics: false narrows ONLY H1/H2 — Tier 1, the floor and Contract C are unaffected. Unknown/absent config defaults ON.

Returns:

  • (Boolean) 


452
453
454
455
456
457
458
459
 
# File 'lib/foam/otel/redaction.rb', line 452

def heuristics_enabled?(config)
  return true if config.nil?
  return true unless config.respond_to?(:secret_heuristics)

  config.secret_heuristics != false
rescue StandardError
  true
end

.luhn_valid?(digits) ⇒ Boolean

The Luhn checksum over a plain digit string. credit_card is masked ONLY when this passes (fixture "checksum" rule) — a card-shaped number failing Luhn (a tracking id, a serial) is never masked.

Returns:

  • (Boolean) 


550
551
552
553
554
555
556
557
558
559
560
561
 
# File 'lib/foam/otel/redaction.rb', line 550

def luhn_valid?(digits)
  sum = 0
  digits.reverse.each_char.with_index do |ch, i|
    d = ch.ord - 48
    if i.odd?
      d *= 2
      d -= 9 if d > 9
    end
    sum += d
  end
  (sum % 10).zero?
end

.malformed_header_key?(name) ⇒ Boolean

C2.3 (security-fixes-design): a semconv header attribute whose peeled suffix is PURELY NUMERIC (http.request.header.0) is malformed header capture (upstream's array-of-pairs Object.keys artifact) — the real header name is unknowable, so the value is masked WHOLE, fail-closed.

Returns:

  • (Boolean) 


212
213
214
215
216
217
218
219
220
 
# File 'lib/foam/otel/redaction.rb', line 212

def malformed_header_key?(name)
  c = canon(name)
  FLOOR_HEADER_PREFIXES.each do |prefix|
    next unless c.start_with?(prefix)

    return c[prefix.length..].match?(/\A[0-9]+\z/)
  end
  false
end

.mask_attributes(attributes, config, value_scan: true) ⇒ Object

---- the floor + value-layer + customer key pass over attributes ------ Returns a NEW attributes hash with the credential floor, the value-pattern secret layer and the customer's key lists applied. Runs UNCONDITIONALLY (neither the floor nor Tier 1 has an off switch); UNSHAPED values under unmatched names pass through raw (same objects). Never raises; a key that fails to mask redacts in full rather than leaking. value_scan: false is the ONE mandated exemption — metric datapoint attributes (design §V.7): the key passes still run there, only the value-shape scan is off.



608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
 
# File 'lib/foam/otel/redaction.rb', line 608

def mask_attributes(attributes, config, value_scan: true)
  return attributes unless attributes.is_a?(Hash)

  attributes.each_with_object({}) do |(key, value), out|
    out[key] = mask_one(key.to_s, value, config, value_scan: value_scan)
  rescue StandardError
    # Fail closed at the VALUE level, loudly (rule 15): the key name only
    # — never the value — reaches the diagnostics channel. The warning is
    # itself guarded: a poisoned key (#to_s raises) must still redact.
    begin
      Diagnostics.warn("masking failed for attribute #{key.to_s.inspect} — value redacted in full " \
                       "(fail-closed, rule 14)")
    rescue StandardError, SystemStackError
      Diagnostics.warn("masking failed for an attribute (unprintable key) — value redacted in full " \
                       "(fail-closed, rule 14)")
    end
    out[key] = REDACTED
  end
end

.mask_body(body, config) ⇒ Object

A log body is AnyValue: a free-text string rides the central leaf-string pass (C1 tokenizer + value-pattern scan — UNSHAPED text is captured RAW), a structured map/array DESCENDS so a floor-listed or customer-listed key nested inside is honored (depth-guarded). Runs unconditionally.



1031
1032
1033
1034
1035
 
# File 'lib/foam/otel/redaction.rb', line 1031

def mask_body(body, config)
  deep_mask(body, config, 0)
rescue StandardError
  REDACTED
end

.mask_body_attribute(text, config, value_scan = true) ⇒ Object 



680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
 
# File 'lib/foam/otel/redaction.rb', line 680

def mask_body_attribute(text, config, value_scan = true)
  if text.lstrip.start_with?("{", "[")
    parsed = parse_json_body(text)
    unless parsed.nil?
      masked = deep_mask(parsed, config, 0, value_scan: value_scan)
      return text if masked == parsed # nothing masked: byte-identical raw capture

      begin
        return JSON.generate(masked)
      rescue StandardError, SystemStackError
        return REDACTED # fail closed once the structural pass committed
      end
    end
  end
  value_scan ? redact_leaf_string(text, config) : text
rescue StandardError, SystemStackError
  REDACTED
end

.mask_exemplars!(point, config) ⇒ Object

Exemplars carry filtered_attributes the OTLP exporter encodes RAW, one field over from the point attributes just masked — the floor must reach them too (design §2.3). Two postures, both fail-closed:

* customer keys configured → DROP exemplars entirely (the documented
pre-floor behavior for configured redaction, unchanged);
* floor only (the default path) → keep the exemplars but run their
filtered_attributes through the same engine pass, so floor-listed
names are [REDACTED] and everything else stays raw (§9: the only
default-path wire delta is floor values becoming [REDACTED]). An
exemplar whose rebuild fails is dropped, never exported raw.


1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
 
# File 'lib/foam/otel/redaction.rb', line 1005

def mask_exemplars!(point, config)
  return unless point.respond_to?(:exemplars=)

  if redaction_configured?(config)
    point.exemplars = nil
    return
  end
  exemplars = point.respond_to?(:exemplars) ? point.exemplars : nil
  return if exemplars.nil?

  point.exemplars = exemplars.filter_map do |exemplar|
    copy = exemplar.dup
    copy.filtered_attributes = mask_attributes(exemplar.filtered_attributes || {}, config)
    copy
  rescue StandardError
    nil # fail closed: a poisoned exemplar drops, never ships raw
  end
rescue StandardError
  point.exemplars = nil # fail closed at the list level too
end

.mask_log_record_data(record, config) ⇒ Object 



939
940
941
942
943
944
945
946
 
# File 'lib/foam/otel/redaction.rb', line 939

def mask_log_record_data(record, config)
  masked = record.dup
  masked.attributes = mask_attributes(record.attributes, config) if record.attributes
  masked.body = mask_body(record.body, config) if record.body
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_metric_data(metric_data, config) ⇒ Object

Door 1 masks metric attributes at capture; foreign (door-2) instrument data never passes those helpers, so the ingest metric path applies the floor + key lists here. Rebuilds every data point as a COPY (the collected points' attributes hash is the LIVE aggregation key upstream — masking in place would corrupt the customer's aggregation state). Runs UNCONDITIONALLY (the floor has no off switch).

FAIL CLOSED: returns NIL on a hard failure — the caller drops the metric loudly rather than exporting a half-masked point.



974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
 
# File 'lib/foam/otel/redaction.rb', line 974

def mask_metric_data(metric_data, config)
  masked = metric_data.dup
  points = metric_data.data_points
  if points
    masked.data_points = points.map do |point|
      copy = point.dup
      # Metric datapoint attributes are the ONE surface the value-
      # pattern layer is mandated OFF for (design §V.7: low-cardinality
      # by construction; a value-shape mask would break the customer's
      # aggregation dimensions). The floor + customer key passes still
      # apply in full.
      copy.attributes = mask_attributes(point.attributes || {}, config, value_scan: false)
      mask_exemplars!(copy, config)
      copy
    end
  end
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_one(key, value, config, value_scan: true) ⇒ Object

Ordering per value (design §V.0): floor name-match (terminal) → coverage tokenizers with per-pair name match → value-pattern scan (terminal per span) → customer keys.



631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
 
# File 'lib/foam/otel/redaction.rb', line 631

def mask_one(key, value, config, value_scan: true)
  if malformed_header_key?(key)
    # C2.3 fail-closed: a numeric header suffix means the real header
    # name (possibly Authorization) is unknowable — mask whole.
    warn_secret_scan_once(:numeric_header,
                          "malformed header-capture attribute (numeric suffix) — value redacted " \
                          "in full (fail-closed, coverage contract C2.3)")
    return REDACTED
  end
  case classify_key(key, config)
  when :floor then floor_mask(floor_kind(key), value)
  when :erase then REDACTED
  when :mask  then masked_value_for(value, config, value_scan)
  else
    # The two payload-capture BODY attributes carry customer FIELD
    # NAMES inside a JSON string — the key passes must reach them
    # (deep-redaction below), not scan them as free text.
    if value.is_a?(String) && body_attribute?(key)
      mask_body_attribute(value, config, value_scan)
    else
      descend(value, config, 0, value_scan: value_scan)
    end
  end
end

.mask_span_data(span_data, config) ⇒ Object

---- exporter-boundary pass (SpanData / LogRecordData) ---------------- Rebuild the mutable Struct with the credential floor and the customer's key lists applied. A dup keeps foam's export copy independent of the span the tenant seam saw. Runs UNCONDITIONALLY (the floor has no off switch); values under unmatched names are carried through raw.

FAIL CLOSED: on a hard failure these return NIL — never the raw payload once the pass has begun. The redacting exporters drop a nil entry (with a loud [foam] warning) rather than export a half-masked record.



910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
 
# File 'lib/foam/otel/redaction.rb', line 910

def mask_span_data(span_data, config)
  masked = span_data.dup
  masked.attributes = mask_attributes(span_data.attributes, config) if span_data.attributes
  # status.message is a value-bearing string (design §V.7): a shaped
  # secret inside an exception-derived status description must not ship.
  masked.status = mask_status(span_data.status, config) if span_data.respond_to?(:status) && span_data.status
  # Link attributes ride the key pass too: links are reachable on BOTH
  # doors and the OTLP exporter encodes link.attributes RAW. Copy-not-
  # mutate, exactly as events are handled (Link freezes its attributes at
  # construction).
  if span_data.links && !span_data.links.empty?
    masked.links = span_data.links.map do |link|
      next link unless link.respond_to?(:attributes) && link.attributes

      link.class.new(link.span_context, mask_attributes(link.attributes, config))
    end
  end
  if span_data.events && !span_data.events.empty?
    masked.events = span_data.events.map do |event|
      next event unless event.respond_to?(:attributes) && event.attributes

      event.class.new(event.name, mask_attributes(event.attributes, config), event.timestamp)
    end
  end
  masked
rescue StandardError, SystemStackError
  nil
end

.mask_status(status, config) ⇒ Object

Rebuild a span Status with the leaf-string pass applied to its description. The Status constructor is private API — the public ok/error/unset builders are used per code.



951
952
953
954
955
956
957
958
959
960
961
962
963
 
# File 'lib/foam/otel/redaction.rb', line 951

def mask_status(status, config)
  description = status.respond_to?(:description) ? status.description : nil
  return status unless description.is_a?(String) && !description.empty?

  scanned = redact_leaf_string(description, config)
  return status if scanned == description

  case status.code
  when OpenTelemetry::Trace::Status::ERROR then OpenTelemetry::Trace::Status.error(scanned)
  when OpenTelemetry::Trace::Status::OK then OpenTelemetry::Trace::Status.ok(scanned)
  else OpenTelemetry::Trace::Status.unset(scanned)
  end
end

.masked_value_for(value, config, value_scan) ⇒ Object

A redact_keys match: the value-pattern scan runs FIRST (design §V.0 — a shaped secret must never survive as a 4-char tail reveal), then the opt-in detect tier (2026-07-28 ruling: detect sits BETWEEN the secret layer and customer key masking); a V or detect hit is terminal for its span. Otherwise: tail-mask a scalar leaf, redact anything else in full.



711
712
713
714
715
716
717
718
719
720
 
# File 'lib/foam/otel/redaction.rb', line 711

def masked_value_for(value, config, value_scan)
  return value.map { REDACTED } if value.is_a?(Array)

  if value_scan && value.is_a?(String)
    scanned = scan_value_secrets(value, heuristics_enabled?(config))
    scanned = detect_pii(scanned, config) if detect_enabled?(config)
    return scanned unless scanned == value
  end
  scalar_or_full(value)
end

.match_any_substring?(lower, keys) ⇒ Boolean

Returns:

  • (Boolean) 


782
783
784
785
786
 
# File 'lib/foam/otel/redaction.rb', line 782

def match_any_substring?(lower, keys)
  return false if keys.nil?

  keys.any? { |k| lower.include?(k) }
end

.parse_json_body(text) ⇒ Object 



699
700
701
702
703
704
 
# File 'lib/foam/otel/redaction.rb', line 699

def parse_json_body(text)
  parsed = JSON.parse(text)
  parsed.is_a?(Hash) || parsed.is_a?(Array) ? parsed : nil
rescue StandardError
  nil
end

.pem_spans(text, low) ⇒ Object

private-key-block linear scan (design §V.2 normative algorithm): every -----BEGIN … PRIVATE KEY block is a secret; mask BEGIN→END inclusive, or BEGIN→end-of-value when END is missing within PEM_BLOCK_CAP (truncated capture ⇒ fail closed). PUBLIC KEY / CERTIFICATE blocks never match. Covers GCP service-account JSON via its embedded PEM.



333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
 
# File 'lib/foam/otel/redaction.rb', line 333

def pem_spans(text, low)
  spans = []
  start = 0
  n = text.length
  while (i = low.index("-----begin", start))
    unless low[i, 40].include?("private key")
      start = i + 10
      next
    end
    limit = [n, i + SECRET_PEM_BLOCK_CAP].min
    close = -1
    probe = i + 10
    while probe < limit
      j = low.index("-----end", probe)
      break if j.nil? || j >= limit

      if low[j, 40].include?("private key") && (k = text.index("-----", j + 8)) && k < limit
        close = k + 5
        break
      end
      probe = j + 8
    end
    if close > i
      spans << [i, close]
      start = close
    else
      spans << [i, n] # truncated ⇒ mask to end of value, fail-closed
      break
    end
  end
  spans
end

.redact(value) ⇒ Object

---- public helper (section 0 helper set) ---------------------------- Mask a value the caller knows is sensitive. Scalars get the tail mask; anything structured redacts in full. Fail closed: never return the unredacted payload — empty string on hard failure.



88
89
90
91
92
93
94
95
96
97
 
# File 'lib/foam/otel/redaction.rb', line 88

def redact(value)
  case value
  when String, Numeric, Symbol then tail_mask(value.to_s)
  else REDACTED
  end
rescue StandardError, SystemStackError
  # SystemStackError is not a StandardError: a value whose #to_s recurses
  # unboundedly must still degrade to empty here, never escape (rule 9).
  ""
end

.redact_leaf_string(text, config) ⇒ Object

The central leaf-string pass (design §V.10 ruby row): TWO always-on controls compose — (1) the C1 tokenizer (per-pair name match over query AND fragment, per-pair value scan), (2) the whole-string value-pattern scan for shaped secrets outside k=v pairs (a bearer token in an exception message, a PEM block, a bare JWT). [REDACTED] emitted by (1) never re-matches in (2). The OPT-IN detect tier (fleet ruling 2026-07-28) composes THIRD, after the secret layer — its typed placeholders never weaken (1)/(2), and with no opted-in entities it is a zero-cost no-op (the byte-identical default path).



754
755
756
757
758
759
760
761
 
# File 'lib/foam/otel/redaction.rb', line 754

def redact_leaf_string(text, config)
  out = scan_value_secrets(redact_url_like(text, config), heuristics_enabled?(config))
  out = detect_pii(out, config) if detect_enabled?(config)
  # Raw capture stands byte-identical: when no pass changed anything,
  # the ORIGINAL object rides through (content- and identity-preserving,
  # pinned by the floor negative controls).
  out == text ? text : out
end

.redact_pair(pair, config) ⇒ Object 



873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
 
# File 'lib/foam/otel/redaction.rb', line 873

def redact_pair(pair, config)
  eq = pair.index("=")
  return pair if eq.nil?

  raw_name = pair[0...eq]
  decoded = decode_component(raw_name).downcase
  if floor_match?(decoded) ||
     match_any_substring?(decoded, config&.redact_keys) ||
     match_any_substring?(decoded, config&.redact_pii_keys)
    return "#{raw_name}=#{REDACTED}"
  end

  # C1.5: the pair VALUE gets the value-pattern scan on a percent-
  # decoded copy; ANY hit masks the whole pair value.
  raw_value = pair[(eq + 1)..]
  decoded_value = decode_component(raw_value)
  scanned = scan_value_secrets(decoded_value, heuristics_enabled?(config))
  scanned == decoded_value ? pair : "#{raw_name}=#{REDACTED}"
end

.redact_query_pairs(query, config) ⇒ Object

Per-pair redaction of a '&'-delimited segment, never re-encoding, never leaking on failure. X3/F-JS2 parity: at most QUERY_PAIR_CAP (256) pairs are processed, scanning segment-by-segment (never a whole-string split — that is the memory-amplification shape); the overflow remainder is replaced wholesale with [REDACTED] + one loud warning (fail-closed).



845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
 
# File 'lib/foam/otel/redaction.rb', line 845

def redact_query_pairs(query, config)
  return query if query.nil? || query.empty?

  out = []
  processed = 0
  length = query.length
  start = 0
  while start <= length
    if processed >= SECRET_QUERY_PAIR_CAP
      warn_secret_scan_once(:pair_cap,
                            "query/fragment carried more than #{SECRET_QUERY_PAIR_CAP} pairs — " \
                            "remainder masked (fail-closed, coverage contract C1.3)")
      out << REDACTED
      break
    end
    amp = query.index("&", start)
    pair = amp ? query[start...amp] : query[start..]
    out << redact_pair(pair, config)
    processed += 1
    break if amp.nil?

    start = amp + 1
  end
  out.join("&")
rescue StandardError
  REDACTED
end

.redact_url_like(text, config) ⇒ Object

---- URL / query / fragment redaction (coverage contract C1) ---------- ONE canonical tokenizer decodes-and-scans the query AND the fragment (security-fixes-design C1; the F-JS1/F-BR1 fragment blind spot never existed here because EVERY leaf string rides this pass). Split at the FIRST '#' → (head, fragment); split head at the FIRST '?' → (path, query). NO early return when '?' is absent — a fragment-only or bare k=v&k2=v2 string is still tokenized; a hash-router fragment (#/reset?token=x) has BOTH of its '?'-delimited segments tokenized. Pair NAMES are percent-decoded before matching (an access%5Ftoken name must match access_token), the floor AND the customer's lists both apply, and the replacement is the LITERAL [REDACTED] — never percent-encoded. Pair VALUES get the value-pattern scan (a fragment-embedded JWT must not survive as recognizable fragments).



801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
 
# File 'lib/foam/otel/redaction.rb', line 801

def redact_url_like(text, config)
  return text unless text.is_a?(String)

  hash_i = text.index("#")
  head = hash_i ? text[0...hash_i] : text
  fragment = hash_i ? text[(hash_i + 1)..] : nil

  q = head.index("?")
  new_head =
    if q.nil?
      head.include?("=") ? tokenize_segment(head, config) : head
    elsif head.include?("=")
      head[0..q] + redact_query_pairs(head[(q + 1)..], config)
    else
      head
    end
  return new_head if hash_i.nil?

  fq = fragment.index("?")
  new_fragment =
    if fq.nil?
      tokenize_segment(fragment, config)
    else
      "#{tokenize_segment(fragment[0...fq], config)}?#{tokenize_segment(fragment[(fq + 1)..], config)}"
    end
  "#{new_head}##{new_fragment}"
rescue StandardError
  REDACTED # fail closed: never the raw URL-shaped value on an engine fault
end

.redaction_configured?(config) ⇒ Boolean

True when the customer enumerated at least one key to redact — the CUSTOMER lists only. The credential floor runs regardless: this predicate gates only customer-list behavior (e.g. the exemplar-drop posture below), never the floor.

Returns:

  • (Boolean) 


144
145
146
147
148
149
150
151
152
 
# File 'lib/foam/otel/redaction.rb', line 144

def redaction_configured?(config)
  return false if config.nil?

  keys = config.redact_keys
  pii = config.redact_pii_keys
  (!keys.nil? && !keys.empty?) || (!pii.nil? && !pii.empty?)
rescue StandardError
  false
end

.reset_secret_scan_warnings! ⇒ Object

test hook (mirrors Foam::Otel.reset_for_tests!)



595
596
597
 
# File 'lib/foam/otel/redaction.rb', line 595

def reset_secret_scan_warnings!
  @secret_scan_warned = {}
end

.scalar_or_full(value) ⇒ Object 



722
723
724
725
726
727
 
# File 'lib/foam/otel/redaction.rb', line 722

def scalar_or_full(value)
  case value
  when String, Numeric then tail_mask(value.to_s)
  else REDACTED
  end
end

.scan_secrets_unguarded(text, heuristics) ⇒ Object

rubocop:disable Metrics/MethodLength, Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity



257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
 
# File 'lib/foam/otel/redaction.rb', line 257

def scan_secrets_unguarded(text, heuristics)
  if text.length > SECRET_VALUE_SCAN_CAP
    # Oversize: NEVER regex-scan (the X3 DoS shape). The linear anchor
    # scan decides: every Tier-1 rule carries a literal anchor, so an
    # oversized value cannot smuggle a Tier-1-shaped secret past this.
    if secret_anchor?(text.downcase)
      warn_secret_scan_once(:oversize,
                            "value over the #{SECRET_VALUE_SCAN_CAP}-char scan cap carries a secret " \
                            "anchor — masked whole (fail-closed)")
      return REDACTED
    end
    return text
  end

  low = text.downcase
  # H2 runs even with no Tier-1 anchor (a bare high-entropy token has none).
  return REDACTED if heuristics && h2_whole_token?(text)
  return text unless secret_anchor?(low)
  # PuTTY: the PPK header means Private-Lines ride inline — whole value.
  return REDACTED if low.include?("putty-user-key-file")

  spans = pem_spans(text, low)
  SECRET_RULES_COMPILED.each do |rule|
    next unless rule[:anchors].any? { |a| low.include?(a) }

    flood = false
    group = rule[:group]
    text.scan(rule[:re]) do
      m = Regexp.last_match
      cand = m[group]
      next if cand.nil? || !tier1_filter_ok?(rule, cand)

      spans << [m.begin(group), m.end(group)]
      if spans.length > SECRET_MAX_MATCHES
        flood = true
        break
      end
    end
    if flood
      warn_secret_scan_once(:flood, "value exceeded the match-flood cap — masked whole (fail-closed)")
      return REDACTED
    end
  end
  spans.concat(h1_spans(text)) if heuristics
  return text if spans.empty?
  return REDACTED if spans.length > SECRET_MAX_MATCHES

  apply_secret_spans(text, spans)
end

.scan_unknown_object(value, config) ⇒ Object 



1075
1076
1077
1078
1079
 
# File 'lib/foam/otel/redaction.rb', line 1075

def scan_unknown_object(value, config)
  redact_leaf_string(value.to_s, config)
rescue StandardError, SystemStackError
  REDACTED
end

.scan_value_secrets(text, heuristics = true) ⇒ Object

---- THE VALUE-PATTERN SECRET LAYER (design CONTRACT V) ---------------- scan_value_secrets(text) masks credential-SHAPED spans to [REDACTED] regardless of key name: Tier 1 (frozen SECRET_VALUE_RULES + the private-key/PuTTY linear scans) ALWAYS; the H1/H2 heuristic tier when heuristics (the config's secret_heuristics, default true).

Execution discipline (design §V.5, mandatory — the engines backtrack):

1. compile once at load (a bad pattern fails the require, loudly);
2. anchor pre-filter: no lowercase literal anchor in the value ⇒ NO
 regex runs at all (the zero-cost path for normal telemetry);
3. values over VALUE_SCAN_CAP (256 KiB) are NEVER regex-scanned —
 any anchor hit ⇒ whole value masked (fail-closed), none ⇒ pass;
4. more than PATTERN_MAX_MATCHES_PER_VALUE matches ⇒ whole value
 masked (flood cap);
5. per-Regexp timeout (Ruby >= 3.2) — a timeout fires the
 fail-closed whole-value mask via the rescue below;
6. ANY throw inside the scanner ⇒ [REDACTED], never the raw value.


239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
 
# File 'lib/foam/otel/redaction.rb', line 239

def scan_value_secrets(text, heuristics = true)
  return text unless text.is_a?(String) && !text.empty?

  # Invalid encodings would raise inside String#downcase/regex match —
  # scan a scrubbed COPY; when nothing matches, the original bytes ride
  # through untouched (raw capture stands).
  work = text.valid_encoding? ? text : scrub_utf8(text)
  work = text unless work.is_a?(String)
  result = scan_secrets_unguarded(work, heuristics)
  result.equal?(work) ? text : result
rescue StandardError, SystemStackError
  # Fail closed (design §V.5.8): a scanner fault — Regexp::TimeoutError
  # included — ships the mask, NEVER the raw value.
  warn_secret_scan_once(:fault, "value-pattern scan failed for a value — masked whole (fail-closed)")
  REDACTED
end

.scrub_utf8(value) ⇒ Object

---- encoding hygiene (NOT redaction) --------------------------------- OTLP protobuf requires valid UTF-8, and the upstream Ruby exporter's encode failure drops the WHOLE batch — one invalid-encoding attribute from a helper call would silently kill every healthy span/log/metric batched beside it (the rule-15 silent-loss shape; observed live: the conformance /hostile route's bad-UTF-8 attribute dropping the /health span's batch). Foam scrubs String values to valid UTF-8 AT CAPTURE on its helper surface: invalid bytes become U+FFFD — content otherwise untouched (this is byte-validity, not masking; raw capture stands).



108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
 
# File 'lib/foam/otel/redaction.rb', line 108

def scrub_utf8(value)
  case value
  when String
    if value.encoding == Encoding::UTF_8
      value.valid_encoding? ? value : value.scrub("�")
    elsif value.encoding == Encoding::ASCII_8BIT
      # binary payloads carry no declared charset — treat as UTF-8 bytes
      value.dup.force_encoding(Encoding::UTF_8).scrub("�")
    else
      value.encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
    end
  when Array
    value.map { |entry| scrub_utf8(entry) }
  else
    value
  end
rescue StandardError, SystemStackError
  "" # a value that cannot be made encodable degrades, never poisons a batch
end

.secret_anchor?(low) ⇒ Boolean

The single multi-substring anchor pre-filter over the lowercase copy.

Returns:

  • (Boolean) 


309
310
311
 
# File 'lib/foam/otel/redaction.rb', line 309

def secret_anchor?(low)
  SECRET_ALL_ANCHORS.any? { |a| low.include?(a) }
end

.secret_placeholder?(cand) ⇒ Boolean

Returns:

  • (Boolean) 


409
410
411
 
# File 'lib/foam/otel/redaction.rb', line 409

def secret_placeholder?(cand)
  SECRET_PLACEHOLDER_RES.any? { |re| re.match?(cand) }
end

.secret_stopword?(cand) ⇒ Boolean

Returns:

  • (Boolean) 


413
414
415
416
 
# File 'lib/foam/otel/redaction.rb', line 413

def secret_stopword?(cand)
  low = cand.downcase
  SECRET_VALUE_STOPWORDS.any? { |w| low.include?(w) }
end

.shannon_entropy(str) ⇒ Object 



418
419
420
421
422
423
 
# File 'lib/foam/otel/redaction.rb', line 418

def shannon_entropy(str)
  return 0.0 if str.empty?

  n = str.length.to_f
  str.each_char.tally.each_value.sum { |c| p = c / n; -p * Math.log2(p) }
end

.tail_mask(str) ⇒ Object

---- the two mask shapes --------------------------------------------- A redact_keys match on a SCALAR leaf: eight mask chars, plus the last four when the value is long enough to make a tail safe. The mask body is always exactly eight chars — a secret's length is itself information (entropy-safe masking).



133
134
135
136
137
138
 
# File 'lib/foam/otel/redaction.rb', line 133

def tail_mask(str)
  s = str.to_s
  return MASK_BODY if s.length < TAIL_MIN_LENGTH

  "#{MASK_BODY}#{s[-4..]}"
end

.tier1_filter_ok?(rule, cand) ⇒ Boolean

Tier-1 post-filters (design §V.2): optional entropy floor, optional digit / class-mix requirement. NO value allowlist for Tier 1 — doc examples (AKIAIOSFODNN7EXAMPLE) are deliberately masked.

Returns:

  • (Boolean) 


316
317
318
319
320
321
322
323
324
325
326
 
# File 'lib/foam/otel/redaction.rb', line 316

def tier1_filter_ok?(rule, cand)
  entropy = rule[:entropy]
  return false if entropy && shannon_entropy(cand) < entropy

  case rule[:filter]
  when :digit then cand.match?(/[0-9]/)
  when :classmix
    (cand.match?(/[A-Z]/) && cand.match?(/[a-z]/)) || cand.match?(/[0-9]/) || cand.match?(%r{[+/=]})
  else true
  end
end

.tokenize_segment(segment, config) ⇒ Object

A segment is tokenized per-pair iff it contains '='; otherwise it still gets the whole-segment value scan (a bare fragment secret).



833
834
835
836
837
 
# File 'lib/foam/otel/redaction.rb', line 833

def tokenize_segment(segment, config)
  return redact_query_pairs(segment, config) if segment.include?("=")

  scan_value_secrets(segment, heuristics_enabled?(config))
end

.warn_secret_scan_once(key, message) ⇒ Object

Rate-limited (once per process per class) scanner warnings — loud (rule 15) but never a log flood an adversary controls.



584
585
586
587
588
589
590
591
592
 
# File 'lib/foam/otel/redaction.rb', line 584

def warn_secret_scan_once(key, message)
  @secret_scan_warned ||= {}
  return if @secret_scan_warned[key]

  @secret_scan_warned[key] = true
  Diagnostics.warn(message)
rescue StandardError
  nil
end