Module: Restless::Redact

Defined in:
lib/restless/redact.rb

Overview

CONTRACT.md section 4. Redaction for sensitive values in captured requests. Runs at the single choke point before anything enters the upload queue; no adapter may bypass it.

Format: <REDACTED:<length>[:<last4>]>

Constant Summary collapse

TAIL_MIN_LENGTH = 
8
TAIL_CHARS = 
4
DEFAULT_HEADER_DENYLIST =

REDACT-011. Matched after REDACT-010 normalization.

%w[
  authorization
  cookie
  set-cookie
  proxy-authorization
  x-api-key
  x-auth-token
].freeze
DEFAULT_BODY_KEY_DENYLIST =

REDACT-012.

%w[
  password
  pass
  pwd
  token
  secret
  apikey
  accesstoken
  refreshtoken
  idtoken
  sessionid
  ssn
  creditcard
  ccnumber
  cvv
  cvc
].freeze
DEFAULT_QUERY_PARAM_DENYLIST =

REDACT-013. Query params use the same list as body keys.

DEFAULT_BODY_KEY_DENYLIST
SCHEME_PREFIX_HEADERS =

REDACT-016. Headers that carry an HTTP auth-scheme prefix. For these the scheme word survives, so a debugger reading the dashboard can see at a glance whether the caller used Bearer, Basic or something custom.

Set.new(%w[authorization proxyauthorization]).freeze
MAX_BODY_BYTES =

REDACT-030. 256 KiB, measured in UTF-8 bytes.

262_144
HEX_PAIR_RE =

PRIM-006. Hex digits are validated explicitly rather than handed to String#to_i(16), which silently returns 0 for garbage.

/\A[0-9a-fA-F]{2}\z/.freeze
UNRESERVED_BYTES =

REDACT-028. The RFC 3986 unreserved set, spelled out. No two languages' builtins agree: encodeURIComponent also leaves !'()* alone, Ruby's CGI.escape turns a space into + and escapes ~, and Go's differs from both.

begin
  set = Array.new(256, false)
  "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~"
    .each_byte { |b| set[b] = true }
  set.freeze
end

Class Method Summary collapse

Class Method Details

.build_deny_set(defaults, extra) ⇒ Object 



106
107
108
109
110
 
# File 'lib/restless/redact.rb', line 106

def build_deny_set(defaults, extra)
  # REDACT-014: defaults are ALWAYS applied. Configuration extends the
  # lists; it can never shrink or replace them.
  Set.new((defaults + Array(extra)).map { |n| normalize_name(n.to_s) })
end

.contains_denied_key?(val, deny) ⇒ Boolean

REDACT-021. Walk the PARSED value rather than pattern-matching the raw text, so every SDK reaches the same verdict with its own JSON parser and there is no regex dialect or escape decoding to get wrong.

Returns:

  • (Boolean) 


278
279
280
281
282
283
284
285
286
287
288
289
290
 
# File 'lib/restless/redact.rb', line 278

def contains_denied_key?(val, deny)
  return false if val.nil?
  return val.any? { |v| contains_denied_key?(v, deny) } if val.is_a?(Array)

  if val.is_a?(Hash)
    val.each do |k, v|
      return true if deny.include?(normalize_name(k.to_s))
      return true if contains_denied_key?(v, deny)
    end
  end

  false
end

.normalize_name(name) ⇒ Object

REDACT-010. Lowercase (PRIM-020, FULL Unicode) and drop every - and _, so api_key, apiKey, API-KEY and APIKEY all match apikey.

Full Unicode, and that is a SECURITY requirement rather than a cosmetic one. This function is what decides whether a value gets redacted, so the safe direction is to fold MORE aggressively, never less. A header named x-api-Key or a body key toKen whose K is U+212A KELVIN SIGN full-lowercases to x-api-key / token -- both denylisted -- and the reference redacts them.

An ASCII-only fold leaves the KELVIN spelling unmatched and ships the plaintext secret to the dashboard. Do not reintroduce it, and do not substitute casefold-style folding either (PRIM-020).



102
103
104
 
# File 'lib/restless/redact.rb', line 102

def normalize_name(name)
  Text.full_lower(name).delete("-_")
end

.percent_decode(value) ⇒ Object

REDACT-029. + is a space, %XX is a byte, a % not followed by two hex digits is literal, and invalid UTF-8 in the decoded bytes becomes U+FFFD instead of raising.

Iterates CODE POINTS. Node's original indexed UTF-16 code units, walked into the middle of an astral character and handed each surrogate half to the encoder separately, turning one emoji into two U+FFFD -- so the sentinel reported the wrong length.



180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
 
# File 'lib/restless/redact.rb', line 180

def percent_decode(value)
  bytes = []
  chars = value.chars
  i = 0
  len = chars.length
  while i < len
    ch = chars[i]
    if ch == "+"
      bytes << 0x20
      i += 1
      next
    end
    if ch == "%"
      pair = chars[i + 1, 2]
      if pair && pair.length == 2
        hex = pair.join
        if HEX_PAIR_RE.match?(hex)
          bytes << hex.to_i(16)
          i += 3
          next
        end
      end
    end
    Text.to_utf8(ch).each_byte { |b| bytes << b }
    i += 1
  end
  Text.bytes_to_utf8(bytes)
end

.percent_encode(value) ⇒ Object

REDACT-028.



164
165
166
167
168
169
170
 
# File 'lib/restless/redact.rb', line 164

def percent_encode(value)
  out = +""
  Text.to_utf8(value).each_byte do |b|
    out << (UNRESERVED_BYTES[b] ? b.chr : format("%%%02X", b))
  end
  out
end

.redact_body(body, content_type, extra = []) ⇒ Object

REDACT-020..024.

When the body contains NOTHING to redact, the caller's original string is returned BYTE FOR BYTE rather than a re-serialized copy. That is a fidelity requirement, not an optimization: a parse/serialize round trip is lossy in every language, differently, and re-serialization is exactly where SDKs disagree.



299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
 
# File 'lib/restless/redact.rb', line 299

def redact_body(body, content_type, extra = [])
  return body if body.nil? || body.empty?
  # REDACT-023: case-insensitive substring match on the content type.
  return body unless Text.full_lower(content_type.to_s).include?("application/json")

  begin
    # max_nesting: false because the reference parser has no depth limit;
    # a 200-deep body must not silently take a different branch here.
    parsed = JSON.parse(body, max_nesting: false)
  rescue StandardError
    # REDACT-024: a body that does not parse passes through unchanged.
    return body
  end

  deny = build_deny_set(DEFAULT_BODY_KEY_DENYLIST, extra)
  return body unless contains_denied_key?(parsed, deny)

  # PRIM-030 (compact), PRIM-031 (insertion order), PRIM-032 (literal
  # UTF-8). Ruby's JSON encoder does all three by default.
  JSON.generate(redact_json_value(parsed, deny))
rescue StandardError
  body
end

.redact_headers(headers, extra = []) ⇒ Object

REDACT-018, REDACT-019. Returns a NEW hash; never mutates the caller's.



139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
 
# File 'lib/restless/redact.rb', line 139

def redact_headers(headers, extra = [])
  deny = build_deny_set(DEFAULT_HEADER_DENYLIST, extra)
  out = {}
  headers.each do |key, value|
    norm = normalize_name(key.to_s)
    unless deny.include?(norm)
      out[key] = value
      next
    end

    if SCHEME_PREFIX_HEADERS.include?(norm)
      split = split_auth_scheme(value.to_s)
      if split
        out[key] = "#{split[:scheme]}#{split[:gap]}" \
                   "#{redact_value(split[:credential])}"
        next
      end
    end

    out[key] = redact_value(value.to_s)
  end
  out
end

.redact_json_value(val, deny) ⇒ Object

REDACT-022. Recursively redact denylisted keys in a parsed JSON value.



247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
# File 'lib/restless/redact.rb', line 247

def redact_json_value(val, deny)
  return val if val.nil?
  return val.map { |v| redact_json_value(v, deny) } if val.is_a?(Array)

  if val.is_a?(Hash)
    out = {}
    val.each do |k, v|
      if deny.include?(normalize_name(k.to_s))
        # REDACT-004: a non-string value becomes the bare `<REDACTED>`;
        # null is left alone (there is nothing to leak, and nulling it out
        # would lose schema information).
        out[k] = if v.nil?
                   nil
                 elsif v.is_a?(String)
                   redact_value(v)
                 else
                   "<REDACTED>"
                 end
      else
        out[k] = redact_json_value(v, deny)
      end
    end
    return out
  end

  val
end

.redact_url(url, extra = []) ⇒ Object

REDACT-025..027. Rewrite denylisted query values IN PLACE.

Scheme, host, port, path, parameter order, separators and fragment come through byte for byte; only the matched values change. Parsing and re-serializing through a URL library is wrong twice over: it loses repeated parameters (?token=a&token=b collapses to one) and it applies WHATWG normalization no other language reproduces.



216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
 
# File 'lib/restless/redact.rb', line 216

def redact_url(url, extra = [])
  deny = build_deny_set(DEFAULT_QUERY_PARAM_DENYLIST, extra)

  q = url.index("?")
  return url if q.nil?

  head = url[0, q + 1]
  rest = url[(q + 1)..-1] || ""

  hash = rest.index("#")
  query = hash.nil? ? rest : rest[0, hash]
  tail  = hash.nil? ? "" : rest[hash..-1]
  return url if query.empty?

  # `split("&", -1)`: without the negative limit Ruby drops trailing empty
  # fields, so `?a=1&` would come back as `?a=1`.
  parts = query.split("&", -1).map do |pair|
    eq = pair.index("=")
    next pair if eq.nil?

    raw_key = pair[0, eq]
    raw_val = pair[(eq + 1)..-1] || ""
    next pair unless deny.include?(normalize_name(percent_decode(raw_key)))

    "#{raw_key}=#{percent_encode(redact_value(percent_decode(raw_val)))}"
  end

  head + parts.join("&") + tail
end

.redact_value(value) ⇒ Object

REDACT-001, REDACT-002. Length and tail are counted in CODE POINTS.

An emoji is 1 code point, 2 UTF-16 code units and 4 UTF-8 bytes; a sentinel built from any other unit disagrees with every other SDK for the same secret, and a byte-wise tail slice produces an ill-formed string.



81
82
83
84
85
86
87
 
# File 'lib/restless/redact.rb', line 81

def redact_value(value)
  points = value.chars
  len = points.length
  return "<REDACTED:#{len}>" if len < TAIL_MIN_LENGTH

  "<REDACTED:#{len}:#{points.last(TAIL_CHARS).join}>"
end

.split_auth_scheme(value) ⇒ Object

REDACT-016. Split Bearer <credential> into its three parts, or nil when there is no scheme prefix to preserve.

An explicit scan over CODE POINTS, never a regex. The obvious pattern ^(\S+)(\s+)(\S.*)$ is unportable and fails silently: JavaScript's . excludes CR/LS/PS so a credential containing a stray CR falls through to whole-value redaction, while a Ruby or Python . with DOTALL matches and preserves the scheme. Same header, same input, two captured values.



120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
 
# File 'lib/restless/redact.rb', line 120

def split_auth_scheme(value)
  chars = value.chars
  i = 0
  i += 1 while i < chars.length && !Text.ws?(chars[i])
  # No whitespace at all, or the value starts with it: nothing to keep.
  return nil if i.zero? || i >= chars.length

  j = i
  j += 1 while j < chars.length && Text.ws?(chars[j])
  return nil if j >= chars.length # whitespace but no credential after it

  {
    scheme: chars[0, i].join,
    gap: chars[i, j - i].join,
    credential: chars[j..-1].join
  }
end

.truncate_body(body, max_bytes) ⇒ Object

REDACT-030..032. The cut is made at the byte limit and then backed off to the nearest character boundary, so the kept prefix is always a complete sequence of Unicode scalar values.



326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
 
# File 'lib/restless/redact.rb', line 326

def truncate_body(body, max_bytes)
  return body if body.nil? || body.empty?

  buf = Text.to_utf8(body).dup.force_encoding(Encoding::ASCII_8BIT)
  total = buf.bytesize
  return body if total <= max_bytes

  cut = max_bytes
  cut = 0 if cut.negative?
  # Walk back off any UTF-8 continuation byte (0b10xxxxxx).
  cut -= 1 while cut.positive? && (buf.getbyte(cut) & 0xC0) == 0x80

  kept = buf.byteslice(0, cut).force_encoding(Encoding::UTF_8)
  "#{kept}\n[...TRUNCATED: original #{total} bytes]"
end