Module: ReactOnRails::LenientJson

Defined in:
lib/react_on_rails/lenient_json.rb

Overview

Parses JSON produced by the JavaScript renderer, tolerating a Unicode edge case that crashes Ruby's stock JSON.parse.

The problem

JavaScript strings are UTF-16, so an astral character such as 😀 is stored as a surrogate pair (a high half U+D800..U+DBFF followed by a low half U+DC00..U+DFFF). When application code slices a string in the middle of such a pair -- e.g. truncating text for an excerpt with +slice+/+substring+ -- it leaves a lone surrogate: half a character. JSON.stringify happily serializes it as the escape \ud83d, but Ruby's JSON.parse rejects that escape with JSON::ParserError: incomplete surrogate pair, taking down the whole server render over one bad character.

The fix

Ruby's handling of a lone surrogate escape splits three ways, so a rescue alone is not enough:

* a lone HIGH surrogate usually *raises* (json >= 2.17) -- we rescue and repair;
* a lone LOW surrogate is *accepted* and returns invalid UTF-8 -- no raise;
* a lone HIGH followed by a non-low escape (e.g. "\ud83d") is *silently mis-decoded*
into a bogus astral character -- no raise;
* older json (< 2.17) may silently degrade a lone HIGH to "?" -- no raise.

So we repair on both paths, gated by a cheap check: only payloads whose text actually contains a "\ud" escape can be affected (+JSON.stringify+ emits "\u" only for control characters and surrogates, so this is false for the vast majority of payloads and the common path pays only one substring scan). When present, we run the full repair and reparse. A lone surrogate is replaced with U+FFFD (+�+), the Unicode "replacement character" a browser would render for broken text anyway -- passing the content through instead of failing the render.

Input assumption: valid UTF-8

Every input reaching these parse sites is the output of JavaScript's JSON.stringify (renderer metadata and object payloads), which is guaranteed well-formed UTF-16 (ES2019) and therefore always encodes to valid UTF-8 -- a lone surrogate is emitted as the ASCII escape "\udXXX", never as raw invalid bytes. The repair regex thus only ever sees valid bytes. A hypothetical non-+JSON.stringify+ producer that injected raw invalid UTF-8 could make the repair raise ArgumentError instead of JSON::ParserError; that is out of scope by construction of this pipeline, not handled here. See issue #4710.

Constant Summary collapse

LONE_SURROGATE_ESCAPE =

Matches a JSON \uXXXX escape whose value is a lone surrogate, so it can be replaced with the U+FFFD escape. A valid high+low pair is captured separately and preserved (JavaScript never emits a valid pair AS escapes -- it writes the raw character -- but JSON from other producers might, so the pair branch keeps us correct there too).

(?<!\\)((?:\\\\)*)  an even run of backslashes, so literal "\\ud83d" text is skipped
                  and the escape is only matched when the backslash count is odd
group 2 + group 3   a valid high surrogate immediately followed by a low: keep as-is
group 4             a lone surrogate (high or low, U+D800..U+DFFF): replace
/
  (?<!\\)((?:\\\\)*)
  \\u(?:
    ([dD][89abAB]\h{2})\\u([dD][c-fC-F]\h{2})   # valid pair -> keep
    |([dD][89a-fA-F]\h{2})                       # lone surrogate -> U+FFFD
  )
/x

Class Method Summary collapse

Class Method Details

.parse(json) ⇒ Object

Parses json, transparently repairing lone-surrogate escapes that Ruby's JSON.parse would otherwise reject, silently mis-decode, or return as invalid UTF-8. Raises the original error for any failure not caused by a repairable lone surrogate.



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
 
# File 'lib/react_on_rails/lenient_json.rb', line 75

def parse(json)
  result = JSON.parse(json)
  # Success does not mean the text was clean: a lone LOW surrogate (invalid UTF-8), a lone
  # HIGH before a non-low escape (mis-decoded), and older-json lone HIGH ("?") all parse
  # without raising. Repair only when a "\ud" escape is actually present.
  return result unless surrogate_escape?(json)

  repaired = repair_lone_surrogates(json)
  repaired == json ? result : JSON.parse(repaired)
rescue JSON::ParserError => e
  # Genuinely malformed JSON with no surrogate escape: re-raise the true error without
  # paying for a repair scan that cannot change anything.
  raise e unless surrogate_escape?(json)

  repaired = repair_lone_surrogates(json)
  raise e if repaired == json

  JSON.parse(repaired)
end

.repair_lone_surrogates(json) ⇒ Object

Returns json with every lone-surrogate escape replaced by U+FFFD, leaving valid surrogate pairs and literal backslash text untouched. Input is assumed valid UTF-8 (see "Input assumption" above), which JSON.stringify output always is.



107
108
109
110
111
112
113
114
115
116
 
# File 'lib/react_on_rails/lenient_json.rb', line 107

def repair_lone_surrogates(json)
  json.gsub(LONE_SURROGATE_ESCAPE) do
    match = Regexp.last_match
    if match[2] # a valid high+low pair: rebuild it unchanged
      "#{match[1]}\\u#{match[2]}\\u#{match[3]}"
    else # a lone surrogate: swap for the replacement character
      "#{match[1]}#{REPLACEMENT_CHARACTER}"
    end
  end
end

.surrogate_escape?(json) ⇒ Boolean

True when the text contains a "\ud" escape (the only thing repair can act on). A single plain substring search -- it never scans for UTF-8 validity, so it is cheap on clean payloads (a few hundred ns for typical metadata). Lowercase-only is sufficient because JSON.stringify always emits surrogate escapes in lowercase (and the module's input is JSON.stringify output, per the "Input assumption" note above).

Returns:

  • (Boolean) 


100
101
102
 
# File 'lib/react_on_rails/lenient_json.rb', line 100

def surrogate_escape?(json)
  json.include?('\ud')
end