Module: Html2rss::AutoSource::Scraper::JsonState::DocumentScanner

Defined in:
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb

Overview

Scans DOM nodes for JSON payloads containing article data.

Constant Summary collapse

JSON_SCRIPT_SELECTOR =

Selector for JSON-only script tags.

'script[type="application/json"]'
GLOBAL_ASSIGNMENT_PATTERNS =

Regex patterns for known global JavaScript state assignments.

[
  /(?:window|self|globalThis)\.__NEXT_DATA__\s*=\s*/m,
  /(?:window|self|globalThis)\.__NUXT__\s*=\s*/m,
  /(?:window|self|globalThis)\.STATE\s*=\s*/m,
  /(?:window|self|globalThis)\.__REDUX_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__PRELOADED_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__APOLLO_STATE__\s*=\s*/m,
  /(?:window|self|globalThis)\.__remixContext\s*=\s*/m,
  /(?:window|self|globalThis)\.__sveltekit_data\s*=\s*/m,
  /(?:window|self|globalThis)\.GATSBY_STATE\s*=\s*/m,
  /(?:window|self|globalThis)\.__ember_meta\s*=\s*/m,
  /(?:window|self|globalThis)\.angular\s*=\s*/m
].freeze
GLOBAL_ASSIGNMENT_REGEXP =

Combined regex for faster matching of global assignments.

Regexp.union(GLOBAL_ASSIGNMENT_PATTERNS).freeze

Class Method Summary collapse

Class Method Details

.assignment_documents(parsed_body) ⇒ Array<Hash, Array>

Returns JSON documents extracted from global assignments.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    JSON documents extracted from global assignments



52
53
54
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 52

def assignment_documents(parsed_body)
  parsed_body.css('script').filter_map { parse_assignment(_1.text) }
end

.assignment_payload(text) ⇒ String?

Returns extracted JSON-like assignment payload.

Parameters:

  • text (String) —

    script text to inspect for known assignment patterns

Returns:

  • (String, nil) —

    extracted JSON-like assignment payload



65
66
67
68
69
70
71
72
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 65

def assignment_payload(text)
  trimmed = text.to_s.strip
  return if trimmed.empty?
  return unless trimmed.match?(GLOBAL_ASSIGNMENT_REGEXP)

  payload = trimmed.sub(GLOBAL_ASSIGNMENT_REGEXP, '')
  extract_assignment_payload(payload)
end

.coerce_javascript_object(payload) ⇒ String

Returns JSON-compatible payload string.

Parameters:

  • payload (String) —

    JavaScript object-literal payload

Returns:

  • (String) —

    JSON-compatible payload string



154
155
156
157
158
159
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 154

def coerce_javascript_object(payload)
  string = payload.dup

  # KISS approach: mutate common JS literal quirks instead of a full parser.
  strip_trailing_commas(quote_unquoted_keys(string))
end

.extract_assignment_payload(text) ⇒ String?

Returns normalized assignment payload.

Parameters:

  • text (String) —

    text potentially containing JSON-like payloads

Returns:

  • (String, nil) —

    normalized assignment payload



76
77
78
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 76

def extract_assignment_payload(text)
  extract_json_block(text) || text
end

.extract_json_block(text) ⇒ String?

Returns extracted JSON block spanning balanced brackets.

Parameters:

  • text (String) —

    text potentially containing JSON blocks

Returns:

  • (String, nil) —

    extracted JSON block spanning balanced brackets



82
83
84
85
86
87
88
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 82

def extract_json_block(text)
  start_index = text.index(/[\[{]/)
  return unless start_index

  stop_index = scan_for_json_end(text, start_index)
  text[start_index..stop_index] if stop_index
end

.json_documents(parsed_body) ⇒ Array<Hash, Array>

Returns parsed JSON documents discovered in scripts.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    parsed JSON documents discovered in scripts



35
36
37
38
39
40
41
42
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 35

def json_documents(parsed_body)
  # Use identity-based cache to avoid double-parsing of the same document.
  # WeakMap allows the Nokogiri Document (key) to be garbage collected.
  # rubocop:disable ThreadSafety/ClassInstanceVariable
  (@cache ||= ObjectSpace::WeakMap.new)[parsed_body] ||=
    script_documents(parsed_body) + assignment_documents(parsed_body)
  # rubocop:enable ThreadSafety/ClassInstanceVariable
end

.parse_assignment(text) ⇒ Hash, ...

Returns parsed assignment payload when available.

Parameters:

  • text (String) —

    script text that may contain a global assignment

Returns:

  • (Hash, Array, nil) —

    parsed assignment payload when available



58
59
60
61
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 58

def parse_assignment(text)
  payload = assignment_payload(text)
  parse_json(payload) if payload
end

.parse_js_object(payload, _original_error) ⇒ Hash, ...

Returns parsed payload after JavaScript coercion.

Parameters:

  • payload (String) —

    JavaScript object-literal payload

  • _original_error (JSON::ParserError) —

    original JSON parse error

Returns:

  • (Hash, Array, nil) —

    parsed payload after JavaScript coercion



140
141
142
143
144
145
146
147
148
149
150
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 140

def parse_js_object(payload, _original_error)
  coerced = coerce_javascript_object(payload)
  return unless coerced

  # Some sites emit JavaScript object literals (unquoted keys, trailing commas).
  # Coerce those payloads into valid JSON so we keep the same parsing pipeline.
  JSON.parse(coerced, symbolize_names: true)
rescue JSON::ParserError => error
  Html2rss::Log.debug("JsonState: failed to parse coerced JavaScript object (#{error.message})")
  nil
end

.parse_json(payload) ⇒ Hash, ...

Returns parsed payload or nil when parsing fails.

Parameters:

  • payload (String, nil) —

    JSON payload to parse

Returns:

  • (Hash, Array, nil) —

    parsed payload or nil when parsing fails



129
130
131
132
133
134
135
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 129

def parse_json(payload)
  return unless payload

  JSON.parse(payload, symbolize_names: true)
rescue JSON::ParserError => error
  parse_js_object(payload, error)
end

.quote_unquoted_keys(jsonish) ⇒ String

Returns payload with unquoted object keys quoted.

Parameters:

  • jsonish (String) —

    JSON-like string with potentially unquoted keys

Returns:

  • (String) —

    payload with unquoted object keys quoted



163
164
165
166
167
168
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 163

def quote_unquoted_keys(jsonish)
  jsonish.gsub(/(?<prefix>\A\s*|[{,\[]\s*)(?<key>[A-Za-z_]\w*)(?<suffix>\s*:)/) do
    captures = Regexp.last_match.named_captures(symbolize_names: true)
    "#{captures[:prefix]}\"#{captures[:key]}\"#{captures[:suffix]}"
  end
end

.scan_for_json_end(text, start_index) ⇒ Integer?

Returns index where the balanced JSON payload ends.

Parameters:

  • text (String) —

    text starting with a JSON object/array opening token

  • start_index (Integer) —

    index where JSON-like content starts

Returns:

  • (Integer, nil) —

    index where the balanced JSON payload ends



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 93

def scan_for_json_end(text, start_index) # rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity
  stack = []
  in_string = false
  escape = false

  i = start_index
  len = text.length
  while i < len
    char = text[i]

    if in_string
      if escape
        escape = false
      elsif char == '\\'
        escape = true
      elsif char == '"'
        in_string = false
      end
    else
      case char
      when '"' then in_string = true
      when '{' then stack << '}'
      when '[' then stack << ']'
      when '}', ']'
        expected = stack.pop
        return i if expected == char && stack.empty?
      end
    end
    i += 1
  end

  nil
end

.script_documents(parsed_body) ⇒ Array<Hash, Array>

Returns JSON documents extracted from JSON script tags.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    JSON documents extracted from JSON script tags



46
47
48
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 46

def script_documents(parsed_body)
  parsed_body.css(JSON_SCRIPT_SELECTOR).filter_map { parse_json(_1.text) }
end

.strip_trailing_commas(jsonish) ⇒ String

Returns payload without trailing commas before closing tokens.

Parameters:

  • jsonish (String) —

    JSON-like string with potential trailing commas

Returns:

  • (String) —

    payload without trailing commas before closing tokens



172
173
174
 
# File 'lib/html2rss/auto_source/scraper/json_state/document_scanner.rb', line 172

def strip_trailing_commas(jsonish)
  jsonish.gsub(/,(\s*[\]}])/, '\1')
end