Class: Html2rss::AutoSource::Scraper::JsonState

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb

Overview

Scrapes JSON state blobs embedded in script tags such as Next.js, Nuxt, or custom window globals. The scraper searches <script type="application/json"> tags and well-known JavaScript globals for arrays of article-like hashes and normalises them to a structure compatible with Html2rss::Html::ArticleExtractor.

Defined Under Namespace

Modules: ArticleNormalizer, CandidateDetector, DocumentScanner, ValueFinder

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body, url:, **_opts) ⇒ JsonState

Returns a new instance of JsonState.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

  • url (String, Html2rss::Url) —

    page URL used to resolve relative links

  • _opts (Hash) —

    scraper-specific options

Options Hash (**_opts):

  • :_reserved (Object) —

    reserved for future scraper-specific options



66
67
68
69
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 66

def initialize(parsed_body, url:, **_opts)
  @parsed_body = parsed_body
  @url = url
end

Instance Attribute Details

#parsed_body ⇒ Object (readonly)

Returns the value of attribute parsed_body.



71
72
73
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 71

def parsed_body
  @parsed_body
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

Returns:

  • (Boolean) 


19
20
21
22
23
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 19

def articles?(parsed_body)
  return false unless parsed_body

  DocumentScanner.json_documents(parsed_body).any? { CandidateDetector.candidate_array?(_1) }
end

.discover_articles(document, base_url:) {|Hash{Symbol => Object}, nil| ... } ⇒ void

This method returns an undefined value.

Walks a JSON document tree and yields normalized article hashes. Shared with XhrArticles so XHR-captured JSON reuses one discovery algorithm.

Parameters:

  • document (Hash, Array, Object) —

    parsed JSON document node

  • base_url (String, Html2rss::Url) —

    base URL for relative link resolution

Yields:

  • (Hash{Symbol => Object}, nil) —

    normalized article hash



38
39
40
41
42
43
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 38

def discover_articles(document, base_url:, &block)
  case document
  when Array then handle_array(document, base_url:, &block)
  when Hash then document.each_value { discover_articles(_1, base_url:, &block) if traversable?(_1) }
  end
end

.json_documents(parsed_body) ⇒ Array<Hash, Array>

Returns parsed JSON documents discovered in the response body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    parsed JSON documents discovered in the response body



27
28
29
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 27

def json_documents(parsed_body)
  DocumentScanner.json_documents(parsed_body)
end

.options_key ⇒ Symbol

Returns scraper config key.

Returns:

  • (Symbol) —

    scraper config key



15
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 15

def self.options_key = :json_state

Instance Method Details

#each {|Hash{Symbol => Object}| ... } ⇒ Enumerator, void

Returns article enumerator when no block is given.

Yields:

  • (Hash{Symbol => Object}) —

    normalized article hash

Returns:

  • (Enumerator, void) —

    article enumerator when no block is given



80
81
82
83
84
85
86
87
88
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 80

def each
  return enum_for(:each) unless block_given?

  json_documents.each do |document|
    self.class.discover_articles(document, base_url: url) do |article|
      yield article if article
    end
  end
end

#extractable? ⇒ Boolean

Returns true when the page contains article-like arrays in JSON state.

Returns:

  • (Boolean) —

    true when the page contains article-like arrays in JSON state



74
75
76
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 74

def extractable?
  json_documents.any? { CandidateDetector.candidate_array?(_1) }
end