Class: Html2rss::AutoSource::Scraper::JsonState

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb

Overview

Scrapes JSON state blobs embedded in script tags such as Next.js, Nuxt, or custom window globals. The scraper searches <script type="application/json"> tags and well-known JavaScript globals for arrays of article-like hashes and normalises them to a structure compatible with Html2rss::Html::ArticleExtractor.

Defined Under Namespace

Modules: ArticleNormalizer, CandidateDetector, DocumentScanner, ValueFinder

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body, url:, **_opts) ⇒ JsonState

Returns a new instance of JsonState.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

  • url (String, Html2rss::Url) —

    page URL used to resolve relative links

  • _opts (Hash) —

    scraper-specific options

Options Hash (**_opts):

  • :_reserved (Object) —

    reserved for future scraper-specific options



36
37
38
39
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 36

def initialize(parsed_body, url:, **_opts)
  @parsed_body = parsed_body
  @url = url
end

Instance Attribute Details

#parsed_body ⇒ Object (readonly)

Returns the value of attribute parsed_body.



41
42
43
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 41

def parsed_body
  @parsed_body
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

Returns:

  • (Boolean) 


19
20
21
22
23
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 19

def articles?(parsed_body)
  return false unless parsed_body

  DocumentScanner.json_documents(parsed_body).any? { CandidateDetector.candidate_array?(_1) }
end

.json_documents(parsed_body) ⇒ Array<Hash, Array>

Returns parsed JSON documents discovered in the response body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed HTML document

Returns:

  • (Array<Hash, Array>) —

    parsed JSON documents discovered in the response body



27
28
29
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 27

def json_documents(parsed_body)
  DocumentScanner.json_documents(parsed_body)
end

.options_key ⇒ Symbol

Returns scraper config key.

Returns:

  • (Symbol) —

    scraper config key



15
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 15

def self.options_key = :json_state

Instance Method Details

#each {|Hash{Symbol => Object}| ... } ⇒ Enumerator, void

Returns article enumerator when no block is given.

Yields:

  • (Hash{Symbol => Object}) —

    normalized article hash

Returns:

  • (Enumerator, void) —

    article enumerator when no block is given



50
51
52
53
54
55
56
57
58
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 50

def each
  return enum_for(:each) unless block_given?

  json_documents.each do |document|
    discover_articles(document) do |article|
      yield article if article
    end
  end
end

#extractable? ⇒ Boolean

Returns true when the page contains article-like arrays in JSON state.

Returns:

  • (Boolean) —

    true when the page contains article-like arrays in JSON state



44
45
46
 
# File 'lib/html2rss/auto_source/scraper/json_state.rb', line 44

def extractable?
  json_documents.any? { CandidateDetector.candidate_array?(_1) }
end