Class: Html2rss::AutoSource::Scraper::XhrArticles

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/xhr_articles.rb

Overview

Extracts articles from JSON XHR/fetch responses captured during a browser-tier scrape (see RequestService::Response#captured_responses).

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(_parsed_body, url:, captured_responses: [], **_opts) ⇒ XhrArticles

Returns a new instance of XhrArticles.

Parameters:

  • _parsed_body (Nokogiri::HTML::Document, nil) —

    unused HTML document

  • url (String, Html2rss::Url) —

    page URL used to resolve relative links

  • captured_responses (Array<Hash>) (defaults to: []) —

    JSON bodies from Response#captured_responses

  • _opts (Hash) —

    scraper-specific options

Options Hash (**_opts):

  • :_reserved (Object) —

    reserved for future scraper-specific options



31
32
33
34
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 31

def initialize(_parsed_body, url:, captured_responses: [], **_opts)
  @url = url
  @captured_responses = captured_responses
end

Class Method Details

.articles?(_parsed_body) ⇒ Boolean

Not detectable from HTML alone; instance #extractable? uses captures.

Parameters:

  • _parsed_body (Nokogiri::HTML::Document, nil)

Returns:

  • (Boolean) 


24
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 24

def self.articles?(_parsed_body) = false

.options_key ⇒ Symbol

Returns scraper config key.

Returns:

  • (Symbol) —

    scraper config key



14
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 14

def self.options_key = :xhr_articles

.request_slots(_opts = {}) ⇒ Integer

Returns zero — no additional HTTP requests.

Parameters:

  • _opts (Hash) (defaults to: {}) —

    unused scraper options

Returns:

  • (Integer) —

    zero — no additional HTTP requests



18
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 18

def self.request_slots(_opts = {}) = 0

Instance Method Details

#each {|Hash{Symbol => Object}| ... } ⇒ Enumerator, void

Returns article enumerator when no block is given.

Yields:

  • (Hash{Symbol => Object}) —

    normalized article hash

Returns:

  • (Enumerator, void) —

    article enumerator when no block is given



43
44
45
46
47
48
49
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 43

def each
  return enum_for(:each) unless block_given?

  parsed_bodies.each do |doc|
    JsonState.discover_articles(doc, base_url: @url) { |article| yield article if article }
  end
end

#extractable? ⇒ Boolean

Returns true when any captured body contains article-like arrays.

Returns:

  • (Boolean) —

    true when any captured body contains article-like arrays



37
38
39
 
# File 'lib/html2rss/auto_source/scraper/xhr_articles.rb', line 37

def extractable?
  parsed_bodies.any? { |doc| JsonState::CandidateDetector.candidate_array?(doc) }
end