Module: Html2rss::AutoSource::Scraper

Defined in:
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The Scraper module contains all scrapers that can be used to extract articles. Each scraper should implement an each method that yields article hashes. Each scraper should also implement an articles? method that returns true if the scraper can potentially be used to extract articles from the given HTML.

Detection is intentionally shallow for most scrapers, but instance-based matching is available for scrapers that need to carry expensive selection state forward into extraction.

Defined Under Namespace

Classes: Html, JsonState, MetaOembed, Microdata, Microformats2, NoScraperFound, Schema, SemanticHtml, Sitemap, WordpressApi, XhrArticles

Constant Summary collapse

APP_SHELL_ROOT_SELECTORS =

Root markers indicating likely app-shell/client-rendered surfaces.

'#app, #root, #__next, [data-reactroot], [ng-app], [id*="app-shell"]'
APP_SHELL_MAX_ANCHORS =

Maximum anchors tolerated before app-shell detection is considered unlikely.

2
APP_SHELL_MAX_VISIBLE_TEXT_LENGTH =

Maximum visible text length tolerated for app-shell classification.

220
SCRAPER_TIERS =

Extraction tiers: merge within a tier, then stop when AutoSource has enough articles. Heuristic scrapers are separate tiers so SemanticHtml can satisfy before Html runs.

[
  [Schema, Microdata, Microformats2, JsonState, XhrArticles].freeze,
  [WordpressApi, Sitemap, MetaOembed].freeze,
  [SemanticHtml].freeze,
  [Html].freeze
].freeze
SCRAPERS =

Flat ordered list (request slot accounting, detection helpers).

SCRAPER_TIERS.flatten.freeze
HEURISTIC_SCRAPERS =

Heuristic scrapers that share one memoized SST::Document per page.

[SemanticHtml, Html].freeze
REQUEST_SESSION_SCRAPERS =

Scrapers that accept a shared follow-up request_session.

[WordpressApi, Sitemap, MetaOembed].freeze
CAPTURED_RESPONSE_SCRAPERS =

Scrapers that consume browser-captured XHR/fetch JSON bodies.

[XhrArticles].freeze

Class Method Summary collapse

Class Method Details

.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil, link_resolver: nil, captured_responses: []) ⇒ Object?

Builds a scraper when enabled; returns nil when disabled.

rubocop:disable Metrics/ParameterLists, Metrics/MethodLength -- construction context for structured and heuristic scrapers

Parameters:

  • scraper (Class)
  • parsed_body (Nokogiri::HTML::Document)
  • opts (Hash) —

    full scraper options map

  • url (Html2rss::Url, String)
  • request_session (Html2rss::RequestSession, nil) (defaults to: nil)
  • body (String, nil) (defaults to: nil)
  • document (SST::Document, nil) (defaults to: nil)
  • link_resolver (Scoring::LinkResolver, nil) (defaults to: nil)
  • captured_responses (Array<Hash>) (defaults to: []) —

    browser-captured JSON XHR/fetch bodies

Options Hash (opts:):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Object, nil) 


141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/html2rss/auto_source/scraper.rb', line 141

def self.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil,
                        link_resolver: nil, captured_responses: [])
  return unless opts.dig(scraper.options_key, :enabled)
  return if HEURISTIC_SCRAPERS.include?(scraper) && document.nil?

  scraper_opts = opts.fetch(scraper.options_key, {}).except(:enabled)
  kwargs = construction_kwargs(
    scraper,
    request_session:,
    body:,
    document:,
    link_resolver:,
    captured_responses:
  )
  scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
end

.classify_no_scraper_surface(parsed_body, body: nil) ⇒ Symbol

Classifies why scrapers could not extract from a parsed page.

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw body for blocked-surface detection

Returns:



197
198
199
200
201
202
 
# File 'lib/html2rss/auto_source/scraper.rb', line 197

def self.classify_no_scraper_surface(parsed_body, body: nil)
  return :blocked_surface if blocked_surface?(parsed_body, body:)
  return :app_shell if app_shell_surface?(parsed_body)

  :unsupported_surface
end

.extractable_instance?(instance, parsed_body) ⇒ Boolean

Parameters:

  • instance (Object)
  • parsed_body (Nokogiri::HTML::Document)

Returns:

  • (Boolean) 


163
164
165
166
167
 
# File 'lib/html2rss/auto_source/scraper.rb', line 163

def self.extractable_instance?(instance, parsed_body)
  return instance.extractable? if instance.respond_to?(:extractable?)

  instance.class.articles?(parsed_body)
end

.from(parsed_body, opts = ) ⇒ Array<Class>

Returns an array of scraper classes that claim to find articles in the parsed body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    The parsed HTML document.

  • opts (Hash) (defaults to: ) —

    The options hash.

Options Hash (opts):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Array<Class>) —

    An array of scraper classes that can handle the parsed body.



92
93
94
95
96
97
98
99
 
# File 'lib/html2rss/auto_source/scraper.rb', line 92

def self.from(parsed_body, opts = Html2rss::AutoSource::DEFAULT_CONFIG[:scraper])
  scrapers = SCRAPERS.select { |scraper| opts.dig(scraper.options_key, :enabled) }
  scrapers.select! { |scraper| scraper.articles?(parsed_body) }

  raise no_scraper_found_for(parsed_body) if scrapers.empty?

  scrapers
end

.heuristic_tier?(tier) ⇒ Boolean

Parameters:

  • tier (Array<Class>)

Returns:

  • (Boolean) 


104
105
106
 
# File 'lib/html2rss/auto_source/scraper.rb', line 104

def self.heuristic_tier?(tier)
  tier.intersect?(HEURISTIC_SCRAPERS)
end

.no_scraper_found_for(parsed_body, body: nil) ⇒ NoScraperFound

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw response body (preferred for blocked-surface checks)

Returns:



173
174
175
 
# File 'lib/html2rss/auto_source/scraper.rb', line 173

def self.no_scraper_found_for(parsed_body, body: nil)
  NoScraperFound.new(category: classify_no_scraper_surface(parsed_body, body:))
end

.normalize_sst(parsed_body) ⇒ SST::Document?

Parameters:

  • parsed_body (Nokogiri::HTML::Document)

Returns:



111
112
113
114
115
 
# File 'lib/html2rss/auto_source/scraper.rb', line 111

def self.normalize_sst(parsed_body)
  SST::Normalizer.call(parsed_body)
rescue ArgumentError
  nil
end