Module: Html2rss::AutoSource::Scraper

Defined in:
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The Scraper module contains all scrapers that can be used to extract articles. Each scraper should implement an each method that yields article hashes. Each scraper should also implement an articles? method that returns true if the scraper can potentially be used to extract articles from the given HTML.

Detection is intentionally shallow for most scrapers, but instance-based matching is available for scrapers that need to carry expensive selection state forward into extraction.

Defined Under Namespace

Classes: Html, JsonState, MetaOembed, Microdata, Microformats2, NoScraperFound, Schema, SemanticHtml, Sitemap, WordpressApi, XhrArticles

Constant Summary collapse

APP_SHELL_ROOT_SELECTORS =

Root markers indicating likely app-shell/client-rendered surfaces.

'#app, #root, #__next, [data-reactroot], [ng-app], [id*="app-shell"]'
APP_SHELL_MAX_ANCHORS =

Maximum anchors tolerated before app-shell detection is considered unlikely.

2
HIGH_ENTROPY_MIN_ANCHORS =

Minimum same-page anchors suggesting a high-entropy homepage/hub surface.

20
APP_SHELL_MAX_VISIBLE_TEXT_LENGTH =

Maximum visible text length tolerated for app-shell classification.

220
SCRAPER_TIERS =

Extraction tiers: merge within a tier, then stop when AutoSource has enough articles. Heuristic scrapers are separate tiers so SemanticHtml can satisfy before Html runs.

[
  [Schema, Microdata, Microformats2, JsonState, XhrArticles].freeze,
  [WordpressApi, Sitemap, MetaOembed].freeze,
  [SemanticHtml].freeze,
  [Html].freeze
].freeze
SCRAPERS =

Flat ordered list (request slot accounting, detection helpers).

SCRAPER_TIERS.flatten.freeze
HEURISTIC_SCRAPERS =

Heuristic scrapers that share one memoized SST::Document per page.

[SemanticHtml, Html].freeze
REQUEST_SESSION_SCRAPERS =

Scrapers that accept a shared follow-up request_session.

[WordpressApi, Sitemap, MetaOembed].freeze
CAPTURED_RESPONSE_SCRAPERS =

Scrapers that consume browser-captured XHR/fetch JSON bodies.

[XhrArticles].freeze

Class Method Summary collapse

Class Method Details

.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil, link_resolver: nil, captured_responses: []) ⇒ Object?

Builds a scraper when enabled; returns nil when disabled.

rubocop:disable Metrics/ParameterLists, Metrics/MethodLength -- construction context for structured and heuristic scrapers

Parameters:

  • scraper (Class)
  • parsed_body (Nokogiri::HTML::Document)
  • opts (Hash) —

    full scraper options map

  • url (Html2rss::Url, String)
  • request_session (Html2rss::RequestSession, nil) (defaults to: nil)
  • body (String, nil) (defaults to: nil)
  • document (SST::Document, nil) (defaults to: nil)
  • link_resolver (Scoring::LinkResolver, nil) (defaults to: nil)
  • captured_responses (Array<Hash>) (defaults to: []) —

    browser-captured JSON XHR/fetch bodies

Options Hash (opts:):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Object, nil) 


146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
 
# File 'lib/html2rss/auto_source/scraper.rb', line 146

def self.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil,
                        link_resolver: nil, captured_responses: [])
  return unless opts.dig(scraper.options_key, :enabled)
  return if HEURISTIC_SCRAPERS.include?(scraper) && document.nil?

  scraper_opts = opts.fetch(scraper.options_key, {}).except(:enabled)
  kwargs = construction_kwargs(
    scraper,
    request_session:,
    body:,
    document:,
    link_resolver:,
    captured_responses:
  )
  scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
end

.classify_no_scraper_surface(parsed_body, body: nil) ⇒ Symbol

Classifies why scrapers could not extract from a parsed page.

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw body for blocked-surface detection

Returns:



202
203
204
205
206
207
208
 
# File 'lib/html2rss/auto_source/scraper.rb', line 202

def self.classify_no_scraper_surface(parsed_body, body: nil)
  return :blocked_surface if blocked_surface?(parsed_body, body:)
  return :app_shell if app_shell_surface?(parsed_body)
  return :high_entropy_surface if high_entropy_surface?(parsed_body)

  :unsupported_surface
end

.extractable_instance?(instance, parsed_body) ⇒ Boolean

Parameters:

  • instance (Object)
  • parsed_body (Nokogiri::HTML::Document)

Returns:

  • (Boolean) 


168
169
170
171
172
 
# File 'lib/html2rss/auto_source/scraper.rb', line 168

def self.extractable_instance?(instance, parsed_body)
  return instance.extractable? if instance.respond_to?(:extractable?)

  instance.class.articles?(parsed_body)
end

.from(parsed_body, opts = ) ⇒ Array<Class>

Returns an array of scraper classes that claim to find articles in the parsed body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    The parsed HTML document.

  • opts (Hash) (defaults to: ) —

    The options hash.

Options Hash (opts):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Array<Class>) —

    An array of scraper classes that can handle the parsed body.



97
98
99
100
101
102
103
104
 
# File 'lib/html2rss/auto_source/scraper.rb', line 97

def self.from(parsed_body, opts = Html2rss::AutoSource::DEFAULT_CONFIG[:scraper])
  scrapers = SCRAPERS.select { |scraper| opts.dig(scraper.options_key, :enabled) }
  scrapers.select! { |scraper| scraper.articles?(parsed_body) }

  raise no_scraper_found_for(parsed_body) if scrapers.empty?

  scrapers
end

.heuristic_tier?(tier) ⇒ Boolean

Parameters:

  • tier (Array<Class>)

Returns:

  • (Boolean) 


109
110
111
 
# File 'lib/html2rss/auto_source/scraper.rb', line 109

def self.heuristic_tier?(tier)
  tier.intersect?(HEURISTIC_SCRAPERS)
end

.no_scraper_found_for(parsed_body, body: nil) ⇒ NoScraperFound

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw response body (preferred for blocked-surface checks)

Returns:



178
179
180
 
# File 'lib/html2rss/auto_source/scraper.rb', line 178

def self.no_scraper_found_for(parsed_body, body: nil)
  NoScraperFound.new(category: classify_no_scraper_surface(parsed_body, body:))
end

.normalize_sst(parsed_body) ⇒ SST::Document?

Parameters:

  • parsed_body (Nokogiri::HTML::Document)

Returns:



116
117
118
119
120
 
# File 'lib/html2rss/auto_source/scraper.rb', line 116

def self.normalize_sst(parsed_body)
  SST::Normalizer.call(parsed_body)
rescue ArgumentError
  nil
end