Module: Html2rss::AutoSource::Scraper

Defined in:
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/native_feed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The Scraper module contains all scrapers that can be used to extract articles. Each scraper should implement an each method that yields article hashes. Each scraper should also implement an articles? method that returns true if the scraper can potentially be used to extract articles from the given HTML.

Detection is intentionally shallow for most scrapers, but instance-based matching is available for scrapers that need to carry expensive selection state forward into extraction.

Defined Under Namespace

Classes: Html, JsonState, MetaOembed, Microdata, Microformats2, NativeFeed, NoScraperFound, Schema, SemanticHtml, Sitemap, WordpressApi, XhrArticles

Constant Summary collapse

APP_SHELL_ROOT_SELECTORS =

Root markers indicating likely app-shell/client-rendered surfaces.

'#app, #root, #__next, [data-reactroot], [ng-app], [id*="app-shell"]'
APP_SHELL_MAX_ANCHORS =

Maximum anchors tolerated before app-shell detection is considered unlikely.

2
HIGH_ENTROPY_MIN_ANCHORS =

Minimum same-page anchors suggesting a high-entropy homepage/hub surface.

20
APP_SHELL_MAX_VISIBLE_TEXT_LENGTH =

Maximum visible text length tolerated for app-shell classification.

220
SCRAPER_TIERS =

Extraction tiers: merge within a tier, then stop when AutoSource has enough articles. Heuristic scrapers are separate tiers so SemanticHtml can satisfy before Html runs. NativeFeed is tier 0 so syndication wins before structured HTML scrapers.

[
  [NativeFeed].freeze,
  [Schema, Microdata, Microformats2, JsonState, XhrArticles].freeze,
  [WordpressApi, Sitemap, MetaOembed].freeze,
  [SemanticHtml].freeze,
  [Html].freeze
].freeze
SCRAPERS =

Flat ordered list (request slot accounting, detection helpers).

SCRAPER_TIERS.flatten.freeze
HEURISTIC_SCRAPERS =

Heuristic scrapers that share one memoized SST::Document per page.

[SemanticHtml, Html].freeze
REQUEST_SESSION_SCRAPERS =

Scrapers that accept a shared follow-up request_session.

[NativeFeed, WordpressApi, Sitemap, MetaOembed].freeze
CAPTURED_RESPONSE_SCRAPERS =

Scrapers that consume browser-captured XHR/fetch JSON bodies.

[XhrArticles].freeze

Class Method Summary collapse

Class Method Details

.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil, link_resolver: nil, captured_responses: []) ⇒ Object?

Builds a scraper when enabled; returns nil when disabled.

rubocop:disable Metrics/ParameterLists, Metrics/MethodLength -- construction context for structured and heuristic scrapers

Parameters:

  • scraper (Class)
  • parsed_body (Nokogiri::HTML::Document)
  • opts (Hash) —

    full scraper options map

  • url (Html2rss::Url, String)
  • request_session (Html2rss::RequestSession, nil) (defaults to: nil)
  • body (String, nil) (defaults to: nil)
  • document (SST::Document, nil) (defaults to: nil)
  • link_resolver (Scoring::LinkResolver, nil) (defaults to: nil)
  • captured_responses (Array<Hash>) (defaults to: []) —

    browser-captured JSON XHR/fetch bodies

Options Hash (opts:):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Object, nil) 


149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
 
# File 'lib/html2rss/auto_source/scraper.rb', line 149

def self.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil,
                        link_resolver: nil, captured_responses: [])
  return unless opts.dig(scraper.options_key, :enabled)
  return if HEURISTIC_SCRAPERS.include?(scraper) && document.nil?

  scraper_opts = opts.fetch(scraper.options_key, {}).except(:enabled)
  kwargs = construction_kwargs(
    scraper,
    request_session:,
    body:,
    document:,
    link_resolver:,
    captured_responses:
  )
  scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
end

.classify_no_scraper_surface(parsed_body, body: nil) ⇒ Symbol

Classifies why scrapers could not extract from a parsed page.

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw body for blocked-surface detection

Returns:



205
206
207
208
209
210
211
 
# File 'lib/html2rss/auto_source/scraper.rb', line 205

def self.classify_no_scraper_surface(parsed_body, body: nil)
  return :blocked_surface if blocked_surface?(parsed_body, body:)
  return :app_shell if app_shell_surface?(parsed_body)
  return :high_entropy_surface if high_entropy_surface?(parsed_body)

  :unsupported_surface
end

.extractable_instance?(instance, parsed_body) ⇒ Boolean

Parameters:

  • instance (Object)
  • parsed_body (Nokogiri::HTML::Document)

Returns:

  • (Boolean) 


171
172
173
174
175
 
# File 'lib/html2rss/auto_source/scraper.rb', line 171

def self.extractable_instance?(instance, parsed_body)
  return instance.extractable? if instance.respond_to?(:extractable?)

  instance.class.articles?(parsed_body)
end

.from(parsed_body, opts = ) ⇒ Array<Class>

Returns an array of scraper classes that claim to find articles in the parsed body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    The parsed HTML document.

  • opts (Hash) (defaults to: ) —

    The options hash.

Options Hash (opts):

  • :native_feed (Hash) —

    scraper toggle and configuration

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :xhr_articles (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Array<Class>) —

    An array of scraper classes that can handle the parsed body.



100
101
102
103
104
105
106
107
 
# File 'lib/html2rss/auto_source/scraper.rb', line 100

def self.from(parsed_body, opts = Html2rss::AutoSource::DEFAULT_CONFIG[:scraper])
  scrapers = SCRAPERS.select { |scraper| opts.dig(scraper.options_key, :enabled) }
  scrapers.select! { |scraper| scraper.articles?(parsed_body) }

  raise no_scraper_found_for(parsed_body) if scrapers.empty?

  scrapers
end

.heuristic_tier?(tier) ⇒ Boolean

Parameters:

  • tier (Array<Class>)

Returns:

  • (Boolean) 


112
113
114
 
# File 'lib/html2rss/auto_source/scraper.rb', line 112

def self.heuristic_tier?(tier)
  tier.intersect?(HEURISTIC_SCRAPERS)
end

.no_scraper_found_for(parsed_body, body: nil) ⇒ NoScraperFound

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw response body (preferred for blocked-surface checks)

Returns:



181
182
183
 
# File 'lib/html2rss/auto_source/scraper.rb', line 181

def self.no_scraper_found_for(parsed_body, body: nil)
  NoScraperFound.new(category: classify_no_scraper_surface(parsed_body, body:))
end

.normalize_sst(parsed_body) ⇒ SST::Document?

Parameters:

  • parsed_body (Nokogiri::HTML::Document)

Returns:



119
120
121
122
123
 
# File 'lib/html2rss/auto_source/scraper.rb', line 119

def self.normalize_sst(parsed_body)
  SST::Normalizer.call(parsed_body)
rescue ArgumentError
  nil
end