Module: Html2rss::AutoSource::Scraper

Defined in:
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The Scraper module contains all scrapers that can be used to extract articles. Each scraper should implement an each method that yields article hashes. Each scraper should also implement an articles? method that returns true if the scraper can potentially be used to extract articles from the given HTML.

Detection is intentionally shallow for most scrapers, but instance-based matching is available for scrapers that need to carry expensive selection state forward into extraction.

Defined Under Namespace

Classes: Html, JsonState, MetaOembed, Microdata, Microformats2, NoScraperFound, Schema, SemanticHtml, Sitemap, WordpressApi

Constant Summary collapse

APP_SHELL_ROOT_SELECTORS =

Root markers indicating likely app-shell/client-rendered surfaces.

'#app, #root, #__next, [data-reactroot], [ng-app], [id*="app-shell"]'
APP_SHELL_MAX_ANCHORS =

Maximum anchors tolerated before app-shell detection is considered unlikely.

2
APP_SHELL_MAX_VISIBLE_TEXT_LENGTH =

Maximum visible text length tolerated for app-shell classification.

220
SCRAPER_TIERS =

Extraction tiers: merge within a tier, then stop when AutoSource has enough articles. Heuristic scrapers are separate tiers so SemanticHtml can satisfy before Html runs.

[
  [Schema, Microdata, Microformats2, JsonState].freeze,
  [WordpressApi, Sitemap, MetaOembed].freeze,
  [SemanticHtml].freeze,
  [Html].freeze
].freeze
SCRAPERS =

Flat ordered list (request slot accounting, detection helpers).

SCRAPER_TIERS.flatten.freeze
HEURISTIC_SCRAPERS =

Heuristic scrapers that share one memoized SST::Document per page.

[SemanticHtml, Html].freeze
REQUEST_SESSION_SCRAPERS =

Scrapers that accept a shared follow-up request_session.

[WordpressApi, Sitemap, MetaOembed].freeze

Class Method Summary collapse

Class Method Details

.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil, link_resolver: nil) ⇒ Object?

Builds a scraper when enabled; returns nil when disabled.

rubocop:disable Metrics/ParameterLists -- construction context for structured and heuristic scrapers

Parameters:

Options Hash (opts:):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Object, nil) 


136
137
138
139
140
141
142
143
144
 
# File 'lib/html2rss/auto_source/scraper.rb', line 136

def self.build_instance(scraper, parsed_body, opts:, url:, request_session: nil, body: nil, document: nil,
                        link_resolver: nil)
  return unless opts.dig(scraper.options_key, :enabled)
  return if HEURISTIC_SCRAPERS.include?(scraper) && document.nil?

  scraper_opts = opts.fetch(scraper.options_key, {}).except(:enabled)
  kwargs = construction_kwargs(scraper, request_session:, body:, document:, link_resolver:)
  scraper.new(parsed_body, url:, **kwargs, **scraper_opts)
end

.extractable_instance?(instance, parsed_body) ⇒ Boolean

Parameters:

  • instance (Object)
  • parsed_body (Nokogiri::HTML::Document)

Returns:

  • (Boolean) 


151
152
153
154
155
 
# File 'lib/html2rss/auto_source/scraper.rb', line 151

def self.extractable_instance?(instance, parsed_body)
  return instance.extractable? if instance.respond_to?(:extractable?)

  instance.class.articles?(parsed_body)
end

.from(parsed_body, opts = ) ⇒ Array<Class>

Returns an array of scraper classes that claim to find articles in the parsed body.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    The parsed HTML document.

  • opts (Hash) (defaults to: ) —

    The options hash.

Options Hash (opts):

  • :wordpress_api (Hash) —

    scraper toggle and configuration

  • :schema (Hash) —

    scraper toggle and configuration

  • :microdata (Hash) —

    scraper toggle and configuration

  • :microformats2 (Hash) —

    scraper toggle and configuration

  • :json_state (Hash) —

    scraper toggle and configuration

  • :meta_oembed (Hash) —

    scraper toggle and configuration

  • :semantic_html (Hash) —

    scraper toggle and configuration

  • :html (Hash) —

    scraper toggle and configuration

  • :sitemap (Hash) —

    scraper toggle and configuration

Returns:

  • (Array<Class>) —

    An array of scraper classes that can handle the parsed body.



89
90
91
92
93
94
95
96
 
# File 'lib/html2rss/auto_source/scraper.rb', line 89

def self.from(parsed_body, opts = Html2rss::AutoSource::DEFAULT_CONFIG[:scraper])
  scrapers = SCRAPERS.select { |scraper| opts.dig(scraper.options_key, :enabled) }
  scrapers.select! { |scraper| scraper.articles?(parsed_body) }

  raise no_scraper_found_for(parsed_body) if scrapers.empty?

  scrapers
end

.heuristic_tier?(tier) ⇒ Boolean

Parameters:

  • tier (Array<Class>)

Returns:

  • (Boolean) 


101
102
103
 
# File 'lib/html2rss/auto_source/scraper.rb', line 101

def self.heuristic_tier?(tier)
  tier.intersect?(HEURISTIC_SCRAPERS)
end

.no_scraper_found_for(parsed_body, body: nil) ⇒ NoScraperFound

Parameters:

  • parsed_body (Nokogiri::HTML::Document)
  • body (String, nil) (defaults to: nil) —

    raw response body (preferred for blocked-surface checks)

Returns:



161
162
163
 
# File 'lib/html2rss/auto_source/scraper.rb', line 161

def self.no_scraper_found_for(parsed_body, body: nil)
  NoScraperFound.new(category: classify_no_scraper_surface(parsed_body, body:))
end

.normalize_sst(parsed_body) ⇒ SST::Document?

Parameters:

  • parsed_body (Nokogiri::HTML::Document)

Returns:



108
109
110
111
112
 
# File 'lib/html2rss/auto_source/scraper.rb', line 108

def self.normalize_sst(parsed_body)
  SST::Normalizer.call(parsed_body)
rescue ArgumentError
  nil
end