Class: Html2rss::AutoSource::Scraper::Sitemap

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/sitemap.rb

Overview

Scrapes articles from XML sitemaps explicitly linked in HTML or discovered via standard sitemap paths.

Constant Summary collapse 

'link[rel="sitemap"][href]'
MAX_SUB_SITEMAPS =

Maximum number of sub-sitemaps to fetch when fanning out through a sitemapindex.

3

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body, url:, request_session: nil, **opts) ⇒ Sitemap

Returns a new instance of Sitemap.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

  • url (String, Html2rss::Url) —

    canonical page URL

  • request_session (Html2rss::RequestSession, nil) (defaults to: nil) —

    shared request session for follow-up fetches

  • _opts (Hash) —

    a customizable set of options



47
48
49
50
51
52
53
 
# File 'lib/html2rss/auto_source/scraper/sitemap.rb', line 47

def initialize(parsed_body, url:, request_session: nil, **opts)
  @parsed_body = parsed_body
  @url = Html2rss::Url.from_absolute(url)
  @request_session = request_session
  @min_priority = opts.fetch(:min_priority, Discovery::Sitemap::DEFAULT_MIN_PRIORITY)
  @max_age_days = opts.fetch(:max_age_days, Discovery::Sitemap::DEFAULT_MAX_AGE_DAYS)
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Returns whether the page links to a sitemap or is a sitemap.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) —

    parsed document

Returns:

  • (Boolean) —

    whether the page links to a sitemap or is a sitemap



34
35
36
37
38
39
 
# File 'lib/html2rss/auto_source/scraper/sitemap.rb', line 34

def self.articles?(parsed_body)
  return false unless parsed_body

  !parsed_body.at_css(SITEMAP_LINK_SELECTOR).nil? ||
    !parsed_body.at_xpath('//*[local-name()="urlset" or local-name()="sitemapindex"]').nil?
end

.options_key ⇒ Symbol

Returns scraper config key.

Returns:

  • (Symbol) —

    scraper config key



20
 
# File 'lib/html2rss/auto_source/scraper/sitemap.rb', line 20

def self.options_key = :sitemap

.request_slots(_opts = {}) ⇒ Integer

Returns the number of request slots needed for sitemap discovery and fan-out.

Parameters:

  • _opts (Hash) (defaults to: {}) —

    unused options

Returns:

  • (Integer) —

    number of follow-up requests needed



27
28
29
 
# File 'lib/html2rss/auto_source/scraper/sitemap.rb', line 27

def self.request_slots(_opts = {})
  1 + MAX_SUB_SITEMAPS
end

Instance Method Details

#each {|article| ... } ⇒ Enumerator, void

Yields article hashes from the sitemap.

Yield Parameters:

  • article (Hash{Symbol => Object}) —

    normalized article hash

Returns:

  • (Enumerator, void) —

    enumerator when no block is given



60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/html2rss/auto_source/scraper/sitemap.rb', line 60

def each
  return enum_for(:each) unless block_given?

  entries = fetch_and_parse_entries
  return if entries.empty?

  entries.each do |entry|
    article_hash = article_from_entry(entry)
    yield article_hash if article_hash
  end
end