Class: Html2rss::AutoSource

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source.rb,
lib/html2rss/auto_source/cleanup.rb,
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/segment.rb,
lib/html2rss/auto_source/segmenter.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/segmenter/list.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/segmenter/cluster.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/segmenter/semantic.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/segmenter/primary_link.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The AutoSource class automatically extracts articles from a given URL by utilizing a collection of Scrapers. These scrapers analyze and parse popular structured data formats—such as schema, microdata, and open graph—to identify and compile article elements into unified articles.

Scrapers supporting plain HTML are also available for sites without structured data, though results may vary based on page markup.

rubocop:disable Metrics/ClassLength -- defaults + tiered extract stay on the contributor entry type

See Also:

Defined Under Namespace

Modules: Scraper Classes: Cleanup, Segment, Segmenter

Constant Summary collapse

DEFAULT_LIMIT =

Default max articles to keep (also the short-circuit floor across scraper tiers).

25
DEFAULT_CONFIG =

Default auto-source configuration shipped for scraper and cleanup behavior.

{
  limit: DEFAULT_LIMIT,
  scraper: {
    wordpress_api: {
      enabled: true
    },
    sitemap: {
      enabled: true,
      min_priority: Scraper::Sitemap::Parser::DEFAULT_MIN_PRIORITY,
      max_age_days: Scraper::Sitemap::Parser::DEFAULT_MAX_AGE_DAYS
    },
    schema: {
      enabled: true
    },
    microdata: {
      enabled: true
    },
    microformats2: {
      enabled: true
    },
    json_state: {
      enabled: true
    },
    meta_oembed: {
      enabled: true
    },
    semantic_html: {
      enabled: true,
      fallback_anchorless: true
    },
    html: {
      enabled: true,
      minimum_selector_frequency: Scraper::Html::DEFAULT_MINIMUM_SELECTOR_FREQUENCY,
      use_top_selectors: Scraper::Html::DEFAULT_USE_TOP_SELECTORS,
      fallback_anchorless: true
    }
  },
  cleanup: Cleanup::DEFAULT_CONFIG
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(response, opts = DEFAULT_CONFIG, request_session: nil) ⇒ void

Parameters:

Options Hash (opts):

  • :limit (Integer) —

    max articles to keep; later tiers stop once this many survive Cleanup

  • :scraper (Hash) —

    scraper configuration map

  • :cleanup (Hash) —

    cleanup configuration map



92
93
94
95
96
97
98
 
# File 'lib/html2rss/auto_source.rb', line 92

def initialize(response, opts = DEFAULT_CONFIG, request_session: nil)
  @parsed_body = response.parsed_body
  @body = response.body
  @url = response.url
  @opts = opts
  @request_session = request_session
end

Class Method Details

.request_slots_for(config) ⇒ Integer

Returns the sum of required request slots for all enabled scrapers in the config.

Parameters:

  • config (Hash, nil) —

    auto_source configuration hash

Returns:

  • (Integer) —

    total request slots required by scrapers



70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/html2rss/auto_source.rb', line 70

def request_slots_for(config)
  return 0 unless config

  Scraper::SCRAPERS.sum do |scraper|
    if config.dig(:scraper, scraper.options_key, :enabled)
      opts = config.dig(:scraper, scraper.options_key)
      scraper.respond_to?(:request_slots) ? scraper.request_slots(opts) : 0
    else
      0
    end
  end
end

Instance Method Details

#articles ⇒ Array<Html2rss::Article>

Extracts articles by running scraper tiers until a sufficient set is found.

Tiers: in-page structured → follow-up IO → SemanticHtml → Html. SST is built only when a heuristic tier runs. Later tiers are skipped once limit articles with url+title remain after Cleanup; the result is capped to limit.

Returns:



108
109
110
111
112
113
 
# File 'lib/html2rss/auto_source.rb', line 108

def articles
  @articles ||= extract_articles
rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
  Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
  []
end