Class: Html2rss::AutoSource

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source.rb,
lib/html2rss/auto_source/cleanup.rb,
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/discovery.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/link_heuristics.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/discovery/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/discovery/dom_clustering.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/discovery/list_candidates.rb,
lib/html2rss/auto_source/discovery/semantic_containers.rb,
lib/html2rss/auto_source/link_heuristics/anchor_signals.rb,
lib/html2rss/auto_source/link_heuristics/href_extractor.rb,
lib/html2rss/auto_source/link_heuristics/path_classifier.rb,
lib/html2rss/auto_source/link_heuristics/text_classifier.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/link_heuristics/container_signals.rb,
lib/html2rss/auto_source/link_heuristics/destination_facts.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/link_heuristics/container_assessor.rb,
lib/html2rss/auto_source/discovery/semantic_anchor_candidates.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/discovery/dom_clustering/group_scorer.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/discovery/dom_clustering/overlap_resolver.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The AutoSource class automatically extracts articles from a given URL by utilizing a collection of Scrapers. These scrapers analyze and parse popular structured data formats—such as schema, microdata, and open graph—to identify and compile article elements into unified articles.

Scrapers supporting plain HTML are also available for sites without structured data, though results may vary based on page markup.

See Also:

Defined Under Namespace

Modules: Discovery, Scraper Classes: Cleanup, LinkHeuristics

Constant Summary collapse

DEFAULT_CONFIG =

Default auto-source configuration shipped for scraper and cleanup behavior.

{
  scraper: {
    wordpress_api: {
      enabled: true
    },
    sitemap: {
      enabled: true,
      min_priority: Discovery::Sitemap::DEFAULT_MIN_PRIORITY,
      max_age_days: Discovery::Sitemap::DEFAULT_MAX_AGE_DAYS
    },
    schema: {
      enabled: true
    },
    microdata: {
      enabled: true
    },
    microformats2: {
      enabled: true
    },
    json_state: {
      enabled: true
    },
    meta_oembed: {
      enabled: true
    },
    semantic_html: {
      enabled: true,
      fallback_anchorless: true
    },
    html: {
      enabled: true,
      minimum_selector_frequency: Scraper::Html::DEFAULT_MINIMUM_SELECTOR_FREQUENCY,
      use_top_selectors: Scraper::Html::DEFAULT_USE_TOP_SELECTORS,
      fallback_anchorless: true
    }
  },
  cleanup: Cleanup::DEFAULT_CONFIG
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(response, opts = DEFAULT_CONFIG, request_session: nil) ⇒ void

Parameters:

Options Hash (opts):

  • :scraper (Hash) —

    scraper configuration map

  • :cleanup (Hash) —

    cleanup configuration map



86
87
88
89
90
91
 
# File 'lib/html2rss/auto_source.rb', line 86

def initialize(response, opts = DEFAULT_CONFIG, request_session: nil)
  @parsed_body = response.parsed_body
  @url = response.url
  @opts = opts
  @request_session = request_session
end

Class Method Details

.request_slots_for(config) ⇒ Integer

Returns the sum of required request slots for all enabled scrapers in the config.

Parameters:

  • config (Hash, nil) —

    auto_source configuration hash

Returns:

  • (Integer) —

    total request slots required by scrapers



65
66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/html2rss/auto_source.rb', line 65

def request_slots_for(config)
  return 0 unless config

  Scraper::SCRAPERS.sum do |scraper|
    if config.dig(:scraper, scraper.options_key, :enabled)
      opts = config.dig(:scraper, scraper.options_key)
      scraper.respond_to?(:request_slots) ? scraper.request_slots(opts) : 0
    else
      0
    end
  end
end

Instance Method Details

#articles ⇒ Array<Html2rss::Article>

Extracts article candidates by selecting every scraper that can explain the page shape, running those scrapers, and normalizing the resulting hashes into Article objects.

The contributor-facing flow is:

  1. choose scraper instances that match the page
  2. let each scraper collect its own candidates
  3. clean and deduplicate the merged article list

Scrapers with expensive precomputation, such as SemanticHtml, keep that state on the instance so detection and extraction can reuse the same work.

Returns:



107
108
109
110
111
112
 
# File 'lib/html2rss/auto_source.rb', line 107

def articles
  @articles ||= extract_articles
rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
  Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
  []
end