Class: Html2rss::AutoSource

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source.rb,
lib/html2rss/auto_source/cleanup.rb,
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/segment.rb,
lib/html2rss/auto_source/segmenter.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/segmenter/list.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/segmenter/cluster.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/segmenter/semantic.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/segmenter/primary_link.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The AutoSource class automatically extracts articles from a given URL by utilizing a collection of Scrapers. These scrapers analyze and parse popular structured data formats—such as schema, microdata, and open graph—to identify and compile article elements into unified articles.

Scrapers supporting plain HTML are also available for sites without structured data, though results may vary based on page markup.

AutoSource

How html2rss builds feed items when a config has no (or incomplete) CSS selectors.

What and when

Html2rss.auto_source / auto_json_feed (and any feed config with auto_source:) fetch a page once, then run an ordered set of scrapers against that response. Use it when you want “guess articles from this URL” without hand-writing selectors. Prefer explicit selectors: when you already know the list markup — that path stays on Nokogiri + Html::Navigator.

Entry: FeedPipeline → AutoSource#articles → Scraper.build_instance → per-scraper extraction → Cleanup.

Live flow

  1. Request — RequestSession returns a Response with body (String) and parsed_body (Nokogiri HTML).

  2. Scraper tiers — Enabled scrapers that claim the page (shallow articles? or instance extractable?) run in Scraper::SCRAPER_TIERS order. Merge within a tier, then stop when enough articles survive Cleanup:

  3. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles

  4. Follow-up IO: WordPress API, Sitemap, MetaOembed

  5. Heuristic: SemanticHtml

  6. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)

  7. Structured / API scrapers — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST.

  8. Heuristic scrapers — SemanticHtml and Html normalize once into an SST::Document, then:

    SST::Normalizer → AutoSource::Segmenter → Scoring::Engine → extractor / article materialization.

  9. Cleanup — Merge, dedupe, hard-exclude non-article destinations (via PathClassifier facts), drop junk titles, and trim to limit. Html is skipped when earlier tiers already admitted clean items.

Segmenter strategies: :semantic (leaf containers + primary link), :list (repeated tag paths), :cluster (class/structure grids for anchorless cards). Scoring ranks and demotes; LinkDestination::NoisePolicy owns content-anchor eligibility. Cleanup owns feed-item admission.

Nokogiri vs SST boundaries

Surface Owns DOM
Response#parsed_body Single HTML parse for the page
Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection Nokogiri
Sitemap detection (CSS/XPath) Nokogiri; URL list parsing uses raw response.body (XML string)
Selectors path / Sanitize transformers Nokogiri (unchanged)
SST::Normalizer Sole Nokogiri consumer on the heuristic auto-source path
Segmenter, Scoring, Html::SstArticleExtractor, heuristic chrome (SST::Tags / SST::Text) SST only

Production heuristic scrapers should reuse one SST::Document memoized from parsed_body (or a shared Document passed in), not re-parse HTML strings.

Constraints

  • SST::Normalizer::MAX_NODES (5_000) — Beyond this, normalization degrades to a semantic-tag-only keep set and logs a warning.
  • Top-K — Scoring::Engine::TOP_K (99) caps ranked segments materialized into articles; list strategy also budgets use_top_selectors.
  • Typed stages — Pipeline stages take SST::Document / Segment / RankedSegment, not ad-hoc Hash bags. Internal scraper APIs may change; the public gem surface is lib/html2rss.rb.

Non-goals

  • Replacing Selectors, Schema, Microdata, MF2, JsonState, or XhrArticles with SST.
  • A second HTML parser beside Nokogiri.
  • Dual Response payload (Nokogiri + SST always).
  • App-shell classification on SST.
  • Rewriting Sanitize transformers off Nokogiri.

See also CONTEXT.md for module ownership (chrome, scoring, clustering), FeedPipeline for the request-strategy fallback chain (unrelated to article scraping), and Capture to turn a listing URL into a durable items-selector config.

See Also:

Defined Under Namespace

Modules: Scraper Classes: Cleanup, Segment, Segmenter

Constant Summary collapse

DEFAULT_LIMIT =

Default max articles to keep (also the short-circuit floor across scraper tiers).

25
DEFAULT_CONFIG =

Default auto-source configuration shipped for scraper and cleanup behavior.

{
  limit: DEFAULT_LIMIT,
  scraper: {
    wordpress_api: {
      enabled: true
    },
    sitemap: {
      enabled: true,
      min_priority: Scraper::Sitemap::Parser::DEFAULT_MIN_PRIORITY,
      max_age_days: Scraper::Sitemap::Parser::DEFAULT_MAX_AGE_DAYS
    },
    schema: {
      enabled: true
    },
    microdata: {
      enabled: true
    },
    microformats2: {
      enabled: true
    },
    json_state: {
      enabled: true
    },
    xhr_articles: {
      enabled: true
    },
    meta_oembed: {
      enabled: true
    },
    semantic_html: {
      enabled: true,
      fallback_anchorless: true
    },
    html: {
      enabled: true,
      minimum_selector_frequency: Scraper::Html::DEFAULT_MINIMUM_SELECTOR_FREQUENCY,
      use_top_selectors: Scraper::Html::DEFAULT_USE_TOP_SELECTORS,
      fallback_anchorless: true
    }
  },
  cleanup: Cleanup::DEFAULT_CONFIG
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(response, opts = DEFAULT_CONFIG, request_session: nil) ⇒ void

Parameters:

Options Hash (opts):

  • :limit (Integer) —

    max articles to keep; later tiers stop once this many survive Cleanup

  • :scraper (Hash) —

    scraper configuration map

  • :cleanup (Hash) —

    cleanup configuration map



95
96
97
98
99
100
101
102
 
# File 'lib/html2rss/auto_source.rb', line 95

def initialize(response, opts = DEFAULT_CONFIG, request_session: nil)
  @parsed_body = response.parsed_body
  @body = response.body
  @url = response.url
  @captured_responses = response.captured_responses
  @opts = opts
  @request_session = request_session
end

Class Method Details

.request_slots_for(config) ⇒ Integer

Returns the sum of required request slots for all enabled scrapers in the config.

Parameters:

  • config (Hash, nil) —

    auto_source configuration hash

Returns:

  • (Integer) —

    total request slots required by scrapers



73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/html2rss/auto_source.rb', line 73

def request_slots_for(config)
  return 0 unless config

  Scraper::SCRAPERS.sum do |scraper|
    if config.dig(:scraper, scraper.options_key, :enabled)
      opts = config.dig(:scraper, scraper.options_key)
      scraper.respond_to?(:request_slots) ? scraper.request_slots(opts) : 0
    else
      0
    end
  end
end

Instance Method Details

#admission_drops ⇒ Hash{String => Integer}

Reason → count tallies from the final Cleanup pass (empty until #articles runs).

Returns:

  • (Hash{String => Integer}) 


126
127
128
 
# File 'lib/html2rss/auto_source.rb', line 126

def admission_drops
  @admission_drops || {}
end

#articles ⇒ Array<Html2rss::Article>

Extracts articles by running scraper tiers until a sufficient set is found.

Tiers: in-page structured → follow-up IO → SemanticHtml → Html. SST is built only when a heuristic tier runs. Later tiers are skipped once limit articles with url+title remain after Cleanup; the result is capped to limit. Html is skipped when earlier tiers already admitted at least one clean article below limit — quality over padding with weaker heuristic junk.

Returns:



114
115
116
117
118
119
120
 
# File 'lib/html2rss/auto_source.rb', line 114

def articles
  @articles ||= extract_articles
rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
  Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
  @admission_drops = {}.freeze
  []
end