Class: Html2rss::AutoSource

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source.rb,
lib/html2rss/auto_source/cleanup.rb,
lib/html2rss/auto_source/scraper.rb,
lib/html2rss/auto_source/segment.rb,
lib/html2rss/auto_source/segmenter.rb,
lib/html2rss/auto_source/scraper/html.rb,
lib/html2rss/auto_source/scraper/schema.rb,
lib/html2rss/auto_source/segmenter/list.rb,
lib/html2rss/auto_source/scraper/sitemap.rb,
lib/html2rss/auto_source/scraper/microdata.rb,
lib/html2rss/auto_source/segmenter/cluster.rb,
lib/html2rss/auto_source/scraper/json_state.rb,
lib/html2rss/auto_source/segmenter/semantic.rb,
lib/html2rss/auto_source/scraper/meta_oembed.rb,
lib/html2rss/auto_source/scraper/native_feed.rb,
lib/html2rss/auto_source/scraper/schema/thing.rb,
lib/html2rss/auto_source/scraper/xhr_articles.rb,
lib/html2rss/auto_source/scraper/microformats2.rb,
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/wordpress_api.rb,
lib/html2rss/auto_source/scraper/sitemap/parser.rb,
lib/html2rss/auto_source/segmenter/primary_link.rb,
lib/html2rss/auto_source/scraper/schema/item_list.rb,
lib/html2rss/auto_source/scraper/json_state/value_finder.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope.rb,
lib/html2rss/auto_source/scraper/schema/category_extractor.rb,
lib/html2rss/auto_source/scraper/json_state/document_scanner.rb,
lib/html2rss/auto_source/scraper/wordpress_api/posts_endpoint.rb,
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb,
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb,
lib/html2rss/auto_source/scraper/wordpress_api/page_scope/date_archive_range.rb

Overview

The AutoSource class automatically extracts articles from a given URL by utilizing a collection of Scrapers. These scrapers analyze and parse popular structured data formats—such as schema, microdata, and open graph—to identify and compile article elements into unified articles.

Scrapers supporting plain HTML are also available for sites without structured data, though results may vary based on page markup.

AutoSource

How html2rss builds feed items when a config has no (or incomplete) CSS selectors.

What and when

Html2rss.auto_source / auto_json_feed (and any feed config with auto_source:) fetch a page once, then run an ordered set of scrapers against that response. Use it when you want “guess articles from this URL” without hand-writing selectors. Prefer explicit selectors: when you already know the list markup — that path stays on Nokogiri + Html::Navigator.

Entry: FeedPipeline → AutoSource#articles → Scraper.build_instance → per-scraper extraction → Cleanup.

Live flow

  1. Request — RequestSession returns a Response with body (String) and parsed_body (Nokogiri HTML). Direct syndication responses skip HTML scrapers and parse via Syndication::Parser.

  2. Scraper tiers — Enabled scrapers that claim the page (shallow articles? or instance extractable?) run in Scraper::SCRAPER_TIERS order. Merge within a tier, then stop when enough articles survive Cleanup:

  3. Native feed: NativeFeed (head alternates + path discovery → RSS/Atom parse)

  4. In-page structured: Schema, Microdata, Microformats2, JsonState, XhrArticles

  5. Follow-up IO: WordPress API, Sitemap, MetaOembed

  6. Heuristic: SemanticHtml

  7. Heuristic: Html (skipped when earlier tiers already admitted at least one clean article)

  8. Structured / API scrapers — Schema, Microdata, MF2, JSON state, XHR JSON, oEmbed, WordPress REST, and Sitemap work on Nokogiri CSS/XPath or JSON/XML parsers. They do not use SST.

  9. Heuristic scrapers — SemanticHtml and Html normalize once into an SST::Document, then:

    SST::Normalizer → AutoSource::Segmenter → Scoring::Engine → extractor / article materialization.

  10. Cleanup — Merge, dedupe, hard-exclude non-article destinations (via PathClassifier facts), drop junk titles, and trim to limit. Html is skipped when earlier tiers already admitted clean items.

  11. Entry URL resolution (pipeline, not this class) — when AutoFallback sees a weak homepage extract, FeedResolution may rewrite the scrape URL to a listing/feed before escalating strategies. See FeedPipeline and FeedResolution.

Segmenter strategies: :semantic (leaf containers + primary link), :list (repeated tag paths), :cluster (class/structure grids for anchorless cards). Scoring ranks and demotes; LinkDestination::NoisePolicy owns content-anchor eligibility. Cleanup owns feed-item admission.

Nokogiri vs SST boundaries

Surface Owns DOM
Response#parsed_body Single HTML parse for the page
Schema / Microdata / MF2 / JsonState / XhrArticles / MetaOembed / app-shell detection Nokogiri
Sitemap detection (CSS/XPath) Nokogiri; URL list parsing uses raw response.body (XML string)
Selectors path / Sanitize transformers Nokogiri (unchanged)
SST::Normalizer Sole Nokogiri consumer on the heuristic auto-source path
Segmenter, Scoring, Html::SstArticleExtractor, heuristic chrome (SST::Tags / SST::Text) SST only

Production heuristic scrapers should reuse one SST::Document memoized from parsed_body (or a shared Document passed in), not re-parse HTML strings.

Constraints

  • SST::Normalizer::MAX_NODES (5_000) — Beyond this, normalization degrades to a semantic-tag-only keep set and logs a warning.
  • Top-K — Scoring::Engine::TOP_K (99) caps ranked segments materialized into articles; list strategy also budgets use_top_selectors.
  • Typed stages — Pipeline stages take SST::Document / Segment / RankedSegment, not ad-hoc Hash bags. Internal scraper APIs may change; the public gem surface is lib/html2rss.rb.

Non-goals

  • Replacing Selectors, Schema, Microdata, MF2, JsonState, or XhrArticles with SST.
  • A second HTML parser beside Nokogiri.
  • Dual Response payload (Nokogiri + SST always).
  • App-shell classification on SST.
  • Rewriting Sanitize transformers off Nokogiri.

See also CONTEXT.md for module ownership (chrome, scoring, clustering), FeedPipeline for the request-strategy fallback chain (unrelated to article scraping), and Capture to turn a listing URL into a durable items-selector config.

See Also:

Defined Under Namespace

Modules: Scraper Classes: Cleanup, Segment, Segmenter

Constant Summary collapse

DEFAULT_LIMIT =

Default max articles to keep (also the short-circuit floor across scraper tiers).

25
DEFAULT_CONFIG =

Default auto-source configuration shipped for scraper and cleanup behavior.

{
  limit: DEFAULT_LIMIT,
  entry_resolution: {
    enabled: true,
    max_probes: 5
  }.freeze,
  scraper: {
    native_feed: {
      enabled: true
    },
    wordpress_api: {
      enabled: true
    },
    sitemap: {
      enabled: true,
      min_priority: Scraper::Sitemap::Parser::DEFAULT_MIN_PRIORITY,
      max_age_days: Scraper::Sitemap::Parser::DEFAULT_MAX_AGE_DAYS
    },
    schema: {
      enabled: true
    },
    microdata: {
      enabled: true
    },
    microformats2: {
      enabled: true
    },
    json_state: {
      enabled: true
    },
    xhr_articles: {
      enabled: true
    },
    meta_oembed: {
      enabled: true
    },
    semantic_html: {
      enabled: true,
      fallback_anchorless: true
    },
    html: {
      enabled: true,
      minimum_selector_frequency: Scraper::Html::DEFAULT_MINIMUM_SELECTOR_FREQUENCY,
      use_top_selectors: Scraper::Html::DEFAULT_USE_TOP_SELECTORS,
      fallback_anchorless: true
    }
  },
  cleanup: Cleanup::DEFAULT_CONFIG
}.freeze

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(response, opts = DEFAULT_CONFIG, request_session: nil) ⇒ void

Parameters:

Options Hash (opts):

  • :limit (Integer) —

    max articles to keep; later tiers stop once this many survive Cleanup

  • :scraper (Hash) —

    scraper configuration map

  • :cleanup (Hash) —

    cleanup configuration map



102
103
104
105
106
107
108
109
 
# File 'lib/html2rss/auto_source.rb', line 102

def initialize(response, opts = DEFAULT_CONFIG, request_session: nil)
  @parsed_body = response.parsed_body
  @body = response.body
  @url = response.url
  @captured_responses = response.captured_responses
  @opts = opts
  @request_session = request_session
end

Class Method Details

.request_slots_for(config) ⇒ Integer

Returns the sum of required request slots for all enabled scrapers in the config.

Parameters:

  • config (Hash, nil) —

    auto_source configuration hash

Returns:

  • (Integer) —

    total request slots required by scrapers



80
81
82
83
84
85
86
87
88
89
90
91
 
# File 'lib/html2rss/auto_source.rb', line 80

def request_slots_for(config)
  return 0 unless config

  Scraper::SCRAPERS.sum do |scraper|
    if config.dig(:scraper, scraper.options_key, :enabled)
      opts = config.dig(:scraper, scraper.options_key)
      scraper.respond_to?(:request_slots) ? scraper.request_slots(opts) : 0
    else
      0
    end
  end
end

Instance Method Details

#admission_drops ⇒ Hash{String => Integer}

Reason → count tallies from the final Cleanup pass (empty until #articles runs).

Returns:

  • (Hash{String => Integer}) 


133
134
135
 
# File 'lib/html2rss/auto_source.rb', line 133

def admission_drops
  @admission_drops || {}
end

#articles ⇒ Array<Html2rss::Article>

Extracts articles by running scraper tiers until a sufficient set is found.

Tiers: in-page structured → follow-up IO → SemanticHtml → Html. SST is built only when a heuristic tier runs. Later tiers are skipped once limit articles with url+title remain after Cleanup; the result is capped to limit. Html is skipped when earlier tiers already admitted at least one clean article below limit — quality over padding with weaker heuristic junk.

Returns:



121
122
123
124
125
126
127
 
# File 'lib/html2rss/auto_source.rb', line 121

def articles
  @articles ||= extract_articles
rescue Html2rss::AutoSource::Scraper::NoScraperFound => error
  Log.warn "#{self.class}: no scraper matched #{url} (#{error.message})"
  @admission_drops = {}.freeze
  []
end