Class: Html2rss::FeedPipeline

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/feed_pipeline.rb,
lib/html2rss/feed_pipeline/auto_fallback.rb,
lib/html2rss/feed_pipeline/strategy_plan.rb,
lib/html2rss/feed_pipeline/runtime_policy.rb

Overview

Builds feeds from validated config through request, extraction, and rendering stages.

FeedPipeline — auto request strategy

:auto is the default request plan for feed builds (auto_source, auto_json_feed, Capture, and MCP scrape_url / capture_config). FeedPipeline::StrategyPlan resolves it; FeedPipeline::AutoFallback executes AutoFallback::CHAIN.

Use :auto when you want Faraday first and a browser-backed hop only if that fetch fails or yields zero items. Pin a concrete strategy (faraday, botasaurus, local_file) when you need a single transport.

Chain

AutoFallback::CHAIN is:

  1. Faraday — plain HTTP (faster, cheaper).
  2. Botasaurus — attempted when Faraday raises a fallback-eligible error (for example BlockedSurfaceDetected or timeout) or extracts zero feed items.

There is no Browserless / Puppeteer-in-gem tier. Pin botasaurus when you want browser rendering without Faraday first. Botasaurus needs BOTASAURUS_SCRAPER_URL.

Surfaces

Surface :auto behavior
Gem / CLI feed build, MCP scrape_url, Capture Full AutoFallback chain (faraday → botasaurus)
MCP inspect_url Cheap diagnostic: StrategyPlan.concrete_for_diagnostic maps auto to Faraday (pin botasaurus when you need browser rendering)

Fallback vs abort

These typically hop to the next chain member (among other StandardErrors AutoFallback rescues):

  • Html2rss::RequestService::BlockedSurfaceDetected
  • Html2rss::RequestService::RequestTimedOut
  • Faraday connection / timeout errors
  • Empty extraction results when a later chain member may succeed

These abort immediately (AutoFallback::NON_FALLBACK_ERRORS): unknown strategy, invalid URL, unsupported scheme, budget exceeded, private network denied, cross-origin follow-up denied, response too large.

Retries share the feed's request/session policy. Pin a concrete strategy when you need a single-hop budget profile.

Success signals

  • Html2rss::RequestService::Response includes transport metadata for the strategy that produced it.
  • Html2rss::Status records selected strategy and attempt tallies for MCP envelope payload / CLI --explain.
  • Fallback hops log at info/warn (AutoFallback).

See also AutoSource for article scraping (a different pipeline) and Capture for durable configs that stamp the selected strategy.

Defined Under Namespace

Classes: AutoFallback, PipelineOutcome, RuntimePolicy, StrategyPlan

Instance Method Summary collapse

Constructor Details

#initialize(raw_config) ⇒ FeedPipeline

Returns a new instance of FeedPipeline.

Parameters:

  • raw_config (Hash{Symbol => Object}) —

    user-provided feed config



20
21
22
 
# File 'lib/html2rss/feed_pipeline.rb', line 20

def initialize(raw_config)
  @raw_config = raw_config
end

Instance Method Details

#deduplicated_articles(config:, response:, request_session:) ⇒ Array(Array<Html2rss::Article>, Integer, Hash)

Returns unique articles, dedup drops, admission drops.

Parameters:

Returns:

  • (Array(Array<Html2rss::Article>, Integer, Hash)) —

    unique articles, dedup drops, admission drops



74
75
76
77
78
 
# File 'lib/html2rss/feed_pipeline.rb', line 74

def deduplicated_articles(config:, response:, request_session:)
  collected, admission_drops = collect_articles(config:, response:, request_session:)
  unique = Article::Deduplicator.new(collected).call
  [unique, collected.size - unique.size, admission_drops]
end

#request_session_for(config, strategy:, resources:) ⇒ Html2rss::RequestSession

Parameters:

Returns:



60
61
62
63
64
65
66
67
 
# File 'lib/html2rss/feed_pipeline.rb', line 60

def request_session_for(config, strategy:, resources:)
  RequestSession.build(
    config:,
    strategy:,
    budget: resources.budget,
    policy: resources.policy
  )
end

#to_outcome ⇒ PipelineOutcome

Runs the pipeline once and returns scrape-finished outcome (before Channel/Status). Used by Capture which needs the response body for SST selector derivation.

Returns:



29
30
31
32
 
# File 'lib/html2rss/feed_pipeline.rb', line 29

def to_outcome
  config = Config.from_hash(raw_config, params: raw_config[:params])
  pipeline_outcome_for(config)
end

#to_result ⇒ Html2rss::FeedResult

Runs the pipeline once and returns an opaque, Marshal-cacheable result.

rubocop:disable Metrics/AbcSize, Metrics/MethodLength -- Status kwargs stay co-located with Channel

Returns:



39
40
41
42
43
44
45
46
47
48
49
50
51
52
 
# File 'lib/html2rss/feed_pipeline.rb', line 39

def to_result
  config = Config.from_hash(raw_config, params: raw_config[:params])
  outcome = pipeline_outcome_for(config)
  channel = Channel.from_response(outcome.response, overrides: config.channel)
  status = Status.build(
    articles: outcome.articles,
    dedup_dropped: outcome.dedup_dropped,
    selected_strategy: outcome.selected_strategy,
    attempt_count: outcome.attempt_count,
    strategy_attempts: outcome.strategy_attempts,
    admission_drops: outcome.admission_drops
  )
  FeedResult.new(channel:, articles: outcome.articles, status:, stylesheets: config.stylesheets)
end