Class: Html2rss::Capture

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/capture.rb

Overview

Analyzes a URL and produces a durable feed config: items selector + enhance: true.

Fetches via FeedPipeline (including AutoFallback for :auto), extracts articles, then derives a reusable items CSS selector from SST segments (list → cluster → semantic).

Capture

Html2rss.capture (and CLI html2rss capture) analyzes a URL through the feed pipeline and produces a reusable config with items selector + enhance: true only — no title/url/description attribute-selector soup. At feed-build time, enhance: true fills missing article fields via Html::ArticleExtractor on each matched item.

When to use it

Point capture at a listing URL when you want a first-draft YAML config instead of hand-writing selectors. Treat the output as a draft: selector quality depends on page structure.

Gem API

config = Html2rss.capture('https://example.com/articles')

# {
#   channel: { url: "...", title: "...", time_zone: "UTC" },
#   selectors: {
#     items: { selector: "div.post", enhance: true }
#   }
# }

File.write('my-feed.yml', YAML.dump(Html2rss::HashUtil.deep_stringify_keys(config)))
feed = Html2rss.feed(config)

Capture.build returns a CaptureResult with quality meta (has_selectors, segment_strategy, admission_drops, selected_strategy). Html2rss.capture returns only the config hash.

Options

Html2rss.capture('https://spa-site.com', strategy: :botasaurus)
Html2rss.capture('https://example.com', items_selector: '.article-card')
Html2rss.capture('https://example.com', strategy: :local_file, local_file_path: './page.html')

strategy: :auto uses the same AutoFallback chain as scrape (faraday → botasaurus). When AutoFallback selects a concrete strategy (or you pin one), Capture stamps strategy: into the emitted config so later Html2rss.feed(config) replays the same transport.

CLI

html2rss capture https://example.com/articles
html2rss capture https://example.com --strategy botasaurus
html2rss capture https://example.com/articles > my-feed.yml
html2rss capture https://example.com --input ./page.html
html2rss capture https://example.com --explain   # quality JSON on stderr; YAML on stdout

How it works

  1. Request — FeedPipeline (AutoFallback when :auto)
  2. Discover — AutoSource extracts admitted articles
  3. Segment — try SST Segmenter strategies :list → :cluster → :semantic
  4. Gate — emit items selector only when ≥ MIN_SELECTOR_MATCHES (2) articles match
  5. Assemble — { items: { selector:, enhance: true } } plus channel

Constraints

  • Selector quality depends on page structure; treat output as a first draft.
  • When the quality gate fails, selectors are omitted (has_selectors: false) rather than inventing attribute selectors.
  • Botasaurus hops need BOTASAURUS_SCRAPER_URL.

See also AutoSource for article discovery and FeedPipeline for the :auto request chain.

Defined Under Namespace

Classes: CaptureResult

Constant Summary collapse

SEGMENT_STRATEGIES =

Ordered Segmenter strategies tried until the items selector quality gate passes.

%i[list cluster semantic].freeze
MIN_SELECTOR_MATCHES =

Minimum matched segment/article pairs required to emit an items selector.

2

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(url, strategy: :auto, **options) ⇒ Capture

Returns a new instance of Capture.

Parameters:

  • url (String) —

    source page URL

  • strategy (Symbol) (defaults to: :auto) —

    request strategy

  • options (Hash) —

    additional options

Options Hash (**options):

  • :items_selector (String, nil) —

    optional selector hint

  • :max_redirects (Integer, nil) —

    optional redirect limit override

  • :max_requests (Integer, nil) —

    optional request budget override

  • :limit (Integer, nil) —

    max articles to keep

  • :local_file_path (String, nil) —

    optional local HTML file path



54
55
56
57
58
59
60
61
62
 
# File 'lib/html2rss/capture.rb', line 54

def initialize(url, strategy: :auto, **options)
  @url = url
  @strategy = strategy
  @items_selector_hint = options.delete(:items_selector)
  @max_redirects = options.delete(:max_redirects)
  @max_requests = options.delete(:max_requests)
  @limit = options.delete(:limit)
  @local_file_path = options.delete(:local_file_path)
end

Class Method Details

.build(url, strategy: :auto) ⇒ CaptureResult

Analyzes a URL and builds a reusable feed config.

Parameters:

  • url (String) —

    source page URL

  • strategy (Symbol) (defaults to: :auto) —

    request strategy (+:auto+, :faraday, :botasaurus)

  • options (Hash) —

    a customizable set of options

Returns:



40
41
42
 
# File 'lib/html2rss/capture.rb', line 40

def build(url, strategy: :auto, **)
  new(url, strategy:, **).build
end

Instance Method Details

#build ⇒ CaptureResult

Runs the capture pipeline.

Returns:



68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/html2rss/capture.rb', line 68

def build # rubocop:disable Metrics/AbcSize, Metrics/MethodLength -- outcome + selector + result assembly
  outcome = FeedPipeline.new(raw_config).to_outcome
  selectors, segment_strategy = derive_selectors(outcome.response, outcome.articles)

  config = {
    channel: build_channel(outcome.response),
    selectors: selectors.empty? ? nil : selectors,
    **strategy_stamp(outcome),
    **local_file_request_overlay
  }.compact

  CaptureResult.new(
    config:,
    articles_count: outcome.articles.size,
    channel_title: channel_title_from(outcome.response),
    has_selectors: !selectors.empty?,
    segment_strategy:,
    admission_drops: outcome.admission_drops,
    selected_strategy: outcome.selected_strategy
  )
end