Class: Html2rss::AutoSource::Scraper::Html

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/html.rb

Overview

Fallback HTML list/cluster scraper via SST pipeline (Normalizer → Segmenter → Scoring::Engine → Html::SstArticleExtractor).

Constant Summary collapse

DETECTION_BASE_URL =

Absolute base URL used when probe-time detection needs to normalize relative hrefs.

'https://example.com'
DEFAULT_MINIMUM_SELECTOR_FREQUENCY =

Minimum selector frequency required to treat a path as a stable list signal.

2
DEFAULT_USE_TOP_SELECTORS =

Number of most frequent selectors kept for container extraction.

5
TOP_K =

Maximum articles materialized after eligibility filtering.

Scoring::Engine::TOP_K

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body = nil, url:, document: nil, link_resolver: nil, **opts) ⇒ Html

Returns a new instance of Html.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) (defaults to: nil) —

    parsed HTML (when document: omitted)

  • url (String, Html2rss::Url)
  • document (SST::Document, nil) (defaults to: nil) —

    memoized SST document from AutoSource

  • link_resolver (Scoring::LinkResolver, nil) (defaults to: nil) —

    shared page-scoped resolver

  • opts (Hash)

Options Hash (**opts):

  • :fallback_anchorless (Boolean) —

    keep anchorless cluster cards

  • :minimum_selector_frequency (Integer) —

    list frequency floor

  • :use_top_selectors (Integer) —

    list selector budget



44
45
46
47
48
49
50
51
 
# File 'lib/html2rss/auto_source/scraper/html.rb', line 44

def initialize(parsed_body = nil, url:, document: nil, link_resolver: nil, **opts)
  @parsed_body = parsed_body
  @provided_document = document
  @provided_link_resolver = link_resolver
  @url = url
  @opts = opts
  @fallback_anchorless = opts.fetch(:fallback_anchorless, false)
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Parameters:

  • parsed_body (Nokogiri::HTML::Document)

Returns:

  • (Boolean) 


28
29
30
31
32
33
34
 
# File 'lib/html2rss/auto_source/scraper/html.rb', line 28

def self.articles?(parsed_body)
  return false unless parsed_body

  new(parsed_body, url: DETECTION_BASE_URL).extractable?
rescue ArgumentError
  false
end

.options_key ⇒ Symbol

Returns:

  • (Symbol) 


23
 
# File 'lib/html2rss/auto_source/scraper/html.rb', line 23

def self.options_key = :html

Instance Method Details

#each {|article| ... } ⇒ Enumerator

Yield Parameters:

Returns:

  • (Enumerator) 


56
57
58
59
60
 
# File 'lib/html2rss/auto_source/scraper/html.rb', line 56

def each
  return enum_for(:each) unless block_given?

  articles.each { yield _1 }
end

#extractable? ⇒ Boolean

Returns:

  • (Boolean) 


64
65
66
67
68
 
# File 'lib/html2rss/auto_source/scraper/html.rb', line 64

def extractable?
  ranked_segments.any?
rescue ArgumentError
  false
end