Class: Html2rss::AutoSource::Scraper::SemanticHtml

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb

Overview

Scrapes semantic containers via SST: Normalizer → Segmenter → Scoring → SstArticleExtractor.

Defined Under Namespace

Classes: EntryDeduplicator

Constant Summary collapse

TOP_K =

Maximum articles materialized after ranking.

Scoring::Engine::TOP_K

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body = nil, url:, document: nil, link_resolver: nil, **opts) ⇒ SemanticHtml

Returns a new instance of SemanticHtml.

Parameters:

  • parsed_body (Nokogiri::HTML::Document, nil) (defaults to: nil) —

    parsed HTML (when document: omitted)

  • url (String, Html2rss::Url) —

    base url

  • document (SST::Document, nil) (defaults to: nil) —

    memoized SST document from AutoSource

  • link_resolver (Scoring::LinkResolver, nil) (defaults to: nil) —

    shared page-scoped resolver

  • opts (Hash) —

    scraper-specific options

Options Hash (**opts):

  • :fallback_anchorless (Boolean) —

    whether to keep containers without a primary anchor



34
35
36
37
38
39
40
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 34

def initialize(parsed_body = nil, url:, document: nil, link_resolver: nil, **opts)
  @parsed_body = parsed_body
  @provided_document = document
  @provided_link_resolver = link_resolver
  @url = url
  @permit_unanchored = opts.fetch(:fallback_anchorless, false)
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Returns true when at least one semantic container has an eligible anchor.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Boolean) —

    true when at least one semantic container has an eligible anchor



20
21
22
23
24
25
26
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 20

def self.articles?(parsed_body)
  return false unless parsed_body

  new(parsed_body, url: 'https://example.com').extractable?
rescue ArgumentError
  false
end

.options_key ⇒ Symbol

Returns config key used to enable or configure this scraper.

Returns:

  • (Symbol) —

    config key used to enable or configure this scraper



16
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 16

def self.options_key = :semantic_html

Instance Method Details

#each {|article| ... } ⇒ Enumerator<Html2rss::Article>

Yield Parameters:

Returns:



45
46
47
48
49
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 45

def each
  return enum_for(:each) unless block_given?

  articles.each { yield _1 }
end

#extractable? ⇒ Boolean

Returns:

  • (Boolean) 


53
54
55
56
57
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 53

def extractable?
  ranked_segments.any?
rescue ArgumentError
  false
end