Class: Html2rss::AutoSource::Scraper::SemanticHtml

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/auto_source/scraper/semantic_html.rb,
lib/html2rss/auto_source/scraper/semantic_html/entry_deduplicator.rb

Overview

Scrapes semantic containers by choosing one primary content link per block before extraction.

This scraper is intentionally container-first:

  1. collect candidate semantic containers once
  2. select the strongest content-like anchor within each container
  3. extract fields from the container while honoring that anchor choice

The result is lower recall on weak-signal blocks, but much better link quality on modern teaser cards that mix headlines, utility links, and duplicate image overlays.

Defined Under Namespace

Classes: Entry, EntryDeduplicator

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(parsed_body, url:, extractor: Html2rss::Html::ArticleExtractor, **opts) ⇒ SemanticHtml

Returns a new instance of SemanticHtml.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

  • url (String, Html2rss::Url) —

    base url

  • extractor (Class) (defaults to: Html2rss::Html::ArticleExtractor) —

    extractor class used for article extraction

  • opts (Hash) —

    scraper-specific options

Options Hash (**opts):

  • :fallback_anchorless (Boolean) —

    whether to keep containers without a primary anchor



50
51
52
53
54
55
56
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 50

def initialize(parsed_body, url:, extractor: Html2rss::Html::ArticleExtractor, **opts)
  @parsed_body = parsed_body
  @url = url
  @extractor = extractor
  @permit_unanchored = opts.fetch(:fallback_anchorless, false)
  @link_heuristics = LinkHeuristics.new(url)
end

Instance Attribute Details

#parsed_body ⇒ Object (readonly)

Returns the value of attribute parsed_body.



58
59
60
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 58

def parsed_body
  @parsed_body
end

Class Method Details

.articles?(parsed_body) ⇒ Boolean

Returns true when at least one semantic container has an eligible anchor.

Parameters:

  • parsed_body (Nokogiri::HTML::Document) —

    parsed HTML document

Returns:

  • (Boolean) —

    true when at least one semantic container has an eligible anchor



39
40
41
42
43
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 39

def self.articles?(parsed_body)
  return false unless parsed_body

  new(parsed_body, url: 'https://example.com').extractable?
end

.options_key ⇒ Symbol

Returns config key used to enable or configure this scraper.

Returns:

  • (Symbol) —

    config key used to enable or configure this scraper



35
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 35

def self.options_key = :semantic_html

Instance Method Details

#candidate_containers ⇒ Array<Nokogiri::XML::Node>

Returns:

  • (Array<Nokogiri::XML::Node>) 


86
87
88
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 86

def candidate_containers
  @candidate_containers ||= collect_candidate_containers
end

#collect_candidate_containers ⇒ Object

rubocop:enable Metrics/MethodLength



154
155
156
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 154

def collect_candidate_containers
  Discovery::SemanticContainers.call(parsed_body)
end

#each {|article_hash| ... } ⇒ Enumerator<Hash>

Yields extracted article hashes for each semantic container that survives anchor selection.

Detection and extraction share the same memoized entry list so this scraper does not rerun anchor ranking once a page has already been accepted as extractable.

Yield Parameters:

  • article_hash (Hash) —

    extracted article hash

Returns:

  • (Enumerator<Hash>) 


70
71
72
73
74
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 70

def each
  return enum_for(:each) unless block_given?

  ranked_entries.each { yield _1.article }
end

#extractable? ⇒ Boolean

Reports whether the page contains at least one semantic container with a selectable primary anchor.

Returns:

  • (Boolean) —

    true when at least one candidate container yields a primary anchor



81
82
83
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 81

def extractable?
  extractable_entries.any?
end

#extractable_entries ⇒ Object

rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity



100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 100

def extractable_entries
  @extractable_entries ||= candidate_containers.filter_map do |container|
    selected_anchor = primary_anchor_for(container)

    next unless selected_anchor || @permit_unanchored

    destination_facts = selected_anchor ? normalized_destination(selected_anchor) : nil
    next if selected_anchor && !destination_facts
    # Cheap path-only reject before title/DOM hard-junk observations.
    next if destination_facts&.high_confidence_junk_path

    signals = @link_heuristics.assess_container(container, selected_anchor, destination_facts:)
    next if signals.hard_junk?

    Entry.new(
      container:,
      selected_anchor:,
      destination_facts:,
      quality_score: signals.quality_score,
      junk_score: signals.junk_score,
      final_score: signals.final_score,
      position: document_position(container),
      article: nil
    )
  end
end

#primary_anchor_for(container) ⇒ Nokogiri::XML::Node?

Parameters:

  • container (Nokogiri::XML::Node)

Returns:

  • (Nokogiri::XML::Node, nil) 


92
93
94
95
96
97
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 92

def primary_anchor_for(container)
  Discovery::SemanticAnchorCandidates.new(
    container,
    link_heuristics: @link_heuristics
  ).to_a.max_by(&:score)&.anchor
end

#ranked_entries ⇒ Object

rubocop:disable Metrics/MethodLength



129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
 
# File 'lib/html2rss/auto_source/scraper/semantic_html.rb', line 129

def ranked_entries
  @ranked_entries ||= begin
    deduplicator = EntryDeduplicator.new(@url, @extractor)
    entries = deduplicator.call(extractable_entries)
    entries = stable_rank(entries)

    entries.filter_map do |entry|
      article = deduplicator.article_for(entry)
      next unless article

      Entry.new(
        container: entry.container,
        selected_anchor: entry.selected_anchor,
        destination_facts: entry.destination_facts,
        quality_score: entry.quality_score,
        junk_score: entry.junk_score,
        final_score: entry.final_score,
        position: entry.position,
        article:
      )
    end
  end
end