Class: Html2rss::AutoSource::LinkHeuristics

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source/link_heuristics.rb,
lib/html2rss/auto_source/link_heuristics/anchor_signals.rb,
lib/html2rss/auto_source/link_heuristics/href_extractor.rb,
lib/html2rss/auto_source/link_heuristics/path_classifier.rb,
lib/html2rss/auto_source/link_heuristics/text_classifier.rb,
lib/html2rss/auto_source/link_heuristics/container_signals.rb,
lib/html2rss/auto_source/link_heuristics/destination_facts.rb,
lib/html2rss/auto_source/link_heuristics/container_assessor.rb

Overview

Shared link eligibility and scoring policy for AutoSource scrapers.

Scrapers collect DOM observations; this module owns junk/noise rules and numeric weights so eligibility policy stays in one place.

Defined Under Namespace

Classes: AnchorSignals, ContainerAssessor, ContainerSignals, DestinationFacts, HrefExtractor, PathClassifier, TextClassifier

Constant Summary collapse

ANCHOR_SCORE_RULES =

Score weights keyed by AnchorSignals member name.

{
  heading_anchor: 100,
  heading_text_match: 20,
  meaningful_text: 10,
  content_like_destination: 10
}.freeze

Instance Method Summary collapse

Constructor Details

#initialize(base_url) ⇒ LinkHeuristics

Returns a new instance of LinkHeuristics.

Parameters:

  • base_url (String, Html2rss::Url) —

    page URL used to resolve relative hrefs



12
13
14
15
16
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 12

def initialize(base_url)
  @base_url = base_url
  @text_classifier = TextClassifier.new
  @container_assessor = ContainerAssessor.new(text_classifier: @text_classifier)
end

Instance Method Details

#assess_container(container, selected_anchor, destination_facts:) ⇒ ContainerSignals

Observes a container and builds ranking signals, including hard-junk.

Delegates DOM observation to ContainerAssessor so SemanticHtml only orchestrates candidates and extraction, while ContainerSignals keeps scoring policy.

Parameters:

  • container (Nokogiri::XML::Node) —

    semantic container node

  • selected_anchor (Nokogiri::XML::Node, nil) —

    primary anchor for the container

  • destination_facts (DestinationFacts, nil) —

    route facts for the selected anchor

Returns:



84
85
86
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 84

def assess_container(container, selected_anchor, destination_facts:)
  @container_assessor.call(container, selected_anchor, destination_facts:)
end

#destination_facts(anchor_or_href) ⇒ DestinationFacts?

Builds normalized destination facts for an anchor element or href string.

Parameters:

  • anchor_or_href (Nokogiri::XML::Element, String, #to_s) —

    anchor element or href-like value

Returns:

  • (DestinationFacts, nil) —

    normalized destination facts, or nil for blank/invalid URLs



22
23
24
25
26
27
28
29
30
31
32
33
34
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 22

def destination_facts(anchor_or_href)
  return node_facts[anchor_or_href] if node_facts.key?(anchor_or_href)

  href = HrefExtractor.call(anchor_or_href)
  return unless href

  res = memoized_destination_facts(href)

  node_facts[anchor_or_href] = res if anchor_or_href.is_a?(Nokogiri::XML::Node)
  res
rescue ArgumentError
  nil
end

#noise_anchor?(text:, destination_facts:, anchor: nil, container: nil, heading_anchor: false) ⇒ Boolean

Whether an anchor is junk chrome rather than a content permalink.

One eligibility home for Html and SemanticHtml: taxonomy/utility text rules plus optional DOM checks (icon-only, utility landmarks).

Parameters:

  • text (String, #to_s) —

    visible anchor text

  • destination_facts (DestinationFacts, nil) —

    route facts for the href

  • anchor (Nokogiri::XML::Node, nil) (defaults to: nil) —

    anchor node for icon/landmark checks

  • container (Nokogiri::XML::Node, nil) (defaults to: nil) —

    content container bounding landmark walks

  • heading_anchor (Boolean) (defaults to: false) —

    whether the anchor is the container heading link

Returns:

  • (Boolean) —

    true when the anchor should be ignored



60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 60

def noise_anchor?(text:, destination_facts:, anchor: nil, container: nil, heading_anchor: false) # rubocop:disable Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
  return true unless destination_facts

  destination_facts.taxonomy_path ||
    short_utility_label?(text, destination_facts) ||
    recommended_chrome?(text, destination_facts, heading_anchor:) ||
    (utility_prefix_text?(text) && destination_facts.high_confidence_utility_destination) ||
    (utility_text?(text) && destination_facts.vanity_path) ||
    utility_text_chrome?(text, destination_facts, heading_anchor:) ||
    icon_only_anchor?(anchor, text) ||
    utility_landmark_ancestor?(anchor, container)
end

#recommended_text?(text) ⇒ Boolean

Returns true when text identifies recommendation chrome.

Parameters:

  • text (String, #to_s) —

    visible anchor text

Returns:

  • (Boolean) —

    true when text identifies recommendation chrome



46
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 46

def recommended_text?(text) = @text_classifier.recommended?(text)

#utility_prefix_text?(text) ⇒ Boolean

Returns true when text begins with a utility label.

Parameters:

  • text (String, #to_s) —

    visible anchor text

Returns:

  • (Boolean) —

    true when text begins with a utility label



42
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 42

def utility_prefix_text?(text) = @text_classifier.utility_prefix?(text)

#utility_text?(text) ⇒ Boolean

Returns true when text matches a utility label.

Parameters:

  • text (String, #to_s) —

    visible anchor text

Returns:

  • (Boolean) —

    true when text matches a utility label



38
 
# File 'lib/html2rss/auto_source/link_heuristics.rb', line 38

def utility_text?(text) = @text_classifier.utility?(text)