Class: Html2rss::Html::Navigator

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/html/navigator.rb,
lib/html2rss/html/navigator/text_extractor.rb

Overview

Navigator owns DOM chrome recognition and node traversal helpers.

Defined Under Namespace

Classes: TextExtractor

Constant Summary collapse

HEADING_TAGS =

Heading tags used to prioritize title extraction and container assessment.

%w[h1 h2 h3 h4 h5 h6].freeze
IGNORED_CONTAINER_TAGS =

Element tags that indicate ignored DOM chrome when found in a container path.

%w[nav footer header svg script style].to_set.freeze
CLUSTER_EXCLUDED_TAGS =

Layout roots and chrome tags excluded from class-clustering candidate nodes.

Set['html', 'body', 'nav', 'footer', 'header', 'svg', 'script', 'style'].freeze
UTILITY_LANDMARK_TAGS =

Ancestor tags that usually indicate navigation/utility regions inside a content container.

%w[nav aside footer menu].to_set.freeze
CARD_WALK_STOP_TAGS =

Immediate parent walk stops here — not a usable article card.

(UTILITY_LANDMARK_TAGS | IGNORED_CONTAINER_TAGS | %w[html body]).freeze
WRAPPING_ANCHOR_CHILD_TAGS =

Inner tags that mean a wrapping is a card, not a span-styled list link.

(HEADING_TAGS + %w[p]).freeze
MAIN_ANCHOR_SELECTOR =

Anchor selector used to identify the canonical article link element.

begin
  buf = +'a[href]:not([href=""])'
  %w[# javascript: mailto: tel: file:// sms: data:].each do |prefix|
    buf << %[:not([href^="#{prefix}"])]
  end
  buf.freeze
end

Class Method Summary collapse

Class Method Details

.descendant_of?(child_node, parent_node) ⇒ Boolean

Returns true if child_node is a descendant of parent_node. Walks up using parent pointers to avoid NodeSet allocations.

Parameters:

  • child_node (Nokogiri::XML::Node) —

    potential descendant

  • parent_node (Nokogiri::XML::Node) —

    potential ancestor

Returns:

  • (Boolean) —

    true when child_node is a descendant of parent_node



152
153
154
155
156
157
158
159
160
 
# File 'lib/html2rss/html/navigator.rb', line 152

def descendant_of?(child_node, parent_node)
  curr = child_node.respond_to?(:parent) ? child_node.parent : nil
  while curr
    return true if curr == parent_node

    curr = curr.respond_to?(:parent) ? curr.parent : nil
  end
  false
end

.extract_visible_text(tag, separator: ' ', exclude_nodes: nil) ⇒ String?

Extracts visible text from a given node and its children.

Parameters:

  • tag (Nokogiri::XML::Node) —

    the node from which to extract visible text

  • separator (String) (defaults to: ' ') —

    separator used to join text fragments (default is a space)

  • exclude_nodes (Array<Nokogiri::XML::Node>, nil) (defaults to: nil) —

    nodes to exclude from extraction

Returns:

  • (String, nil) —

    the concatenated visible text, or nil if none is found



43
44
45
 
# File 'lib/html2rss/html/navigator.rb', line 43

def extract_visible_text(tag, separator: ' ', exclude_nodes: nil)
  TextExtractor.call(tag, separator:, exclude_nodes:)
end

.find_closest_selector_upwards(current_tag, selector) ⇒ Nokogiri::XML::Node?

Think of it as css_upwards method. It searches for the closest parent that matches the given selector.

Parameters:

  • current_tag (Nokogiri::XML::Node, nil) —

    starting node

  • selector (String) —

    CSS selector to search upwards for

Returns:

  • (Nokogiri::XML::Node, nil) —

    first matching node in upward traversal



122
123
124
125
126
127
128
129
130
131
 
# File 'lib/html2rss/html/navigator.rb', line 122

def find_closest_selector_upwards(current_tag, selector)
  while current_tag
    found = current_tag.at_css(selector)
    return found if found

    return nil unless current_tag.respond_to?(:parent)

    current_tag = current_tag.parent
  end
end

.find_tag_in_ancestors(current_tag, tag_name) ⇒ Nokogiri::XML::Node?

Searches for the closest parent that matches the given tag name.

Parameters:

  • current_tag (Nokogiri::XML::Node) —

    starting node

  • tag_name (String) —

    tag name to find in ancestors

Returns:

  • (Nokogiri::XML::Node, nil) —

    matching ancestor node



139
140
141
142
143
 
# File 'lib/html2rss/html/navigator.rb', line 139

def find_tag_in_ancestors(current_tag, tag_name)
  return current_tag if current_tag.name == tag_name

  current_tag.ancestors(tag_name).first
end

.ignored_container_path?(node, cache = nil) ⇒ Boolean

rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity

Parameters:

  • node (Nokogiri::XML::Node)
  • cache (Hash, nil) (defaults to: nil) —

    identity cache used to store results (must use compare_by_identity)

Returns:

  • (Boolean) —

    true when the node belongs to ignored DOM chrome



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
 
# File 'lib/html2rss/html/navigator.rb', line 61

def ignored_container_path?(node, cache = nil)
  return cache[node] if cache&.key?(node)

  curr = node
  visited = []
  is_ignored = false

  while curr.respond_to?(:parent) && curr
    if cache&.key?(curr)
      is_ignored = cache[curr]
      break
    end

    if IGNORED_CONTAINER_TAGS.include?(curr.name)
      is_ignored = true
      break
    end

    visited << curr
    curr = curr.parent
  end
  visited.each { |n| cache[n] = is_ignored } if cache

  is_ignored
end

.main_anchor_for(article_tag) ⇒ Nokogiri::XML::Node?

Returns first eligible descendant anchor.

Parameters:

  • article_tag (Nokogiri::XML::Node) —

    article-like container to search within

Returns:

  • (Nokogiri::XML::Node, nil) —

    first eligible descendant anchor



50
51
52
53
54
 
# File 'lib/html2rss/html/navigator.rb', line 50

def main_anchor_for(article_tag)
  return article_tag if article_tag.name == 'a' && article_tag.matches?(MAIN_ANCHOR_SELECTOR)

  article_tag.at_css(MAIN_ANCHOR_SELECTOR)
end

.parent_until_condition(node, condition) ⇒ Nokogiri::XML::Node?

Returns the first parent that satisfies the condition. If the condition is met, it returns the node itself.

Parameters:

  • node (Nokogiri::XML::Node) —

    The node to start the search from.

  • condition (Proc) —

    The condition to be met.

Returns:

  • (Nokogiri::XML::Node, nil) —

    The first parent that satisfies the condition.



95
96
97
98
99
100
101
 
# File 'lib/html2rss/html/navigator.rb', line 95

def parent_until_condition(node, condition)
  while node && !node.document? && node.name != 'html'
    return node if condition.call(node)

    node = node.parent
  end
end

.usable_card_parent?(node) ⇒ Boolean

Immediate parent is a usable article card (not html/body/landmark chrome).

Parameters:

Returns:

  • (Boolean) 


108
109
110
111
112
113
 
# File 'lib/html2rss/html/navigator.rb', line 108

def usable_card_parent?(node)
  return false unless node
  return false if node.respond_to?(:document?) && node.document?

  !CARD_WALK_STOP_TAGS.include?(node.name.to_s)
end