Class: Html2rss::Html::Navigator

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/html/navigator.rb,
lib/html2rss/html/navigator/text_extractor.rb

Overview

Navigator owns DOM chrome recognition and node traversal helpers.

Defined Under Namespace

Classes: TextExtractor

Constant Summary collapse

HEADING_TAGS =

Heading tags used to prioritize title extraction and container assessment.

%w[h1 h2 h3 h4 h5 h6].freeze
IGNORED_CONTAINER_TAGS =

Element tags that indicate ignored DOM chrome when found in a container path.

%w[nav footer header svg script style].to_set.freeze
CLUSTER_EXCLUDED_TAGS =

Layout roots and chrome tags excluded from class-clustering candidate nodes.

Set['html', 'body', 'nav', 'footer', 'header', 'svg', 'script', 'style'].freeze
UTILITY_LANDMARK_TAGS =

Ancestor tags that usually indicate navigation/utility regions inside a content container.

%w[nav aside footer menu].to_set.freeze
MAIN_ANCHOR_SELECTOR =

Anchor selector used to identify the canonical article link element.

begin
  buf = +'a[href]:not([href=""])'
  %w[# javascript: mailto: tel: file:// sms: data:].each do |prefix|
    buf << %[:not([href^="#{prefix}"])]
  end
  buf.freeze
end

Class Method Summary collapse

Class Method Details

.descendant_of?(child_node, parent_node) ⇒ Boolean

Returns true if child_node is a descendant of parent_node. Walks up using parent pointers to avoid NodeSet allocations.

Parameters:

  • child_node (Nokogiri::XML::Node) —

    potential descendant

  • parent_node (Nokogiri::XML::Node) —

    potential ancestor

Returns:

  • (Boolean) —

    true when child_node is a descendant of parent_node



134
135
136
137
138
139
140
141
142
 
# File 'lib/html2rss/html/navigator.rb', line 134

def descendant_of?(child_node, parent_node)
  curr = child_node.respond_to?(:parent) ? child_node.parent : nil
  while curr
    return true if curr == parent_node

    curr = curr.respond_to?(:parent) ? curr.parent : nil
  end
  false
end

.extract_visible_text(tag, separator: ' ', exclude_nodes: nil) ⇒ String?

Extracts visible text from a given node and its children.

Parameters:

  • tag (Nokogiri::XML::Node) —

    the node from which to extract visible text

  • separator (String) (defaults to: ' ') —

    separator used to join text fragments (default is a space)

  • exclude_nodes (Array<Nokogiri::XML::Node>, nil) (defaults to: nil) —

    nodes to exclude from extraction

Returns:

  • (String, nil) —

    the concatenated visible text, or nil if none is found



37
38
39
 
# File 'lib/html2rss/html/navigator.rb', line 37

def extract_visible_text(tag, separator: ' ', exclude_nodes: nil)
  TextExtractor.call(tag, separator:, exclude_nodes:)
end

.find_closest_selector_upwards(current_tag, selector) ⇒ Nokogiri::XML::Node?

Think of it as css_upwards method. It searches for the closest parent that matches the given selector.

Parameters:

  • current_tag (Nokogiri::XML::Node, nil) —

    starting node

  • selector (String) —

    CSS selector to search upwards for

Returns:

  • (Nokogiri::XML::Node, nil) —

    first matching node in upward traversal



104
105
106
107
108
109
110
111
112
113
 
# File 'lib/html2rss/html/navigator.rb', line 104

def find_closest_selector_upwards(current_tag, selector)
  while current_tag
    found = current_tag.at_css(selector)
    return found if found

    return nil unless current_tag.respond_to?(:parent)

    current_tag = current_tag.parent
  end
end

.find_tag_in_ancestors(current_tag, tag_name) ⇒ Nokogiri::XML::Node?

Searches for the closest parent that matches the given tag name.

Parameters:

  • current_tag (Nokogiri::XML::Node) —

    starting node

  • tag_name (String) —

    tag name to find in ancestors

Returns:

  • (Nokogiri::XML::Node, nil) —

    matching ancestor node



121
122
123
124
125
 
# File 'lib/html2rss/html/navigator.rb', line 121

def find_tag_in_ancestors(current_tag, tag_name)
  return current_tag if current_tag.name == tag_name

  current_tag.ancestors(tag_name).first
end

.ignored_container_path?(node, cache = nil) ⇒ Boolean

rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength, Metrics/PerceivedComplexity

Parameters:

  • node (Nokogiri::XML::Node)
  • cache (Hash, nil) (defaults to: nil) —

    identity cache used to store results (must use compare_by_identity)

Returns:

  • (Boolean) —

    true when the node belongs to ignored DOM chrome



55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
 
# File 'lib/html2rss/html/navigator.rb', line 55

def ignored_container_path?(node, cache = nil)
  return cache[node] if cache&.key?(node)

  curr = node
  visited = []
  is_ignored = false

  while curr.respond_to?(:parent) && curr
    if cache&.key?(curr)
      is_ignored = cache[curr]
      break
    end

    if IGNORED_CONTAINER_TAGS.include?(curr.name)
      is_ignored = true
      break
    end

    visited << curr
    curr = curr.parent
  end
  visited.each { |n| cache[n] = is_ignored } if cache

  is_ignored
end

.main_anchor_for(article_tag) ⇒ Nokogiri::XML::Node?

Returns first eligible descendant anchor.

Parameters:

  • article_tag (Nokogiri::XML::Node) —

    article-like container to search within

Returns:

  • (Nokogiri::XML::Node, nil) —

    first eligible descendant anchor



44
45
46
47
48
 
# File 'lib/html2rss/html/navigator.rb', line 44

def main_anchor_for(article_tag)
  return article_tag if article_tag.name == 'a' && article_tag.matches?(MAIN_ANCHOR_SELECTOR)

  article_tag.at_css(MAIN_ANCHOR_SELECTOR)
end

.parent_until_condition(node, condition) ⇒ Nokogiri::XML::Node?

Returns the first parent that satisfies the condition. If the condition is met, it returns the node itself.

Parameters:

  • node (Nokogiri::XML::Node) —

    The node to start the search from.

  • condition (Proc) —

    The condition to be met.

Returns:

  • (Nokogiri::XML::Node, nil) —

    The first parent that satisfies the condition.



89
90
91
92
93
94
95
 
# File 'lib/html2rss/html/navigator.rb', line 89

def parent_until_condition(node, condition)
  while node && !node.document? && node.name != 'html'
    return node if condition.call(node)

    node = node.parent
  end
end