Class: Html2rss::Html::ArticleExtractor::CategoryExtractor

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/html/article_extractor/category_extractor.rb

Overview

CategoryExtractor is responsible for extracting categories from HTML elements by looking for CSS class names containing common category-related terms.

Constant Summary collapse

CATEGORY_TERMS =

Shared category vocabulary (owned by Html2rss::Html::ArticleRules::Category).

ArticleRules::Category::CATEGORY_TERMS
CATEGORY_SELECTORS =

CSS selectors to find elements with category-related class names or data attributes

CATEGORY_TERMS.flat_map do |term|
  ["[class*=\"#{term}\"]", "[data-#{term}]", "[#{term}]"]
end.freeze
CONTAINER_CHILD_SELECTOR =

Nested blocks that mean a "category" node is actually a content container.

'p, article, section, h1, h2, h3, h4, h5, h6'

Class Method Summary collapse

Class Method Details

.call(article_tag, title: nil) ⇒ Array<String>

Extracts categories from the given article tag by looking for elements with class names containing common category-related terms.

Parameters:

  • article_tag (Nokogiri::XML::Element) —

    The article element to extract categories from

  • title (String, nil) (defaults to: nil) —

    article title used to reject title-echo values

Returns:

  • (Array<String>) —

    Array of category strings, empty if none found



28
29
30
31
32
33
34
 
# File 'lib/html2rss/html/article_extractor/category_extractor.rb', line 28

def self.call(article_tag, title: nil)
  return [] unless article_tag

  extract_all_categories(article_tag, title:)
    .map(&:strip)
    .reject(&:empty?)
end

.extract_all_categories(article_tag, title: nil) ⇒ Set<String>

Optimized single DOM traversal that extracts all category types.

Parameters:

  • article_tag (Nokogiri::XML::Element) —

    The article element

  • title (String, nil) (defaults to: nil)

Returns:

  • (Set<String>) —

    Set of category strings



42
43
44
45
46
47
48
49
50
51
52
53
54
 
# File 'lib/html2rss/html/article_extractor/category_extractor.rb', line 42

def self.extract_all_categories(article_tag, title: nil)
  Set.new.tap do |categories|
    extract_element_data_categories!(categories, article_tag, title:)
    article_tag.css(CATEGORY_SELECTORS.join(',')).each do |element|
      next if element == article_tag

      if ArticleRules::Category.class_match?(element['class'])
        extract_text_categories!(categories, element, title:)
      end
      extract_element_data_categories!(categories, element, title:)
    end
  end
end

.extract_element_data_categories!(categories, element, title: nil) ⇒ void

This method returns an undefined value.

Extracts categories from data attributes of a single element.

Parameters:

  • categories (Set<String>) —

    Accumulator set

  • element (Nokogiri::XML::Element) —

    metadata element that may contain category links

  • title (String, nil) (defaults to: nil) 


63
64
65
66
67
68
69
 
# File 'lib/html2rss/html/article_extractor/category_extractor.rb', line 63

def self.extract_element_data_categories!(categories, element, title: nil)
  element.attributes.each_value do |attr|
    next unless ArticleRules::Category.attr_name_match?(attr.name)

    ArticleRules::Category.add_text!(categories, attr.value, title:)
  end
end

.extract_text_categories!(categories, element, title: nil) ⇒ void

This method returns an undefined value.

Extracts text-based categories from elements, splitting content into discrete values.

Parameters:

  • categories (Set<String>) —

    Accumulator set

  • element (Nokogiri::XML::Element) —

    metadata element whose text may contain delimiters

  • title (String, nil) (defaults to: nil) 


78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/html2rss/html/article_extractor/category_extractor.rb', line 78

def self.extract_text_categories!(categories, element, title: nil)
  return if category_container?(element)

  anchors = element.name == 'a' ? [element] : element.css('a').to_a
  if anchors.any?
    anchors.each { |node| add_text_to_categories!(categories, node, title:) }
    return
  end

  ArticleRules::Category.add_split_text!(
    categories, Navigator.extract_visible_text(element), title:
  )
end