Class: Html2rss::Selectors

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/selectors.rb,
lib/html2rss/selectors/extractors.rb,
lib/html2rss/selectors/item_scope.rb,
lib/html2rss/selectors/extractors/href.rb,
lib/html2rss/selectors/extractors/html.rb,
lib/html2rss/selectors/extractors/text.rb,
lib/html2rss/selectors/post_processors.rb,
lib/html2rss/selectors/extractors/static.rb,
lib/html2rss/selectors/extractors/attribute.rb,
lib/html2rss/selectors/post_processors/base.rb,
lib/html2rss/selectors/post_processors/gsub.rb,
lib/html2rss/selectors/object_to_xml_converter.rb,
lib/html2rss/selectors/post_processors/template.rb,
lib/html2rss/selectors/post_processors/parse_uri.rb,
lib/html2rss/selectors/post_processors/substring.rb,
lib/html2rss/selectors/post_processors/parse_time.rb,
lib/html2rss/selectors/post_processors/sanitize_html.rb,
lib/html2rss/selectors/post_processors/html_to_markdown.rb,
lib/html2rss/selectors/post_processors/markdown_to_html.rb,
lib/html2rss/selectors/post_processors/html_transformers/wrap_img_in_a.rb,
lib/html2rss/selectors/post_processors/html_transformers/transform_urls_to_absolute_ones.rb

Overview

This scraper is designed to scrape articles from a given HTML page using CSS selectors defined in the feed config.

It supports the traditional feed configs that html2rss originally provided, ensuring compatibility with existing setups.

Additionally, it uniquely offers the capability to convert JSON into XML, extending its versatility for diverse data processing workflows.

Defined Under Namespace

Modules: Extractors, PostProcessors Classes: Context, InvalidSelectorName, ItemScope, ObjectToXmlConverter

Constant Summary collapse

DEFAULT_CONFIG =

Default selectors options merged into user configuration.

{ items: { enhance: true } }.freeze
ITEMS_SELECTOR_KEY =

Selector key that points to the root list of article nodes.

:items
ITEM_TAGS =

Supported RSS item attributes extractable through selectors.

%i[title url description author comments published_at guid enclosure categories].freeze
SPECIAL_ATTRIBUTES =

Item attributes that require dedicated extraction logic.

Set[:guid, :enclosure, :categories].freeze
SELECTOR_TO_ARTICLE_KEY =

Config selector keys that map onto a different Article attribute. :enclosure stays singular in YAML; Article stores :enclosures.

{ enclosure: :enclosures }.freeze
SELECTABLE_SELECTOR_KEYS =

Selector keys that may be copied onto an Article (PROVIDED_KEYS + mapped aliases).

(Html2rss::Article::PROVIDED_KEYS + SELECTOR_TO_ARTICLE_KEY.keys).to_set.freeze

Instance Method Summary collapse

Constructor Details

#initialize(response, selectors:, time_zone:) ⇒ Selectors

Initializes a new Selectors instance.

Parameters:

  • response (RequestService::Response) —

    The response object.

  • selectors (Hash) —

    A hash of CSS selectors.

  • time_zone (String) —

    Time zone string used for date parsing.



46
47
48
49
50
51
52
53
54
 
# File 'lib/html2rss/selectors.rb', line 46

def initialize(response, selectors:, time_zone:)
  @response = response
  @url = response.url
  @selectors = selectors
  @time_zone = time_zone

  prepare_selectors!
  @rss_item_attributes = @selectors.keys.select { |key| SELECTABLE_SELECTOR_KEYS.include?(key) }
end

Instance Method Details

#articles ⇒ Array<Html2rss::Article>

Returns articles extracted from the response. Reverses order if config specifies reverse ordering.

Returns:



61
62
63
 
# File 'lib/html2rss/selectors.rb', line 61

def articles
  @articles ||= @selectors.dig(ITEMS_SELECTOR_KEY, :order) == 'reverse' ? to_a.tap(&:reverse!) : to_a
end

#each {|article| ... } ⇒ Enumerator

Iterates over each scraped article.

Yields:

  • (article) —

    Gives each article as an Html2rss::Article.

Returns:

  • (Enumerator) —

    An enumerator if no block is given.



70
71
72
73
74
75
76
77
78
79
80
81
82
 
# File 'lib/html2rss/selectors.rb', line 70

def each(&)
  return enum_for(:each) unless block_given?

  enhance = enhance?

  parsed_body.css(items_selector).each do |item|
    article_hash = extract_article(item, response)

    enhance_article_hash(article_hash, item, response.url) if enhance

    yield Html2rss::Article.new(**article_hash, scraper: self.class)
  end
end

#enhance? ⇒ Boolean

Returns whether to enhance the article hash with auto_source's semantic HTML extraction.

Returns:

  • (Boolean) —

    whether to enhance the article hash with auto_source's semantic HTML extraction.



90
 
# File 'lib/html2rss/selectors.rb', line 90

def enhance? = !!@selectors.dig(ITEMS_SELECTOR_KEY, :enhance)

#enhance_article_hash(article_hash, article_tag, base_url = @url) ⇒ Hash

Enhances the article hash using semantic HTML extraction. Only adds keys that are missing from the original hash.

rubocop:disable Metrics/MethodLength

Parameters:

  • article_hash (Hash) —

    The original article hash.

  • article_tag (Nokogiri::XML::Element) —

    HTML element to extract additional info from.

  • base_url (String, Html2rss::Url) (defaults to: @url) —

    base URL for normalization during enhancement

Returns:

  • (Hash) —

    The enhanced article hash.



118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
 
# File 'lib/html2rss/selectors.rb', line 118

def enhance_article_hash(article_hash, article_tag, base_url = @url)
  selected_anchor = Html2rss::Html::Navigator.main_anchor_for(article_tag)
  extracted = Html2rss::Html::ArticleExtractor.call(
    article_tag,
    base_url:,
    selected_anchor:,
    fallback_anchorless: true
  )
  return article_hash unless extracted

  extracted.each_with_object(article_hash) do |(key, value), hash|
    next if value.nil? || (hash.key?(key) && hash[key])

    hash[key] = value
  end
end

#extract_article(item, page_response = response) ⇒ Hash

Extracts an article hash for a given item element.

Parameters:

  • item (Nokogiri::XML::Element) —

    The element to extract from.

  • page_response (RequestService::Response) (defaults to: response) —

    response used for selector extraction context

Returns:

  • (Hash) —

    Hash of attributes for the article.



98
99
100
101
102
103
104
105
106
107
 
# File 'lib/html2rss/selectors.rb', line 98

def extract_article(item, page_response = response)
  scope = item_scope_for(item, page_response.url)
  @rss_item_attributes.each_with_object({}) do |selector_key, hash|
    value = scope.select(selector_key)
    next if value.nil?

    article_key = SELECTOR_TO_ARTICLE_KEY.fetch(selector_key, selector_key)
    hash[article_key] = article_key == :enclosures ? wrap_enclosure_value(value) : value
  end
end

#items_selector ⇒ String

Returns the CSS selector for the items.

Returns:

  • (String) —

    the CSS selector for the items



87
 
# File 'lib/html2rss/selectors.rb', line 87

def items_selector = @selectors.dig(ITEMS_SELECTOR_KEY, :selector)

#select(name, item, base_url: @url) ⇒ Object+

Selects the value for a given attribute from an HTML element.

Parameters:

  • name (Symbol, String) —

    Name of the attribute.

  • item (Nokogiri::XML::Element) —

    The HTML element to process.

  • base_url (String, Html2rss::Url) (defaults to: @url) —

    base URL for relative extraction values

Returns:

  • (Object, Array<Object>) —

    The selected value(s).

Raises:



144
145
146
 
# File 'lib/html2rss/selectors.rb', line 144

def select(name, item, base_url: @url)
  select_in_scope(name, item_scope_for(item, base_url))
end

#select_in_scope(name, scope) ⇒ Object+

Selects the value for a given attribute within an existing ItemScope. Used by Html2rss::Selectors::ItemScope#select so nested selects reuse one scope per extraction pass.

Parameters:

  • name (Symbol, String) —

    Name of the attribute.

  • scope (ItemScope) —

    Per-item extraction scope.

Returns:

  • (Object, Array<Object>) —

    The selected value(s).

Raises:



156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/html2rss/selectors.rb', line 156

def select_in_scope(name, scope)
  name = name.to_sym

  raise InvalidSelectorName, "Attribute selector '#{name}' is reserved for items." if name == ITEMS_SELECTOR_KEY

  selector_key, config = selector_config_for(name)

  if SPECIAL_ATTRIBUTES.member?(selector_key)
    select_special(selector_key, scope:, config:)
  else
    select_regular(selector_key, scope:, config:)
  end
end