Class: Html2rss::Selectors

Inherits:
Object
  • Object
show all
Includes:
Enumerable
Defined in:
lib/html2rss/selectors.rb,
lib/html2rss/selectors/extractors.rb,
lib/html2rss/selectors/item_scope.rb,
lib/html2rss/selectors/extractors/href.rb,
lib/html2rss/selectors/extractors/html.rb,
lib/html2rss/selectors/extractors/text.rb,
lib/html2rss/selectors/post_processors.rb,
lib/html2rss/selectors/extractors/static.rb,
lib/html2rss/selectors/extractors/attribute.rb,
lib/html2rss/selectors/post_processors/base.rb,
lib/html2rss/selectors/post_processors/gsub.rb,
lib/html2rss/selectors/object_to_xml_converter.rb,
lib/html2rss/selectors/post_processors/template.rb,
lib/html2rss/selectors/post_processors/parse_uri.rb,
lib/html2rss/selectors/post_processors/substring.rb,
lib/html2rss/selectors/post_processors/parse_time.rb,
lib/html2rss/selectors/post_processors/sanitize_html.rb,
lib/html2rss/selectors/post_processors/html_to_markdown.rb,
lib/html2rss/selectors/post_processors/markdown_to_html.rb,
lib/html2rss/selectors/post_processors/html_transformers/wrap_img_in_a.rb,
lib/html2rss/selectors/post_processors/html_transformers/transform_urls_to_absolute_ones.rb

Overview

This scraper is designed to scrape articles from a given HTML page using CSS selectors defined in the feed config.

It supports the traditional feed configs that html2rss originally provided, ensuring compatibility with existing setups.

Additionally, it uniquely offers the capability to convert JSON into XML, extending its versatility for diverse data processing workflows.

Defined Under Namespace

Modules: Extractors, PostProcessors Classes: Context, InvalidSelectorName, ItemScope, ObjectToXmlConverter

Constant Summary collapse

DEFAULT_CONFIG =

Default selectors options merged into user configuration.

{ items: { enhance: true } }.freeze
ITEMS_SELECTOR_KEY =

Selector key that points to the root list of article nodes.

:items
ITEM_TAGS =

Supported RSS item attributes extractable through selectors.

%i[title url description author comments published_at guid enclosure categories].freeze
SPECIAL_ATTRIBUTES =

Item attributes that require dedicated extraction logic.

Set[:guid, :enclosure, :categories].freeze

Instance Method Summary collapse

Constructor Details

#initialize(response, selectors:, time_zone:) ⇒ Selectors

Initializes a new Selectors instance.

Parameters:

  • response (RequestService::Response) —

    The response object.

  • selectors (Hash) —

    A hash of CSS selectors.

  • time_zone (String) —

    Time zone string used for date parsing.



41
42
43
44
45
46
47
48
49
 
# File 'lib/html2rss/selectors.rb', line 41

def initialize(response, selectors:, time_zone:)
  @response = response
  @url = response.url
  @selectors = selectors
  @time_zone = time_zone

  prepare_selectors!
  @rss_item_attributes = @selectors.keys & Html2rss::Article::PROVIDED_KEYS
end

Instance Method Details

#articles ⇒ Array<Html2rss::Article>

Returns articles extracted from the response. Reverses order if config specifies reverse ordering.

Returns:



56
57
58
 
# File 'lib/html2rss/selectors.rb', line 56

def articles
  @articles ||= @selectors.dig(ITEMS_SELECTOR_KEY, :order) == 'reverse' ? to_a.tap(&:reverse!) : to_a
end

#each {|article| ... } ⇒ Enumerator

Iterates over each scraped article.

Yields:

  • (article) —

    Gives each article as an Html2rss::Article.

Returns:

  • (Enumerator) —

    An enumerator if no block is given.



65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/html2rss/selectors.rb', line 65

def each(&)
  return enum_for(:each) unless block_given?

  enhance = enhance?

  parsed_body.css(items_selector).each do |item|
    article_hash = extract_article(item, response)

    enhance_article_hash(article_hash, item, response.url) if enhance

    yield Html2rss::Article.new(**article_hash, scraper: self.class)
  end
end

#enhance? ⇒ Boolean

Returns whether to enhance the article hash with auto_source's semantic HTML extraction.

Returns:

  • (Boolean) —

    whether to enhance the article hash with auto_source's semantic HTML extraction.



85
 
# File 'lib/html2rss/selectors.rb', line 85

def enhance? = !!@selectors.dig(ITEMS_SELECTOR_KEY, :enhance)

#enhance_article_hash(article_hash, article_tag, base_url = @url) ⇒ Hash

Enhances the article hash using semantic HTML extraction. Only adds keys that are missing from the original hash.

rubocop:disable Metrics/MethodLength

Parameters:

  • article_hash (Hash) —

    The original article hash.

  • article_tag (Nokogiri::XML::Element) —

    HTML element to extract additional info from.

  • base_url (String, Html2rss::Url) (defaults to: @url) —

    base URL for normalization during enhancement

Returns:

  • (Hash) —

    The enhanced article hash.



107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
 
# File 'lib/html2rss/selectors.rb', line 107

def enhance_article_hash(article_hash, article_tag, base_url = @url)
  selected_anchor = Html2rss::Html::Navigator.main_anchor_for(article_tag)
  extracted = Html2rss::Html::ArticleExtractor.call(
    article_tag,
    base_url:,
    selected_anchor:,
    fallback_anchorless: true
  )
  return article_hash unless extracted

  extracted.each_with_object(article_hash) do |(key, value), hash|
    next if value.nil? || (hash.key?(key) && hash[key])

    hash[key] = value
  end
end

#extract_article(item, page_response = response) ⇒ Hash

Extracts an article hash for a given item element.

Parameters:

  • item (Nokogiri::XML::Element) —

    The element to extract from.

  • page_response (RequestService::Response) (defaults to: response) —

    response used for selector extraction context

Returns:

  • (Hash) —

    Hash of attributes for the article.



93
94
95
96
 
# File 'lib/html2rss/selectors.rb', line 93

def extract_article(item, page_response = response)
  scope = item_scope_for(item, page_response.url)
  @rss_item_attributes.to_h { |key| [key, scope.select(key)] }.compact
end

#items_selector ⇒ String

Returns the CSS selector for the items.

Returns:

  • (String) —

    the CSS selector for the items



82
 
# File 'lib/html2rss/selectors.rb', line 82

def items_selector = @selectors.dig(ITEMS_SELECTOR_KEY, :selector)

#select(name, item, base_url: @url) ⇒ Object+

Selects the value for a given attribute from an HTML element.

Parameters:

  • name (Symbol, String) —

    Name of the attribute.

  • item (Nokogiri::XML::Element) —

    The HTML element to process.

  • base_url (String, Html2rss::Url) (defaults to: @url) —

    base URL for relative extraction values

Returns:

  • (Object, Array<Object>) —

    The selected value(s).

Raises:



133
134
135
 
# File 'lib/html2rss/selectors.rb', line 133

def select(name, item, base_url: @url)
  select_in_scope(name, item_scope_for(item, base_url))
end

#select_in_scope(name, scope) ⇒ Object+

Selects the value for a given attribute within an existing ItemScope. Used by Html2rss::Selectors::ItemScope#select so nested selects reuse one scope per extraction pass.

Parameters:

  • name (Symbol, String) —

    Name of the attribute.

  • scope (ItemScope) —

    Per-item extraction scope.

Returns:

  • (Object, Array<Object>) —

    The selected value(s).

Raises:



145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/html2rss/selectors.rb', line 145

def select_in_scope(name, scope)
  name = name.to_sym

  raise InvalidSelectorName, "Attribute selector '#{name}' is reserved for items." if name == ITEMS_SELECTOR_KEY

  selector_key, config = selector_config_for(name)

  if SPECIAL_ATTRIBUTES.member?(selector_key)
    select_special(selector_key, scope:, config:)
  else
    select_regular(selector_key, scope:, config:)
  end
end