Module: Html2rss::AutoSource::Scraper::JsonState::ArticleNormalizer

Defined in:
lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb

Overview

Shapes raw entries into the structure required downstream.

Constant Summary collapse

TITLE_KEYS =

Preferred keys when extracting title-like values from state payloads.

%i[title headline name text].freeze
URL_KEYS =

Preferred keys when extracting URL-like values from state payloads.

%i[url link href permalink slug path canonicalUrl shortUrl].freeze
DESCRIPTION_KEYS =

Preferred keys when extracting description-like values from state payloads.

%i[description summary excerpt dek subheading].freeze
IMAGE_KEYS =

Preferred keys when extracting image-like values from state payloads.

%i[image imageUrl thumbnailUrl thumbnail src featuredImage coverImage heroImage].freeze
PUBLISHED_AT_KEYS =

Preferred keys when extracting publication timestamps from state payloads.

%i[published_at publishedAt datePublished date publicationDate pubDate updatedAt
updated_at createdAt created_at].freeze
CATEGORY_KEYS =

Preferred keys when extracting category-like values from state payloads.

%i[categories tags section sections topic topics channel].freeze
ID_KEYS =

Preferred keys when extracting identifier-like values from state payloads.

%i[id guid uuid slug key].freeze

Class Method Summary collapse

Class Method Details

.categories(entry) ⇒ Array<String>?

rubocop:disable Metrics/MethodLength

Parameters:

  • entry (Hash) —

    raw article entry candidate

Returns:

  • (Array<String>, nil) —

    normalized unique categories



81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
 
# File 'lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb', line 81

def categories(entry)
  raw = ValueFinder.fetch(entry, CATEGORY_KEYS)
  names = case raw
          when Array then raw
          when Hash then raw.values
          when String then [raw]
          else []
          end

  result = names.flat_map do |value|
    case value
    when Hash
      string(ValueFinder.fetch(value, %i[name title label]))
    else
      string(value)
    end
  end.compact

  result.uniq!
  result unless result.empty?
end

.identifier(entry, article_url) ⇒ String

Returns stable article identifier fallbacking to resolved URL.

Parameters:

  • entry (Hash) —

    raw article entry candidate

  • article_url (Html2rss::Url) —

    resolved article URL

Returns:

  • (String) —

    stable article identifier fallbacking to resolved URL



107
108
109
110
111
 
# File 'lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb', line 107

def identifier(entry, article_url)
  value = ValueFinder.fetch(entry, ID_KEYS)
  value = ValueFinder.fetch(value, ID_KEYS) if value.is_a?(Hash)
  string(value) || article_url.to_s
end

.normalise(entry, base_url:) ⇒ Hash{Symbol => Object, nil}

rubocop:disable Metrics/MethodLength

Parameters:

  • entry (Hash) —

    raw article entry candidate

  • base_url (String, Html2rss::Url) —

    base URL for relative link resolution

Returns:

  • (Hash{Symbol => Object, nil}) —

    normalized article hash for downstream extraction



31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb', line 31

def normalise(entry, base_url:)
  return unless entry.is_a?(Hash)

  title = string(ValueFinder.fetch(entry, TITLE_KEYS))
  description = string(ValueFinder.fetch(entry, DESCRIPTION_KEYS))
  article_url = resolve_link(entry, keys: URL_KEYS, base_url:,
                                    log_key: 'JsonState: invalid URL encountered')
  return unless article_url
  return if title.nil? && description.nil?

  {
    title:,
    description:,
    url: article_url,
    image: resolve_link(entry, keys: IMAGE_KEYS, base_url:,
                               log_key: 'JsonState: invalid image URL encountered'),
    published_at: string(ValueFinder.fetch(entry, PUBLISHED_AT_KEYS)),
    categories: categories(entry),
    id: identifier(entry, article_url)
  }.compact
end

.resolve_link(entry, keys:, base_url:, log_key:) ⇒ Html2rss::Url?

Returns resolved absolute URL.

Parameters:

  • entry (Hash) —

    raw article entry candidate

  • keys (Array<String>) —

    preferred link keys

  • base_url (String, Html2rss::Url) —

    base URL for relative link resolution

  • log_key (String) —

    structured log message key

Returns:



66
67
68
69
70
71
72
73
74
75
76
 
# File 'lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb', line 66

def resolve_link(entry, keys:, base_url:, log_key:)
  value = ValueFinder.fetch(entry, keys)
  value = ValueFinder.fetch(value, keys) if value.is_a?(Hash)
  string = string(value)
  return unless string

  Url.from_relative(string, base_url)
rescue ArgumentError
  Log.debug(log_key, url: string)
  nil
end

.string(value) ⇒ String?

Returns normalized non-empty string value.

Parameters:

  • value (Object) —

    candidate scalar value

Returns:

  • (String, nil) —

    normalized non-empty string value



56
57
58
59
 
# File 'lib/html2rss/auto_source/scraper/json_state/article_normalizer.rb', line 56

def string(value)
  trimmed = value.to_s.strip
  trimmed unless trimmed.empty?
end