Module: Html2rss::AutoSource::Scraper::JsonState::CandidateDetector

Defined in:
lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb

Overview

Identifies arrays that look like collections of article hashes.

Constant Summary collapse

TITLE_KEYS =

Preferred keys when extracting title-like values from state payloads.

%i[title headline name text].freeze
URL_KEYS =

Preferred keys when extracting URL-like values from state payloads.

%i[url link href permalink slug path canonicalUrl shortUrl].freeze

Class Method Summary collapse

Class Method Details

.array_of_articles?(array) ⇒ Boolean

Returns whether array includes hash entries with title and URL fields.

Parameters:

  • array (Array<Object>) —

    candidate list of entries

Returns:

  • (Boolean) —

    whether array includes hash entries with title and URL fields



40
41
42
43
44
45
46
 
# File 'lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb', line 40

def array_of_articles?(array)
  array.any? do |element|
    next unless element.is_a?(Hash)

    title_from(element) && url_from(element)
  end
end

.candidate_array?(document) ⇒ Boolean

Returns whether the node contains article-like arrays.

Parameters:

  • document (Hash, Array, Object) —

    candidate document node

Returns:

  • (Boolean) —

    whether the node contains article-like arrays



18
19
20
21
22
23
24
25
26
27
 
# File 'lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb', line 18

def candidate_array?(document)
  case document
  when Array
    return true if array_of_articles?(document)

    document.any? { traversable_candidate?(_1) }
  when Hash then document.each_value.any? { candidate_array?(_1) }
  else false
  end
end

.title_from(object) ⇒ Object?

Returns detected title-like value.

Parameters:

  • object (Hash) —

    article candidate object

Returns:

  • (Object, nil) —

    detected title-like value



50
51
52
 
# File 'lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb', line 50

def title_from(object)
  ValueFinder.fetch(object, TITLE_KEYS)
end

.traversable_candidate?(value) ⇒ Boolean

Returns whether nested value should be traversed for article candidates.

Parameters:

  • value (Hash, Array, Object) —

    candidate nested value

Returns:

  • (Boolean) —

    whether nested value should be traversed for article candidates



31
32
33
34
35
36
 
# File 'lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb', line 31

def traversable_candidate?(value)
  case value
  when Array, Hash then candidate_array?(value)
  else false
  end
end

.url_from(object) ⇒ Object?

Returns detected URL-like value.

Parameters:

  • object (Hash) —

    article candidate object

Returns:

  • (Object, nil) —

    detected URL-like value



56
57
58
 
# File 'lib/html2rss/auto_source/scraper/json_state/candidate_detector.rb', line 56

def url_from(object)
  ValueFinder.fetch(object, URL_KEYS)
end