Class: Html2rss::AutoSource::Scraper::Schema::Thing

Inherits:
Object
  • Object
show all
Defined in:
lib/html2rss/auto_source/scraper/schema/thing.rb

Overview

A Thing is kind of the 'base class' for Schema.org schema_objects.

See Also:

Direct Known Subclasses

ItemList

Constant Summary collapse

SUPPORTED_TYPES =

Supported Schema.org @type values mapped to article extraction.

%w[
  AdvertiserContentArticle AnalysisNewsArticle APIReference Article
  AskPublicNewsArticle BackgroundNewsArticle BlogPosting DiscussionForumPosting
  LiveBlogPosting NewsArticle OpinionNewsArticle Report ReportageNewsArticle
  ReviewNewsArticle SatiricalArticle ScholarlyArticle SocialMediaPosting TechArticle
].to_set.freeze
DEFAULT_ATTRIBUTES =

Attributes exposed by #call in generated article hashes.

%i[id title description author url image published_at categories].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(schema_object, url:) ⇒ Thing

Returns a new instance of Thing.

Parameters:

  • schema_object (Hash{Symbol => Object}) —

    parsed schema.org object

  • url (String, Html2rss::Url, nil) —

    base URL used for relative normalization



27
28
29
30
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 27

def initialize(schema_object, url:)
  @schema_object = schema_object
  @base_url = normalized_base_url(url)
end

Instance Attribute Details

#base_url ⇒ Object (readonly)

Returns the value of attribute base_url.



93
94
95
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 93

def base_url
  @base_url
end

#schema_object ⇒ Object (readonly)

Returns the value of attribute schema_object.



93
94
95
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 93

def schema_object
  @schema_object
end

Instance Method Details

#author ⇒ String?

Returns extracted author or publisher name.

Returns:

  • (String, nil) —

    extracted author or publisher name



57
58
59
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 57

def author
  extract_author_name(schema_object[:author]) || extract_author_name(schema_object[:publisher])
end

#call ⇒ Hash

Returns the scraped article hash with DEFAULT_ATTRIBUTES.

Returns:

  • (Hash) —

    the scraped article hash with DEFAULT_ATTRIBUTES



33
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 33

def call = DEFAULT_ATTRIBUTES.to_h { [_1, public_send(_1)] }

#categories ⇒ Array<String>?

Returns extracted category labels.

Returns:

  • (Array<String>, nil) —

    extracted category labels



89
90
91
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 89

def categories
  @categories ||= CategoryExtractor.call(schema_object)
end

#description ⇒ String?

Returns longest available description field.

Returns:

  • (String, nil) —

    longest available description field



52
53
54
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 52

def description
  schema_object.values_at(:description, :articleBody, :abstract, :schema_object_body).max_by { _1.to_s.size }
end

#id ⇒ String?

Returns stable schema object identifier.

Returns:

  • (String, nil) —

    stable schema object identifier



36
37
38
39
40
41
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 36

def id
  return @id if defined?(@id)

  id = normalized_id(schema_object[:@id], reference_url: url || base_url) || url&.path.to_s
  @id = id.to_s.empty? ? nil : id
end

#image ⇒ Html2rss::Url?

Returns normalized article image URL.

Returns:



75
76
77
78
79
80
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 75

def image
  return @image if defined?(@image)

  img_url = image_urls.first
  @image = img_url ? Url.from_relative(img_url, base_url || img_url) : nil
end

#image_urls ⇒ Array<String>

Returns normalized image URL candidates.

Returns:

  • (Array<String>) —

    normalized image URL candidates



96
97
98
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 96

def image_urls
  @image_urls ||= schema_object.values_at(:image, :thumbnailUrl).filter_map { image_url_from(_1) }
end

#published_at ⇒ String?

Returns published-at timestamp (datePublished, else dateModified).

Returns:

  • (String, nil) —

    published-at timestamp (datePublished, else dateModified)



83
84
85
86
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 83

def published_at
  value = schema_object[:datePublished] || schema_object[:dateModified]
  value.to_s.empty? ? nil : value
end

#title ⇒ String?

Returns article title (schema.org: headline → name → alternativeHeadline; legacy title).

Returns:

  • (String, nil) —

    article title (schema.org: headline → name → alternativeHeadline; legacy title)



44
45
46
47
48
49
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 44

def title
  schema_object.values_at(:headline, :name, :alternativeHeadline, :title)
               .lazy
               .map { _1.to_s.strip }
               .find { !_1.empty? }
end

#url ⇒ Html2rss::Url?

Returns the URL of the schema object.

Returns:



62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/html2rss/auto_source/scraper/schema/thing.rb', line 62

def url
  return @url if defined?(@url)

  url = schema_object[:url]
  if url.to_s.empty?
    Log.debug("Schema#Thing.url: no url in schema_object: #{schema_object.inspect}")
    return @url = nil
  end

  @url = Url.from_relative(url, base_url || url)
end