Module: Automatic::Http

Defined in:
lib/automatic/http.rb

Constant Summary collapse

SCHEMES =

A link in a pipeline item comes from a feed, which is to say from outside. URI.open on such a string will happily read file:///etc/passwd or run an FTP session; a plugin fetching an article body wants neither.

%w[http https].freeze
OPEN_TIMEOUT = 
10
READ_TIMEOUT = 
30
REDIRECT_LIMIT =

open-uri follows redirects itself and refuses an HTTPS-to-HTTP downgrade. This bounds the chain so that a redirect loop ends as an error rather than as an unattended run that never returns.

5
USER_AGENT = 
"Automatic Ruby/#{Automatic::VERSION} " \
'(+https://github.com/id774/automaticruby)'
ESCAPER =

The RFC 2396 parser is named directly: URI.escape was removed in Ruby 3.0, and URI::Parser became the RFC 3986 parser in 3.4, which reports #escape as obsolete. This spelling means the same thing on every supported Ruby.

URI::RFC2396_Parser.new

Class Method Summary collapse

Class Method Details

.fetchable?(url) ⇒ Boolean

Whether a string is a URL this framework will fetch. For a plugin that skips an item rather than failing the run on one.

Returns:

  • (Boolean) 


91
92
93
94
95
96
 
# File 'lib/automatic/http.rb', line 91

def fetchable?(url)
  uri(url)
  true
rescue ArgumentError, URI::InvalidURIError
  false
end

.open(url, &block) ⇒ Object

Fetch a URL and yield the IO, for a caller that would rather stream than hold the whole body.



59
60
61
62
63
64
65
66
67
68
 
# File 'lib/automatic/http.rb', line 59

def open(url, &block)
  uri(url).open(
    'User-Agent' => USER_AGENT,
    open_timeout: OPEN_TIMEOUT,
    read_timeout: READ_TIMEOUT,
    redirect: true,
    max_redirects: REDIRECT_LIMIT,
    &block
  )
end

.read(url) ⇒ Object

Fetch a URL and return its body as a string. Raises rather than returning nil: a plugin's retry handling is built around an exception, and a body that could not be fetched is not an empty body.



53
54
55
 
# File 'lib/automatic/http.rb', line 53

def read(url)
  open(url, &:read)
end

.uri(url) ⇒ Object

Parse a URL into a URI this framework will fetch, or raise. A string carrying characters a URI may not (a space, a Japanese query term) is escaped and parsed again, which is what the plugins used to do for themselves before every call.

Raises:

  • (ArgumentError) 


74
75
76
77
78
79
80
81
82
83
84
85
86
87
 
# File 'lib/automatic/http.rb', line 74

def uri(url)
  string = url.to_s.strip
  raise ArgumentError, 'no URL to fetch' if string.empty?

  parsed = parse(string)
  unless SCHEMES.include?(parsed.scheme)
    raise ArgumentError, "not an HTTP or HTTPS URL: #{string}"
  end
  unless parsed.host
    raise ArgumentError, "HTTP or HTTPS URL has no host: #{string}"
  end

  parsed.normalize
end