Module: Ucode::Fetch::Http

Defined in:
lib/ucode/fetch/http.rb

Overview

Shared HTTP wrapper. Single network boundary for the whole project.

Streaming download with retries and exponential backoff. Raises Ucode::NetworkError on final failure (after http_retries attempts).

Defined Under Namespace

Classes: ValidationFailure

Class Method Summary collapse

Class Method Details

.get(url, dest:, retries: nil, timeout: nil, validate: nil) ⇒ Pathname

Stream url to dest (a Pathname or String path).

Parameters:

  • url (String, URI) —

    full URL.

  • dest (Pathname, String) —

    destination file path. Parent directory is created if absent.

  • retries (Integer, nil) (defaults to: nil) —

    override Config.http_retries.

  • timeout (Integer, nil) (defaults to: nil) —

    override Config.http_timeout.

  • validate (Symbol, nil) (defaults to: nil) —

    when :pdf, after a successful download verify (a) Content-Type starts with application/pdf and (b) the first 4 bytes of the body are %PDF. Raises CodeChartNotFoundError with the offending header value in context: on failure. nil = no validation (the default for non-PDF callers like UcdZip and UnihanZip).

Returns:

  • (Pathname) —

    destination path on success.

Raises:



36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/ucode/fetch/http.rb', line 36

def get(url, dest:, retries: nil, timeout: nil, validate: nil)
  uri = url.is_a?(URI) ? url : URI(url)
  destination = Pathname.new(dest)
  destination.dirname.mkpath

  attempts = retries || Ucode.configuration.http_retries
  read_timeout = timeout || Ucode.configuration.http_timeout
  backoff_sequence = DEFAULT_BACKOFF.take(attempts + 1)

  last_error = nil
  (attempts + 1).times do |attempt|
    begin
      response = stream_to(uri, destination, read_timeout)
      validate_response!(validate, response, destination) if validate
      return destination
    rescue ValidationFailure => e
      raise e.cause
    rescue StandardError => e
      last_error = e
      sleep_for = backoff_sequence[attempt] || backoff_sequence.last
      Ucode.configuration.logger&.warn do
        "Http GET #{uri} failed (attempt #{attempt + 1}/#{attempts + 1}): " \
          "#{e.class}: #{e.message}; retrying in #{sleep_for}s"
      end
      sleep(sleep_for)
    end
  end

  raise Ucode::NetworkError.new(
    "GET #{uri} failed after #{attempts + 1} attempts",
    context: { url: uri.to_s, last_error: last_error&.message },
  )
end