NxtHttpClient

Build http clients with ease. NxtHttpClient is a DSL on top of the typhoeus gem. NxtHttpClient provides configuration functionality to set up HTTP connections on the class level, and attach callbacks that allow you to seamlessly handle responses, as well as configure the original Typhoeus::Request before making a request.

Installation

Add this line to your application's Gemfile:

gem 'nxt_http_client'

And then execute:

bundle

Usage

With NxtHttpClient, you can create client classes for interacting with external services:

class UserServiceClient < NxtHttpClient::Client
  # Set a base URL, and any other request options you need
  configure do |config|
    config.base_url = 'www.example.com'
    config.request_options.deep_merge!(
      headers: { API_KEY: '1993' },
      followlocation: true
    )
    config.json_request = true
    config.raise_response_errors = true
    config.x_request_id_proc = -> { ('a'..'z').to_a.shuffle.take(10).join }
  end

  # You may add a log handler if you wish...
  log do |info|
    Rails.logger.info(info)
  end

  # ...as well as a response handler
  response_handler do |handler|
    # Note: This error handler is set by default when you use 
    # config.raise_response_errors = true
    handler.on(:error) do |response|
      Sentry.set_context('http_error', error.to_h)
      raise StandardError, "I can't handle this: #{response.code}"
    end
  end
end

and then child classes for accessing specific endpoints and adding custom behaviours.

class UserFetcher < UserServiceClient
  def initialize(id)
    @url = ".../users/#{id}"
  end

  def fetch_email
    get(url, { fields: :email }) do |response_handler|
      response_handler.on(:success) do |response|
        JSON(response.body)['email']
      end
    end
  end

  def fetch_user_details
    get(url) do |response_handler|
      response_handler.on(:success) do |response|
        body = JSON(response.body)
        User.new(body)
      end
    end
  end

  private attr_reader :url
end

Usage:

client = UserFetcher.new('1234')
client.fetch_email
client.fetch_user_details

However, if you need a simple ad hoc client for a one-off task, you can use .make to instantiate one.

client = NxtHttpClient::Client.make do 
  configure do |config|
    config.base_url = 'www.httpstat.us'
    config.request_options.deep_merge!(
      headers: { API_KEY: '1993' },
      followlocation: true
    )
    config.json_request = true
    config.json_response = true
  end
end

client.get('/data')
client.post('/data', body: { some: 'content'})

configure

Register your default request options on the class level. Available options are:

• request_options, passed directly to the underlying Typhoeus Request
• base_url=
• x_request_id_proc=
• json_request=: Shorthand to set the Content-Type request header to JSON and automatically convert request bodies to JSON
• json_response=: Shorthand to set the Accept request header and automatically convert success response bodies to JSON (an empty/204 body becomes nil)
• raise_response_errors=: Makes the client raise a generic NxtHttpClient::Error for a non-success response. Superseded by raise_error_taxonomy (which raises typed errors and takes precedence when both are set); kept for backward compatibility.
• raise_error_taxonomy=: Defaults to false. Opt in to raise the mapped NxtHttpClient::Error taxonomy (ClientError/ServerError/NetworkError subclasses) on an unhandled 4xx/5xx/code-0 response instead of returning it. See Error taxonomy.
• bearer_auth=: Set a bearer token to be sent in the Authorization header
• basic_auth=: Pass a Hash containing :username and :password, to be sent as Basic credentials in the Authorization header
• timeout_seconds(total:, connect: nil): Configure timeouts

response_handler

Register a default response handler for your client class. You can reconfigure or overwrite this in subclasses and on the instance level.

fire

All http methods internally are delegate to fire(uri, **request_options). Since fire is a public method you can also use it to fire your requests and use the response handler to register callbacks for specific responses.

Registered callbacks have a hierarchy by which they are executed. Specific callbacks will come first and more common callbacks will come later in case none of the specific callbacks matched. It this is not what you want you can simply put the logic you need into one common callback that is called in any case. You can also use strings with wildcards to match a group of response by status code. handler.on('4**') { ... } basically would match all client errors.

fire('uri', **request_options) do |handler|
  handler.on(:any) do |response|
    raise StandardError, 'This would overwrite all others since it matches first'
  end

  handler.on(:success) do |response|
    response.body
  end

  handler.on(:timed_out) do |response|
    raise StandardError, 'Timeout'
  end

  handler.on(:error) do |response|
    raise StandardError, 'This is bad'
  end

  handler.on(:others) do |response|
    raise StandardError, 'Other problem'
  end

  handler.on(:headers) do |response|
    # This is already executed when the headers are received
  end

  handler.on(:body) do |chunk|
   # Use this to stream the body in chunks
  end
end

Callbacks around fire

Next to implementing callbacks for handling responses there are also callbacks around making requests. Note tht you can have as many callbacks as you want. In case you need to reset them because you do not want to inherit them from your parent class (might be a smell when you need to...) you can reset callbacks via clear_fire_callbacks on the class level.

clear_fire_callbacks # Call this to clear callbacks setup in the parent class

before_fire do |client, request, response_handler|
  # here you have access to the client, request and response_handler  
end

around_fire do |client, request, response_handler, fire|
  # here you have access to the client, request and response_handler
  fire.call # You have to call fire here and return the result to the next callback in the chain
end

after_fire do |client, request, response, result, error|
  result # The result of the last callback in the chain is the result of fire!
end

NxtHttpClient::Error

NxtHttpClient also provides an error base class that you might want to use as the base for your client errors. It comes with a nice set of useful methods. You can ask the error for the request and response options since it requires the response for initialization. Furthermore it has a handy to_h method that provides you all info about the request and response.

Timeouts

You must set a timeout on every request (or when configuring the client class). Otherwise, this gem will raise an error. The idea is to enforce the best practice of always setting a timeout.

To set a timeout, use the timeout_seconds config method:

configure do |config|
  config.timeout_seconds(total: 10)
  # You can also set a connect timeout
  config.timeout_seconds(total: 10, connect: 2)
end

NxtHttpClient::Error exposes the timed_out? method from Typhoeus::Response, so you can check if an error is raised due to a timeout. This is useful when setting a custom timeout value in your configuration.

Error taxonomy

Set config.raise_error_taxonomy = true and the client raises a typed subclass of NxtHttpClient::Error for an unhandled 4xx, 5xx, or code-0 (network) response, so you no longer need to hand-roll per-status on(400)/on(422)/on(5xx)/on(0) handlers just to get a usable taxonomy. (3xx and other codes are returned as before.) It is off by default (the client returns the response); all classes inherit from NxtHttpClient::Error, so existing rescue NxtHttpClient::Error handlers keep working.

HTTP status:

status	class	retryable?
400	NxtHttpClient::Error::BadRequest	no
401	NxtHttpClient::Error::Unauthorized	no
403	NxtHttpClient::Error::Forbidden	no
404	NxtHttpClient::Error::NotFound	no
422	NxtHttpClient::Error::UnprocessableEntity	no
429	NxtHttpClient::Error::TooManyRequests	up to you
other 4xx	NxtHttpClient::Error::ClientError	no
5xx	NxtHttpClient::Error::ServerError	yes

Network / code 0. Typhoeus/libcurl surfaces network failures and timeouts as a response with HTTP code 0 (no response received); the real cause lives in libcurl's return_code:

libcurl return_code	class	retryable?
:operation_timedout	NxtHttpClient::Error::Timeout	yes
:couldnt_connect	NxtHttpClient::Error::ConnectionFailed	yes
:couldnt_resolve_host / :couldnt_resolve_proxy	NxtHttpClient::Error::NameResolutionError	yes
:ssl_connect_error and other non-cert :ssl_*	NxtHttpClient::Error::TlsError	yes
any other code-0	NxtHttpClient::Error::NetworkError	yes
cert verification (:peer_failed_verification, …)	NxtHttpClient::Error::CertificateError	no

Retrying

The retryable errors share two base classes, so a job retries them in one place:

retry_on NxtHttpClient::Error::NetworkError, NxtHttpClient::Error::ServerError

ClientError (4xx) is not retried — those are caller mistakes. CertificateError is a sibling of NetworkError (not a child), so it is excluded from retry_on NetworkError — a failed cert/CA verification is permanent. TooManyRequests (429) is left out of the retryable set; add your own retry_on NxtHttpClient::Error::TooManyRequests (ideally honoring Retry-After) if you want it.

Precedence and opting out

A consumer's own on(<code>) / on(:error) / on(:timed_out) callback always takes precedence; the raise only fires when nothing else handled the response. So you can enable the taxonomy for retries yet still handle, say, a 404 inline with your own on(404).

Domain-typed errors

Map a status to your own error class with map_error to get a domain error that parses the response body. It is inherited by subclasses and overrides the default for that status. map_error only takes effect when raise_error_taxonomy is enabled — it customizes the taxonomy, it does not enable raising on its own:

class MyService::Client < NxtHttpClient::Client
  configure { |config| config.raise_error_taxonomy = true }
  map_error 422, MyService::Error::ValidationFailed
end

class MyService::Error::ValidationFailed < NxtHttpClient::Error
  def default_message
    body.dig('errors', 0, 'detail') # `body` parses JSON response bodies for you
  end
end

Logging

NxtHttpClient also comes with a log method on the class level that you can pass a proc if you want to log your request. Your proc needs to accept an argument in order to get access to information about the request and response made.

log do |info|
  Rails.logger.info(info)
end

# info is a hash that is implemented as follows:

{
  client: client,
  started_at: started_at,
  request: request,
  finished_at: now,
  elapsed_time_in_milliseconds: finished_at - started_at,
  response: request.response,
  http_status: request.response&.code
}

Caching

Typhoeus ships with caching built in. Checkout the typhoeus docu to figure out how to set it up. NxtHttpClient builds some functionality on top of this and offer to cache requests within the current thread or globally. You can simply make use of it by providing one of the caching options :thread or:global as config request option or the actual request options when building the request.

class Client < NxtHttpClient::Client
  configure do |config|
    config.request_options = { cache: :thread }
  end

  response_handler do |handler|
    handler.on(200) do |response|
      # ...
    end
  end

  def call
    get('.../url.com', cache: :thread) # configure caching per request level
  end
end

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install.

Releasing

RubyGems

First, if you don't want to always log in with your RubyGems password, you can create an API key on Rubygems.org, and then run:

bundle config set --local gem.push_key rubygems

Add to ~/.gem/credentials (create if it doesn't exist):

:rubygems: <your Rubygems API key>

To release a new version follow the steps strictly:

• Commit all your feature changes
• Update the version number in version.rb,
• Run bundle install to update the Gemfile.lock
• Open your PR and get it approved and merged
• And then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Github Package Registry

To release a new version follow the steps strictly:

• Commit all your feature changes
• Update the version number in version.rb,
• Run bundle install to update the Gemfile.lock
• Open your PR and get it approved and merged
• Checkout to main and then run bin/release, which will create a git tag for the version, push git commits and tags, and push the .gem file to the github package registry.

Before releasing a new version, make sure you have authenticated with the Github package registry. To do so, create a personal access token (in your Github account settings)

Then create or add to the existing file ~/.gem/credentials, replacing TOKEN with your personal access token.

---
:github: Bearer TOKEN

Then run

bundle config set --local gem.push_key github

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/nxt-insurance/nxt_http_client.

License

The gem is available as open source under the terms of the MIT License.