Module: Usps::Support::DependencyGate

Extended by:

ActiveSupport::Concern

Defined in:

app/controllers/concerns/usps/support/dependency_gate.rb

Overview

Fails requests fast, with the branded 503 page, while a dependency the app cannot work without is known to be down — instead of letting every request discover that for itself by blocking on the dead dependency.

Why

The HQ member database is in the request path of every authenticated request: authenticate_user_from_jwt! resolves the JWT's certificate through Members::Member.find. While HQ is unreachable each of those requests blocks until the database client's connect timeout expires, then raises a connection error that ApplicationController rescues into the 503 page. The page is correct; paying a connect timeout to produce it is what hurts. The web tier serves a small, fixed number of requests concurrently, so a per-request stall of seconds is enough to occupy every worker: requests queue, nginx gives up on the upstream, and an app that is up and answering correctly reads to users as hard down.

This gate closes that gap: the app already knows HQ is down, because HealthController's monitor probes it on a background thread and leaves the verdict in memory (see health_check_refresh_design.md). Consulting that costs no I/O and no worker time, so the request returns the same 503 immediately instead of after a timeout.

Contract

• /health is untouched. The gate is a reader of the snapshot; it never runs a check, never changes what /health probes, renders, or reports to CloudWatch. The synthetic probes and Statuspage components keep behaving exactly as they do today — the app's own gating must never become the thing that decides whether we call ourselves up.
• Never blocks. It reads the last completed snapshot only (HealthMonitor#cached), so a process that has not yet run a round serves the request normally rather than paying for a round on the request thread.
• Fails open. No snapshot, a stale one (a wedged refresher), or any error reading it means "no opinion" and the request proceeds — the pre-existing rescue_from for the connection errors is still the backstop, so failing open costs latency, never correctness.
• Requires a sustained failure. Gating opens only after dependency_gate_threshold consecutive failed rounds, so a single slow round — one that merely overran its check budget — can't blackhole a healthy app. It closes again on the first passing round, with no deploy or restart needed.

Wiring

class ApplicationController < ActionController::Base
include Usps::Support::DependencyGate
self.dependency_gate_title = 'Exams System'
end

gated_dependencies defaults to %i[hq_database] — the only check that is in the user request path. iMIS deliberately is not: it is an external service reached outside the VPN and is touched only by rescued admin reports and Sidekiq jobs, so an iMIS outage must not gate the app.

Actions that genuinely do not need the gated dependency opt out the same way they opt out of authentication, and for the same reason — keeping work that still functions during an HQ outage (e.g. the unauthenticated online-examination flow, which lives entirely in RDS) working:

skip_before_action :require_gated_dependencies!, only: %i[exam submit_response]

Class Method Summary

• .log_transition(down) ⇒ Object

  Logs gate open/close transitions once per process rather than once per request: during an outage every request is gated, and a per-request line would bury everything else in the log.

• .reset_transition_log! ⇒ Object

  Forgets the last logged state.

Class Method Details

.log_transition(down) ⇒ Object

Logs gate open/close transitions once per process rather than once per request: during an outage every request is gated, and a per-request line would bury everything else in the log. A race between threads can duplicate a line; that is cheaper than a lock on the request path.

# File 'app/controllers/concerns/usps/support/dependency_gate.rb', line 84

def self.log_transition(down)
  return if @down == down

  was_open = @down.present?
  @down = down

  if down.any?
    ::Rails.logger.warn("[dependency-gate] open: #{down.join(', ')} down, serving 503 until recovered")
  elsif was_open
    ::Rails.logger.warn('[dependency-gate] closed: gated dependencies recovered, serving normally')
  end
  # A process whose first request is already healthy has nothing to announce.
end

.reset_transition_log! ⇒ Object

Forgets the last logged state. For tests.

# File 'app/controllers/concerns/usps/support/dependency_gate.rb', line 99

def self.reset_transition_log! = @down = nil