Class: Usps::Support::HealthController

Inherits:
ActionController::Base
  • Object
show all
Defined in:
app/controllers/usps/support/health_controller.rb

Overview

Base controller for external synthetic health probes (see the infrastructure repo's doc/application_health_probes.md). Unlike Rails' /up (boot-only) and nginx's static /instance-health-check, this exercises an app's critical dependencies and returns the sentinel "HEALTHCHECK_OK " only when every check passes, so a Route 53 HTTPS_STR_MATCH probe asserting that prefix flips the Statuspage component even with no organic traffic.

Host apps subclass this and declare which dependencies to probe with a CHECKS constant — a list of symbols, each naming a #check_ method:

class HealthController < Usps::Support::HealthController
CHECKS = %i[database redis hq_database imis].freeze
end

and route to it (get 'health' => 'health#show'). For a bespoke dependency, define a #check_ in the subclass (e.g. wrapping #check_http(url)) and add to CHECKS.

Responds 503 when any check fails, EXCEPT to callers that pass ?probe=1, which always get 200 and are expected to judge health by the sentinel in the body. That exists so a failing dependency does not register as an application 5XX on the load balancer -- see #response_status. Point external string-matching probes at /health?probe=1; leave it off everywhere else.

Checks run on a background thread, not the request thread: a per-process refresher updates a cached snapshot every health_refresh_interval seconds and #show serves that snapshot, so a slow or wedged dependency can't tie up the Passenger worker pool. Each check also runs under health_check_timeout, and a raise or timeout reads as a failed check (its error captured) so the action always responds. See doc/health_check_refresh_design.md.

Per-component results (ok, latency, error) are handed to Usps::Support::HealthDiagnostics once per refresh which — when enabled — publishes them to CloudWatch so a past failure can be attributed to a specific component after it has recovered.

Inherits ActionController::Base directly — NOT the host's ApplicationController — to bypass Devise auth, Pundit authorization, PaperTrail, and especially allow_browser, whose modern-browser gate would 406 Route 53's non-browser prober and fail the probe for the wrong reason.

Defined Under Namespace

Classes: CheckTimeout

Constant Summary collapse

SENTINEL_PREFIX =

rubocop:disable Rails/ApplicationController

'HEALTHCHECK_OK'
CHECKS =

Symbols naming the dependency checks to run; each invokes the matching #check_. Override per app by defining a CHECKS constant. Default probes the primary database only, which presumably every app needs.

%i[database].freeze

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.dependency_status ⇒ Object

Read-only, non-blocking view of the last completed round, for callers on a user request thread — specifically Usps::Support::DependencyGate, which uses it to fail a request fast rather than discovering the outage by blocking on the dead dependency itself:

{ hq_database: { ok: false, consecutive_failures: 4 }, database: { ok: true, ... } }

Runs no check and takes no lock; it reads whatever the background refresher last stored. Returns {} — "no opinion", so callers fail open — when no round has completed yet (cold worker) or the snapshot has aged past health_status_max_age, which means the refresher itself is wedged and its last verdict is no longer evidence about the dependency.



90
91
92
93
94
95
 
# File 'app/controllers/usps/support/health_controller.rb', line 90

def dependency_status
  snapshot = health_monitor.cached
  return {} unless snapshot && health_monitor.age_of(snapshot) <= health_status_max_age

  snapshot.data&.[](:status) || {}
end

.health_monitor ⇒ Object

One monitor per concrete subclass, per process. Built lazily so the background thread starts after Passenger forks its workers (threads do not survive fork).



70
71
72
73
74
75
76
77
78
 
# File 'app/controllers/usps/support/health_controller.rb', line 70

def health_monitor
  klass = self
  @health_monitor ||= HealthMonitor.new(
    interval: health_refresh_interval,
    max_age: health_snapshot_max_age,
    background: health_background_refresh,
    runner: -> { klass.new.send(:run_and_record) }
  )
end

.record_streaks(results) ⇒ Object

Consecutive failed rounds per check, updated once per refresh so a gate can require a sustained failure and ignore a single slow round. Called only from the monitor's runner, and the monitor serialises refreshes under its own mutex, so this needs no lock of its own.



100
101
102
103
104
 
# File 'app/controllers/usps/support/health_controller.rb', line 100

def record_streaks(results)
  @streaks ||= Hash.new(0)
  results.each { |name, ok| ok ? @streaks[name] = 0 : @streaks[name] += 1 }
  results.to_h { |name, ok| [name, { ok:, consecutive_failures: @streaks[name] }] }
end

.reset_health_monitor! ⇒ Object

Drops the memoized monitor (stopping its thread) and the failure streaks. For tests.



107
108
109
110
111
 
# File 'app/controllers/usps/support/health_controller.rb', line 107

def reset_health_monitor!
  @health_monitor&.stop
  @health_monitor = nil
  @streaks = nil
end

Instance Method Details

#show ⇒ Object 



114
115
116
117
118
119
 
# File 'app/controllers/usps/support/health_controller.rb', line 114

def show
  data = self.class.health_monitor.current
  return render_unavailable unless data

  render_health(data)
end