Module: Usps::Support::HealthDiagnostics

Defined in:
lib/usps/support/health_diagnostics.rb

Overview

Fire-and-forget publisher of per-component health-check results to CloudWatch Logs in Embedded Metric Format (EMF). CloudWatch extracts a per-(App, Component) metric from each line while the raw JSON — status, latency, error — stays queryable in Logs Insights. See the infrastructure repo's doc/health_component_metrics.md.

Why: /health returns a single 503 when ANY dependency fails, and Route 53's last-failure-reason keeps only the most recent probe, so once the dependency recovers we can no longer tell WHICH component failed (e.g. the Exams/Merit Marks flaps, where hq_database and imis are indistinct from AWS telemetry). This records each component every probe, so a past incident is attributable.

Disabled unless HEALTH_DIAGNOSTICS_LOG_GROUP is set, so dev/test and unprovisioned apps are no-ops. Delivery runs on a single background thread off the request path; anything that goes wrong (queue full, SDK missing, API error) drops the record. Diagnostics must never slow or fail /health — that endpoint is exactly what stalls during an incident.

Constant Summary collapse

NAMESPACE = 
'USPS/HealthCheck'
MAX_QUEUE = 
1_000

Class Method Summary collapse

Class Method Details

.emf_events(app:, components:, timestamp_ms:) ⇒ Object

Builds the EMF log events for one probe. Pure — the background worker just ships these.



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
 
# File 'lib/usps/support/health_diagnostics.rb', line 41

def emf_events(app:, components:, timestamp_ms:)
  components.map do |component|
    message = {
      '_aws' => {
        'Timestamp' => timestamp_ms,
        'CloudWatchMetrics' => [{
          'Namespace' => NAMESPACE,
          'Dimensions' => [%w[App Component]],
          'Metrics' => [
            { 'Name' => 'Unhealthy', 'Unit' => 'Count' },
            { 'Name' => 'LatencyMs', 'Unit' => 'Milliseconds' }
          ]
        }]
      },
      'App' => app,
      'Component' => component[:label],
      'Unhealthy' => component[:ok] ? 0 : 1,
      'LatencyMs' => component[:latency_ms],
      'status' => component[:ok] ? 'ok' : 'fail'
    }
    message['error'] = component[:error] if component[:error]
    { timestamp: timestamp_ms, message: message.to_json }
  end
end

.enabled? ⇒ Boolean

Returns:

  • (Boolean) 


36
 
# File 'lib/usps/support/health_diagnostics.rb', line 36

def enabled? = !log_group.nil?

.log_group ⇒ Object 



38
 
# File 'lib/usps/support/health_diagnostics.rb', line 38

def log_group = ENV.fetch('HEALTH_DIAGNOSTICS_LOG_GROUP', nil)

.record(app:, components:) ⇒ Object

Enqueue one probe's component results for delivery. Non-blocking; returns nil. components: [{ label:, ok:, latency_ms:, error: }, ...]



29
30
31
32
33
34
 
# File 'lib/usps/support/health_diagnostics.rb', line 29

def record(app:, components:)
  return unless enabled?

  enqueue(app: app, components: components, timestamp_ms: now_ms)
  nil
end