Module: RailsPodKit::GlobalScheduler::Heartbeat

Defined in:
lib/rails_pod_kit/global_scheduler/heartbeat.rb

Overview

The one failure the rest of the machinery cannot see.

A poller thread that dies is restarted by the Supervisor, and a process that stops serving is caught by the exporter's own probe. But a poller that is running and no longer enqueueing is indistinguishable from an idle one from the outside: every gauge stays fresh, /metrics answers 200, the pod is Running and Ready. On the only process carrying the schedule that is a silently stopped schedule — exactly the failure hosting the poller here was meant to eliminate, coming back through another door.

So publish the age of the last completed tick, as sidekiq_cron_poll_age_seconds. It measures the loop turning, which means it stays flat on a healthy but idle schedule and climbs the moment ticks stop — the one shape an alert can be written against.

Deliberately not "time since last enqueue": that climbs on any quiet schedule, so it would alert on nothing happening. Answering "should this job have run by now?" needs a per-job check against the cron expression, not a gauge.

Class Method Summary collapse

Class Method Details

.age ⇒ Object

nil until started, which is what keeps the series off any process that hosts no poller.



75
76
77
78
79
80
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 75

def age
  reference = @last_poll_at || @started_at
  return nil unless reference

  monotonic_now - reference
end

.declare! ⇒ Object 



38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 38

def declare!
  Yabeda.configure do
    group :sidekiq do
      gauge :cron_poll_age,
            unit: :seconds,
            tags: [],
            aggregation: :most_recent,
            comment: 'Seconds since the sidekiq-cron poller last completed a tick'

      collect do
        age = RailsPodKit::GlobalScheduler::Heartbeat.age
        Yabeda.sidekiq.cron_poll_age.set({}, age) if age
      end
    end
  end
end

.install! ⇒ Object

Declares the gauge. One-shot, and safe either side of Yabeda.configure! — a metric declared after it is registered with the adapters immediately.



30
31
32
33
34
35
36
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 30

def install!
  return false if @installed

  require 'yabeda'
  declare!
  @installed = true
end

.monotonic_now ⇒ Object

Monotonic: this is a duration, and a wall-clock step (NTP, a node coming back from suspend) must not read as the schedule having stalled.



84
85
86
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 84

def monotonic_now
  ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
end

.record! ⇒ Object 



69
70
71
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 69

def record!
  @last_poll_at = monotonic_now
end

.start! ⇒ Object

Begins measuring, from before the first tick — so a poller that never manages one reads as climbing rather than as no-data.



57
58
59
60
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 57

def start!
  @started_at = monotonic_now
  @last_poll_at = nil
end

.stop! ⇒ Object

Drops the series with the poller: a stopped scheduler should read as no-data, not as an age climbing forever.



64
65
66
67
 
# File 'lib/rails_pod_kit/global_scheduler/heartbeat.rb', line 64

def stop!
  @started_at = nil
  @last_poll_at = nil
end