Module: RailsPodKit::SolidQueue::Metrics

Defined in:
lib/rails_pod_kit/solid_queue/metrics.rb

Overview

DB-backed queue gauges for SolidQueue, in the shape yabeda-sidekiq exposes for Redis: SolidQueue ships no metrics endpoint and there is no yabeda-solid_queue.

Two series, both per queue, both computed at scrape time from a yabeda collect block — no background thread, no cached snapshot:

solid_queue_backlog          how many jobs could be claimed right now
solid_queue_latency_seconds  how long the oldest of them has been waiting

"Claimable right now" is ready executions plus scheduled ones whose time has come (the dispatcher has yet to move them across). Backlog alone misses a small-but-stalled queue and latency alone misses a large-but-moving one, so the pair is what dashboards, alerts and an HPA feed actually need.

::SolidQueue is the host's — the gem declares no dependency on it and nothing here loads until the host calls install!.

Constant Summary collapse

SOURCE = 
'rails_pod_kit.solid_queue_metrics'

Class Method Summary collapse

Class Method Details

.age_in_seconds(waiting_since, now) ⇒ Object 



170
171
172
173
174
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 170

def age_in_seconds(waiting_since, now)
  return 0 if waiting_since.nil?

  [(now - waiting_since).to_f, 0].max
end

.baseline_queues ⇒ Object

The queues the app is known to use, pinned by the host or discovered once per process from the jobs table (one index scan, never repeated). Discovery is best-effort by construction: that table is bounded by clear_finished_jobs_after, so a queue idle for longer than the retention window leaves no trace. Pin queues: where the zero has to be guaranteed.



166
167
168
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 166

def baseline_queues
  @baseline_queues || ::SolidQueue::Job.distinct.pluck(:queue_name).compact
end

.claimable_by_queue(now) ⇒ Object

=> { "default" => { backlog: 12, waiting_since: 



95
96
97
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 95

def claimable_by_queue(now)
  merge(ready_rows, due_rows(now))
end

.collect! ⇒ Object

Called by yabeda on every scrape.

An error is always reported, and then either swallowed or re-raised depending on who owns the endpoint. Swallowing keeps a transient DB failure from taking the Puma series down with it on a shared endpoint — at the cost of serving the last reading, which a consumer cannot tell apart from a live one. On the dedicated pod (run_exporter!) there is nothing else to protect, so failing the scrape is the honest answer: the gauges go to no-data and the scraper's own up series carries the failure.



80
81
82
83
84
85
86
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 80

def collect!
  now = ::Time.now.utc
  with_connection { publish_all(claimable_by_queue(now), now) }
rescue StandardError => e
  ErrorReporter.report(e, source: SOURCE)
  raise if @fail_scrape_on_error
end

.declare! ⇒ Object 



54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 54

def declare!
  Yabeda.configure do
    group :solid_queue do
      gauge :backlog,
            tags: %i[queue],
            comment: 'Jobs claimable right now: ready executions plus scheduled ones whose time has come'
      gauge :latency,
            unit: :seconds,
            tags: %i[queue],
            comment: 'How long the oldest claimable job has been waiting'

      collect { RailsPodKit::SolidQueue::Metrics.collect! }
    end
  end
end

.due_rows(now) ⇒ Object

A scheduled execution whose time has come is claimable too. Its wait started at scheduled_at, not created_at — a job enqueued a week ahead of its slot is not a week late.



106
107
108
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 106

def due_rows(now)
  rows(::SolidQueue::ScheduledExecution.where(scheduled_at: ..now), :scheduled_at)
end

.install!(queues: nil, fail_scrape_on_error: nil) ⇒ Object

Declares the gauges and registers the scrape-time collector. Safe before or after Yabeda.configure! (yabeda replays configurators either way).

queues: pins the zero baseline (see #baseline_queues) instead of discovering it. fail_scrape_on_error: makes a collection failure fail the whole response rather than serve the last reading — right when this collector owns the endpoint, wrong when it shares one (see #collect!).

Declaration is one-shot but the options are not: the dedicated pod boots the host's initializers before run_exporter! runs, so by the time the pod asks for fail_scrape_on_error the app's own install_metrics! has normally already declared the gauges. An option given here always wins; one left out keeps whatever an earlier call set.



43
44
45
46
47
48
49
50
51
52
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 43

def install!(queues: nil, fail_scrape_on_error: nil)
  @baseline_queues = queues unless queues.nil?
  @fail_scrape_on_error = fail_scrape_on_error unless fail_scrape_on_error.nil?

  return false if @installed

  require 'yabeda'
  declare!
  @installed = true
end

.merge(*row_sets) ⇒ Object 



120
121
122
123
124
125
126
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 120

def merge(*row_sets)
  row_sets.flatten(1).each_with_object({}) do |(queue, count, waiting_since), acc|
    entry = acc[queue] ||= { backlog: 0, waiting_since: nil }
    entry[:backlog] += count
    entry[:waiting_since] = [entry[:waiting_since], waiting_since].compact.min
  end
end

.publish(queue, backlog:, latency:) ⇒ Object 



136
137
138
139
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 136

def publish(queue, backlog:, latency:)
  Yabeda.solid_queue.backlog.set({ queue: queue }, backlog)
  Yabeda.solid_queue.latency.set({ queue: queue }, latency)
end

.publish_all(queues, now) ⇒ Object 



128
129
130
131
132
133
134
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 128

def publish_all(queues, now)
  queues.each do |queue, entry|
    publish(queue, backlog: entry[:backlog], latency: age_in_seconds(entry[:waiting_since], now))
  end

  zero_drained_queues(queues.keys)
end

.ready_rows ⇒ Object 



99
100
101
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 99

def ready_rows
  rows(::SolidQueue::ReadyExecution.all, :created_at)
end

.reset! ⇒ Object

Test/reset hook — drops the published-label-set memo and the install options.



178
179
180
181
182
183
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 178

def reset!
  @seen_queues = nil
  @baseline_queues = nil
  @fail_scrape_on_error = false
  @installed = false
end

.rows(relation, waiting_since_column) ⇒ Object

Two grouped aggregates over an indexed, normally-small table. Both go through ActiveRecord's calculations so the timestamp comes back type-cast on every adapter.



113
114
115
116
117
118
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 113

def rows(relation, waiting_since_column)
  counts = relation.group(:queue_name).count
  oldest = relation.group(:queue_name).minimum(waiting_since_column)

  counts.map { |queue, count| [queue, count, oldest[queue]] }
end

.seen_queues ⇒ Object

Seeded with the baseline, so the zeroing above also covers queues this process has never seen busy. A gauge only exists once it has been set: without the seed an exporter that boots while the queue is empty — the steady state of a scale-to-zero deployment — publishes no series at all, and every consumer reads no-data where it should read 0.



156
157
158
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 156

def seen_queues
  @seen_queues ||= baseline_queues
end

.with_connection ⇒ Object

Collection runs on the exporter's HTTP thread, which would otherwise check out a connection and pin it there for the life of the process.



90
91
92
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 90

def with_connection(&)
  ::ActiveRecord::Base.connection_pool.with_connection(&)
end

.zero_drained_queues(current) ⇒ Object

A gauge keeps its last value per label set, so a queue that just drained would stay pinned at its final backlog forever — the exact reading that would keep an alert firing on an idle system. Track the label sets this process has published and zero the ones missing from this round.



145
146
147
148
149
 
# File 'lib/rails_pod_kit/solid_queue/metrics.rb', line 145

def zero_drained_queues(current)
  seen = seen_queues
  (seen - current).each { |queue| publish(queue, backlog: 0, latency: 0) }
  @seen_queues = seen | current
end