Class: Silas::DeadJobRescuerJob

Inherits:
ActiveJob::Base
  • Object
show all
Defined in:
app/jobs/silas/dead_job_rescuer_job.rb

Overview

Mandatory durability infrastructure (proven in the Phase 0 spike): Solid Queue FAILS a dead worker's claimed jobs (ProcessExitError on supervisor reap, ProcessPrunedError on heartbeat prune) and nothing retries them — retry_on can't see queue-level errors. This job, run as a recurring task, is the recovery path; recovery latency ≈ alive_threshold + its cadence.

It also sweeps expired approvals while it's here.

Constant Summary collapse

DEAD_PROCESS_ERRORS = 
%w[
  SolidQueue::Processes::ProcessExitError
  SolidQueue::Processes::ProcessPrunedError
].freeze

Instance Method Summary collapse

Instance Method Details

#perform ⇒ Object 



17
18
19
20
21
22
23
24
25
26
27
28
29
 
# File 'app/jobs/silas/dead_job_rescuer_job.rb', line 17

def perform
  ToolInvocation.expire_stale!
  return 0 unless defined?(SolidQueue)

  rescued = 0
  SolidQueue::FailedExecution.includes(:job).find_each do |failed|
    next unless DEAD_PROCESS_ERRORS.include?(failed.error&.dig("exception_class"))

    failed.retry
    rescued += 1
  end
  rescued
end