SidekiqSolidFetch

A reliable fetch strategy for Sidekiq that ensures jobs are never lost, even if a worker crashes mid-execution. This is an open-source implementation similar to Sidekiq Pro's super_fetch.

How It Works

SidekiqSolidFetch uses Redis LMOVE command to atomically move jobs from the queue to a per-worker processing queue. This approach ensures:

  • No job loss: Jobs are moved (not copied) to a processing queue before execution. If a worker crashes, the job remains in the processing queue.
  • Automatic recovery: Jobs stranded by dead workers are requeued on startup and periodically afterwards.
  • Graceful shutdown: When Sidekiq shuts down, in-progress jobs are moved back to their original queues.

Flow

  1. Worker boots → it registers its processing queues in a Redis registry (sidekiq_solid_fetch:registry)
  2. Worker fetches a job → job is atomically moved from queue:default to a per-worker processing queue (sidekiq_solid_fetch:processing:{capsule}:{identity}:queue:default)
  3. Worker processes the job successfully → job is removed from the processing queue
  4. Worker crashes → job stays in the processing queue
  5. Recovery (on startup of any worker, then every 5 minutes) walks the registry, detects workers whose Sidekiq heartbeat has expired, and moves their jobs back to the original queues

Recovery never touches queues belonging to live workers: liveness is checked via Sidekiq's TTL-based heartbeat key, and freshly registered queues get a grace period so a worker that just booted can't be mistaken for a dead one. Delivery semantics are at-least-once — a job interrupted mid-execution will run again.

Installation

Add to your Gemfile:

gem "sidekiq_solid_fetch"

Then run:

bundle install

Usage

Enable SidekiqSolidFetch in your Sidekiq configuration:

# config/initializers/sidekiq.rb (Rails)
# or wherever you configure Sidekiq

require "sidekiq_solid_fetch"

Sidekiq.configure_server do |config|
  SidekiqSolidFetch.enable!(config)
end

That's it! SidekiqSolidFetch will now handle job fetching with crash recovery.

Using with sidekiq-throttled

Do not prepend Sidekiq::Throttled::Patches::BasicFetch to the fetcher yourself: sidekiq-throttled's default :enqueue requeue pushes a copy of the throttled job back to the queue without removing the original from the processing queue, leaking one duplicate per throttle cycle that shutdown or orphan recovery would later re-inject as real jobs.

Use the built-in integration instead (requires sidekiq-throttled >= 2.0):

require "sidekiq_solid_fetch/throttled"

Sidekiq.configure_server do |config|
  SidekiqSolidFetch.enable!(config)
  SidekiqSolidFetch::Throttled.setup!
end

setup! applies sidekiq-throttled's fetch patches to the fetcher and makes throttled requeues an atomic move back to the queue, so nothing is copied or lost. The :schedule requeue strategy works out of the box either way.

Requirements

  • Ruby >= 3.3.0
  • Sidekiq >= 7.0

Development

After checking out the repo, run bin/setup to install dependencies.

The test suite flushes its Redis database before every test, so it runs against a dedicated Redis on port 6381 (db 15) by default — never point REDIS_URL at a Redis instance a real app uses.

# Run tests
docker compose up -d
make test

# Run linter
make lint-fix

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/k0va1/sidekiq_solid_fetch.

License

The gem is available as open source under the terms of the MIT License.