Postburner

Fast Beanstalkd-backed job queue with optional PostgreSQL records via ActiveRecord.

Built for the real world where you may want fast background processing for most jobs, but comprehensive auditing for critical operations.

  • ActiveJob Adapter - To use with Rails, ActionMailer, ActiveStorage
  • Dual-mode execution - Beanstalkd only or tracked (database backed)
  • Rich audit trail - Logs, timing, errors, instrumentation, retry tracking (tracked jobs only)
  • ActiveRecord - Query jobs with ActiveRecord (on opt in)
  • Scheduler - Schedule jobs at fixed intervals, cron expressions, and calendar-aware anchor points.
  • Test-friendly - Testing jobs can be tricky, so we go beyond just inline.
  • Process isolation - Forking workers with optional threading for throughput
  • Beanstalkd - Fast, reliable queue separate from your database, persistent storage
# Default job (fast, no PostgreSQL overhead)
class SendSmsJob < ApplicationJob # i.e. ActiveJob
  def perform(user_id)
    user = User.find(user_id)
    TextMessage.welcome(
      to: user.phone_number,
      body: "Welcome to our app!"
    ).deliver_now
  end
end

# Default job with Beanstalkd configuration
class DoSomethingJob < ApplicationJob # i.e. ActiveJob
  include Postburner::Beanstalkd # optional, allow access to beanstalkd

  priority 5000 # 0=highest, 65536=default, can set per job
  ttr 30        # 30 second timeout

  def perform(user_id)
    # Do something
  end
end

# Tracked job (full audit trail, includes Beanstalkd automatically)
class ProcessPaymentJob < ApplicationJob # i.e. ActiveJob
  include Postburner::Tracked  # ← Opt-in to tracking (includes Beanstalkd)

  priority 0  # Highest priority
  ttr 600     # 10 minute timeout

  def perform(payment_id)
    log "Processing payment #{payment_id}"
    Payment.find(payment_id).process!
    log! "Payment processed successfully"
  end
end

# Native Postburner::Job (always tracked, full api)
class RunReportJob < Postburner::Job
  queue 'critical'
  priority 0
  max_retries 0

  def perform(args)
    report = Report.find(args['report_id'])
    report.run!
    log "Ran report #{report.id}"
  end
end

# Scheduled job (fixed interval, anchor based)
Postburner::Schedule.create!(
  name: 'daily_event_retention',
  job_class: EventRetentionJob, # either Postburner::Job or ActiveJob ancestor
  anchor: Time.zone.parse('2025-01-01 09:30:00'),
  interval: 1,
  interval_unit: 'days',
  timezone: 'America/New_York',
  catch_up: false,
  args: { retention_days: 30 }
)

# Run worker (bin/postburner)
bundle exec postburner --worker default

# Or with rake task
bundle exec rake postburner:work WORKER=default

Table of Contents

Why Postburner?

Postburner supports Async, Queues, Delayed, Priorities, Timeouts, and Retries from the Backend Matrix. But uniquely, priorities are per job, in addition to the class level. Timeouts are per job and class level as well, and can be extended dynamically. Postburner also supports scheduling jobs at fixed intervals, cron expressions, and calendar-aware anchor points.

Postburner is inspired by the Backburner gem i.e. "burner", Postgres i.e. "Post", and the database backends (SolidQueue, Que, etc), and the in memory/redis backends (Sidekiq, Resque, etc). And puma's concurrency model.

Postburner beanstalkd is used with PostgreSQL to cover all the cases we care about.

  • Fast when you want it (light, ephemeral jobs like delayed turbo_stream rendering, communications)
  • Tracked when you need it critical operations like payments, and processes.
  • Able to record jobs before and after execution in PostgreSQL, with foreign keys and constraints.
  • Store the jobs outside of the database, but also persist them to disk for disaster recovery (beanstalkd binlogs)
  • Schedule jobs at fixed intervals, cron expressions, and calendar-aware anchor points.
  • Introspect the jobs either with ActiveRecord or Beanstalkd.
  • Only one worker type, that can be single/multi-process, with optional threading, and optional GC (Garbage Collection) limits (kill fork after processing N jobs).
  • Easy testing.

Quick Start

# Gemfile
gem 'postburner', '~> 1.0.0.pre.18'

# config/application.rb
config.active_job.queue_adapter = :postburner

# config/postburner.yml
development:  # <- environment config
  beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
  forks: 2
  threads: 10
  gc_limit: 500

  workers:  # <- worker config (overrides env-level)
    default:
      threads: 16        # Overrides env-level threads
      queues:
        - default
        - mailers
    critical:
      forks: 4           # Overrides env-level forks
      threads: 8         # Overrides env-level threads
      gc_limit: 24       # Overrides env-level gc_limit
      queues:
        - payments
    slow:
      forks: 4           # Overrides env-level forks
      threads: 1         # Overrides env-level threads
      gc_limit: 1        # Overrides env-level gc_limit
      queues:
        - imports
        - video

sudo apt-get install beanstalkd # OR brew install beanstalkd

beanstalkd -l 127.0.0.1 -p 11300 # Start beanstalkd

bundle exec rails generate postburner:install
bundle exec rails db:migrate

# application.rb
config.active_job.queue_adapter = :postburner
# Note: Postburner auto-prefixes queue names with "postburner.{env}." similar to:
#config.active_job.queue_name_prefix = Postburner.tube_prefix(Rails.env) # i.e. "postburner.#{Rails.env}"
config.action_mailer.deliver_later_queue_name = 'mailers' # gets prefixed by config.active_job.queue_name_prefix

bundle exec postburner           # start with bin/postburner
bundle exec rake postburner:work # or with rake task

Enqueueing Jobs

# ActiveJob (standard Rails API)
SendEmailJob.perform_later(user_id)                           # Enqueue immediately
SendEmailJob.set(wait: 1.hour).perform_later(user_id)         # Delay by duration
SendEmailJob.set(wait_until: Date.tomorrow.noon).perform_later(user_id)  # Run at specific time
SendEmailJob.set(queue: 'critical').perform_later(user_id)    # Override queue
SendEmailJob.set(priority: 0).perform_later(user_id)          # Override priority

# Postburner::Job (always tracked, full API)
job = ProcessPayment.create!(args: { 'payment_id' => 123 })
job.queue!                                    # Enqueue immediately
job.queue!(delay: 1.hour)                     # Delay by duration
job.queue!(at: Date.tomorrow.noon)            # Run at specific time
job.queue!(queue: 'critical')                 # Override queue
job.queue!(priority: 0, ttr: 600)             # Set priority and TTR

ActiveJob vs Postburner::Job TL;DR

Postburner::Job as simple subclass of ActiveRecord, so the normal ActiveRecord API applies! Thus the workflow is to create an instance, then queue it!

Operation ActiveJob Postburner::Job
Enqueue immediately MyJob.perform_later(args) MyJob.create!(args: {}).queue!
Delay .set(wait: 1.hour) job.queue!(delay: 1.hour)
Run at .set(wait_until: time) job.queue!(at: time)
Set queue .set(queue: 'critical') job.queue!(queue: 'critical')
Set priority .set(priority: 0) job.queue!(priority: 0)
Set TTR .set(ttr: 300) job.queue!(ttr: 300)
Retries ActiveJob Postburner::Job
Default No retries (discarded) No retries (buried)
Disable retries discard_on StandardError (default behavior)
Enable retries retry_on StandardError max_retries 5
  • ActiveJob: No automatic worker-level retries. Use ActiveJob's retry_on/discard_on for retry behavior. Failed jobs without retry configuration are discarded.
  • Postburner::Job: No automatic retries by default. On failure, the job is buried in Beanstalkd for inspection. Use max_retries (1-32) to enable retries with exponential backoff (2^n seconds).
# ActiveJob: Use retry_on for retries
class MyActiveJob < ApplicationJob
  retry_on StandardError, wait: :polynomially_longer, attempts: 5

  def perform(args)
    # ...
  end
end

# Postburner::Job: Use max_retries for retries
class MyJob < Postburner::Job
  max_retries 5  # Exponential backoff: 1s, 2s, 4s, 8s, 16s
end

# Or use a fixed delay
class MyJob < Postburner::Job
  max_retries 5
  retry_delay 10  # 10 seconds between retries
end

Usage

Default Jobs

Default jobs execute quickly via Beanstalkd without PostgreSQL overhead. Perfect for emails, cache warming, notifications, etc.

class SendWelcomeEmail < ApplicationJob
  queue_as :mailers

  def perform(user_id)
    UserMailer.welcome(user_id).deliver_now
  end
end

# Enqueue immediately
SendWelcomeEmail.perform_later(123)

# Enqueue with delay
SendWelcomeEmail.set(wait: 1.hour).perform_later(123)

# Enqueue at specific time
SendWelcomeEmail.set(wait_until: Date.tomorrow.noon).perform_later(123)

Overview:

  • Fast execution (no database writes)
  • Low overhead
  • Standard ActiveJob retry_on/discard_on support
  • No audit trail
  • No logging or statistics

Configuring Beanstalkd Priority and TTR

For default jobs that need custom Beanstalkd configuration, include Postburner::Beanstalkd:

class SendWelcomeEmail < ApplicationJob
  include Postburner::Beanstalkd

  queue_as :mailers
  priority 100    # Lower = higher priority (0 is highest)
  ttr 300         # Time-to-run in seconds (5 minutes)

  def perform(user_id)
    UserMailer.welcome(user_id).deliver_now
  end
end

Configuration options:

  • priority - Beanstalkd priority (0-4294967295, lower = higher priority)
  • ttr - Time-to-run in seconds before job times out

Jobs without Postburner::Beanstalkd use defaults from config/postburner.yml:

  • default_priority: 65536
  • default_ttr: 300
  • default_queue: default (if not specified, defaults to 'default')

Configuring Third-Party Jobs

For jobs you don't control (e.g., Turbo::Streams::BroadcastJob, ActionMailer::MailDeliveryJob), use the enqueue_options hook:

# config/initializers/postburner.rb
Postburner.configure do |config|
  config.enqueue_options = ->(job) do
    case job.class.name
    when 'Turbo::Streams::BroadcastJob'
      { priority: 100, ttr: 60 }
    when 'ActionMailer::MailDeliveryJob'
      { priority: 500, ttr: 120 }
    else
      {}  # Use defaults
    end
  end
end

Priority cascade: job.priority (from .set(priority: n)) > enqueue_options hook > class-level > default_priority

TTR cascade: class-level ttr > enqueue_options hook > default_ttr

Tracked Jobs

Tracked jobs store full execution details in PostgreSQL, providing comprehensive audit trails (i.e. logging, timing, errors, retry tracking) for critical operations.

Note: Postburner::Tracked automatically includes Postburner::Beanstalkd, giving you access to priority, ttr, bk, and extend!.

class ProcessPayment < ApplicationJob
  include Postburner::Tracked  # ← Opt-in to PostgreSQL tracking (includes Beanstalkd)

  queue_as :critical
  priority 0              # Highest priority (Beanstalkd included automatically)
  ttr 600                 # 10 minute timeout
  retry_on StandardError, wait: :exponentially_longer, attempts: 5

  def perform(payment_id)
    payment = Payment.find(payment_id)

    log "Starting payment processing for $#{payment.amount}"

    begin
      payment.charge!
      log! "Payment charged successfully"
    rescue PaymentError => e
      log_exception!(e)
      raise
    end

    log "Payment complete", level: :info
  end
end

Tracking provides:

  • Complete execution history (queued_at, processing_at, processed_at)
  • Custom logs with log and log!
  • Exception tracking with log_exception() and log_exception!()
  • Timing statistics (lag, duration)
  • Retry attempts tracking
  • Query with ActiveRecord
  • Foreign key relationships
  • Beanstalkd operations via bk accessor
  • TTR extension via extend! method

Query tracked jobs:

# Find by state
Postburner::TrackedJob.where(processed_at: nil)
Postburner::TrackedJob.where.not(removed_at: nil)

# Find with errors
Postburner::TrackedJob.where("error_count > 0")

# Statistics
Postburner::TrackedJob.average(:duration)  # Average execution time
Postburner::TrackedJob.maximum(:lag)       # Worst lag

# Inspect execution
job = Postburner::TrackedJob.last
job.logs      # Array of log entries with timestamps
job.errata    # Array of exceptions with backtraces
job.attempts  # Array of attempt timestamps
job.duration  # Execution time in milliseconds
job.lag       # Queue lag in milliseconds

Postburner::Job Usage

Direct Postburner::Job subclasses are always tracked:

class ProcessPayment < Postburner::Job
  queue 'critical'
  priority 0
  max_retries 0

  def perform(args)
    payment = Payment.find(args['payment_id'])
    payment.process!
    log "Processed payment #{payment.id}"
  end
end

# Create and enqueue
job = ProcessPayment.create!(args: { 'payment_id' => 123 })
job.queue!

# With delay
job.queue!(delay: 1.hour)

# At specific time
job.queue!(at: 2.days.from_now)

Note: The args parameter in perform(args) is optional. It's a convenience accessor to self.args, which is stored in a JSONB column on the job record. You can omit the parameter and access args directly:

def perform
  payment = Payment.find(self.args['payment_id'])
  # ...
end

Instance-Level Queue Configuration

Override queue priority and TTR per job instance for dynamic behavior:

# Set priority during creation
job = ProcessPayment.create!(
  args: { 'payment_id' => 123 },
  priority: 1500,  # Override class-level priority
  ttr: 300         # Override class-level TTR
)
job.queue!

# Set dynamically after creation
job = ProcessPayment.create!(args: { 'payment_id' => 456 })
job.priority = 2000
job.ttr = 300
job.queue!

# Use in before_enqueue callback for conditional behavior
class ProcessOrder < Postburner::Job
  queue 'orders'
  priority 100   # Default priority
  ttr 120  # Default TTR

  before_enqueue :set_priority_based_on_urgency

  def perform(args)
    order = Order.find(args['order_id'])
    order.process!
  end

  private

  def set_priority_based_on_urgency
    if args['express_shipping']
      self.priority = 0    # High priority for express orders
      self.ttr = 600       # Allow 10 minutes to complete
    else
      self.priority = 1000 # Low priority for standard orders
      self.ttr = 120       # Standard 2 minute timeout
    end
  end
end

# Express order gets high priority automatically
ProcessOrder.create!(args: { 'order_id' => 789, 'express_shipping' => true }).queue!

Overview:

  • Same as Tracked jobs
  • Access the ActiveRecord Postburner::Job directly.

Scheduler

Postburner includes a lightweight, fixed-rate scheduler for recurring jobs. Perfect for daily reports, weekly cleanups, or any task that needs to run on a predictable schedule.

The scheduler uses immediate enqueue combined with a watchdog safety net:

  1. When an execution is created, it's immediately enqueued to Beanstalkd's delayed queue with the appropriate delay until run_at
  2. For Postburner::Job and ActiveJob with Postburner::Tracked schedules the next execution when the current job runs - providing immediate pickup without waiting for the watchdog. Normal ActiveJob schedules need to rely on the watchdog to create the next execution, so set the scheduler_interval to pick up executions appropriately.
  3. A lightweight watchdog job in the scheduler tube acts as a safety net: json { "scheduler": true, "interval": 300 }
  4. When a worker reserves the watchdog, it instantiates Postburner::Scheduler which:
    • Acquires a PostgreSQL advisory lock for coordination
    • Auto-bootstraps any unstarted schedules
    • Ensures each schedule has a future execution queued
    • Re-queues a new watchdog with delay for the next interval

NOTE: The watchdog is ephemeral data in Beanstalkd, not a database record. Postburner::Scheduler is the handler class that does the work. This design requires no dedicated scheduler process - existing workers handle everything.

# Create a schedule for a daily report at 9:30 AM
Postburner::Schedule.create!(
  name: 'daily_metrics',
  job_class: 'GenerateMetricsReportJob',
  anchor: Time.zone.parse('2025-01-01 09:30:00'),
  interval: 1,
  interval_unit: 'days',
  timezone: 'America/New_York',
  args: { report_type: 'daily' }  # Passed as keyword args to perform(report_type:)
)

The args hash is passed to each job execution. For ActiveJob classes, hash args become keyword arguments (perform(report_type:)). For Postburner::Job subclasses, args are passed as a hash to perform(args).

The scheduler automatically creates executions and enqueues jobs at the scheduled times.

Configuration

Add scheduler settings to config/postburner.yml:

production:
  beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
  scheduler_interval: 300  # Pickup new schedules every 5 minutes
  scheduler_priority: 100  # Scheduler jobs run at priority 100

  workers:
    default:
      queues:
        - default
        - mailers

Configuration options:

  • scheduler_interval - How often (in seconds) to check for due schedules (default: 300)
  • scheduler_priority - Beanstalkd priority for watchdog jobs (default: 100)

Choosing an interval: Since executions are enqueued immediately to Beanstalkd's delayed queue, the watchdog interval primarily affects:

  • How quickly new schedules are auto-bootstrapped (if you don't call start!)
  • Recovery time if an execution somehow fails to enqueue
  • How often last_audit_at is updated for monitoring

For most use cases, the default 300 seconds is appropriate. Lower values increase database queries without significant benefit since jobs are already queued in Beanstalkd.

No manual setup required: Workers automatically watch the scheduler tube and create the watchdog job if one doesn't exist. Just start your workers and create schedules - the rest happens automatically.

Creating Schedules

Use an anchor time plus interval for predictable, drift-free scheduling:

# Every day at 9:00 AM Eastern
Postburner::Schedule.create!(
  name: 'daily_cleanup',
  job_class: 'CleanupJob',
  anchor: Time.zone.parse('2025-01-01 09:00:00'),
  interval: 1,
  interval_unit: 'days',
  timezone: 'America/New_York'
)

# Every 6 hours starting from midnight
Postburner::Schedule.create!(
  name: 'sync_data',
  job_class: 'DataSyncJob',
  anchor: Time.zone.parse('2025-01-01 00:00:00'),
  interval: 6,
  interval_unit: 'hours',
  timezone: 'UTC'
)

# Weekly on Saturday at 2:00 AM
Postburner::Schedule.create!(
  name: 'weekly_report',
  job_class: 'WeeklyReportJob',
  anchor: Time.zone.parse('2025-01-04 02:00:00'),  # A Saturday
  interval: 1,
  interval_unit: 'weeks',
  timezone: 'America/Los_Angeles'
)

# Monthly on the 1st at midnight
Postburner::Schedule.create!(
  name: 'monthly_billing',
  job_class: 'MonthlyBillingJob',
  anchor: Time.zone.parse('2025-01-01 00:00:00'),
  interval: 1,
  interval_unit: 'months',
  timezone: 'UTC'
)

Available interval units: seconds, minutes, hours, days, weeks, months, years

Cron-Based Scheduling

For complex schedules, use cron expressions (requires the fugit gem):

# Every weekday at 8:00 AM
Postburner::Schedule.create!(
  name: 'weekday_standup',
  job_class: 'StandupReminderJob',
  cron: '0 8 * * 1-5',
  timezone: 'America/Chicago'
)

# Every 15 minutes
Postburner::Schedule.create!(
  name: 'health_check',
  job_class: 'HealthCheckJob',
  cron: '*/15 * * * *',
  timezone: 'UTC'
)

Schedule Options

Postburner::Schedule.create!(
  name: 'important_job',              # Required: unique identifier
  job_class: 'MyJob',                 # Required: ActiveJob or Postburner::Job class name

  # Scheduling (choose anchor+interval OR cron)
  anchor: Time.zone.parse('...'),     # Start time for interval calculation
  interval: 1,                        # Number of interval units
  interval_unit: 'days',              # seconds/minutes/hours/days/weeks/months/years
  # OR
  cron: '0 9 * * *',                  # Cron expression (requires fugit gem)

  # Optional
  timezone: 'America/New_York',       # Default: 'UTC'
  args: { key: 'value' },             # Arguments passed to job
  queue: 'critical',                  # Override default queue
  priority: 0,                        # Override default priority
  enabled: true,                      # Enable/disable schedule
  catch_up: false,                    # Skip missed executions (default: false)
  description: 'Daily metrics run'    # Human-readable description
)

Catch-Up Policy

The catch_up option controls what happens when the worker is down and misses scheduled executions:

  • catch_up: false (default) - Skip missed executions, create next future execution only (health checks, status updates, cache warming, monitoring)
  • catch_up: true - Run all missed executions when worker restarts (data processing pipelines, billing cycles, audit requirements, SLA-sensitive operations)

Notes: 1) Previously created executions and enqueued jobs are not affected by the catch-up policy. 2) If catch_up if off, then there the skipped executions are not created or tracked. It resumes from the next future execution.

# Skip missed executions (default)
Postburner::Schedule.create!(
  name: 'health_check',
  job_class: 'HealthCheckJob',
  interval: 1,
  interval_unit: 'minutes',
  catch_up: false  # Worker down 9:00-9:05 → skips 9:01-9:04, creates 9:06
)

# Run all missed executions
Postburner::Schedule.create!(
  name: 'process_billing',
  job_class: 'BillingJob',
  interval: 1,
  interval_unit: 'hours',
  catch_up: true  # Worker down 9:00-12:00 → runs 9:00, 10:00, 11:00, 12:00
)

Managing Schedules

# Find schedules
schedule = Postburner::Schedule.find_by(name: 'daily_cleanup')
Postburner::Schedule.enabled  # All enabled schedules

# Disable temporarily
schedule.update!(enabled: false)

# Change catch-up policy
schedule.update!(catch_up: true)

# Check next run time
schedule.next_run_at  # => 2025-01-02 09:00:00 -0500

# Preview upcoming runs
schedule.next_run_at_times(5)  # Next 5 run times

# View executions
schedule.executions.pending
schedule.executions.scheduled
schedule.executions.skipped

Starting Schedules

When you create a schedule, it won't run until the first execution is created. You have two options:

Option 1: Explicit start (immediate enqueue)

Call start! to create and enqueue the first execution immediately to Beanstalkd:

schedule = Postburner::Schedule.create!(
  name: 'daily_report',
  job_class: 'DailyReportJob',
  anchor: Time.zone.parse('2025-01-01 09:00:00'),
  interval: 1,
  interval_unit: 'days'
)
schedule.start!  # Creates execution AND enqueues to Beanstalkd
schedule.started?  # => true

The job is immediately in Beanstalkd's delayed queue and will run at the scheduled time.

Option 2: Auto-bootstrap (eventual pickup)

If you don't call start!, the scheduler watchdog will automatically bootstrap the schedule on its next run. This adds up to one scheduler_interval of delay before the first execution is enqueued:

schedule = Postburner::Schedule.create!(...)
schedule.started?  # => false

# Later, scheduler runs and auto-bootstraps...
schedule.reload.started?  # => true

Use start! when you need predictable timing. Skip it when eventual consistency is acceptable.

Schedule Executions

Each scheduled run creates an execution record for tracking:

execution = Postburner::ScheduleExecution.find(123)

execution.status        # pending, scheduled, skipped
execution.run_at        # Scheduled time
execution.enqueued_at   # When job was queued
execution.beanstalk_job_id  # Beanstalkd job ID
execution.job_id            # Postburner::Job ID (if using Postburner::Job)

Execution lifecycle:

  1. Execution created with pending status and immediately enqueued to Beanstalkd
  2. Status changes to scheduled once enqueued
  3. At run_at time, Beanstalkd releases job to worker
  4. For Postburner::Job (and ActiveJob with Postburner::Tracked) schedules: before_attempt callback creates next execution
  5. Watchdog periodically verifies future executions exist (safety net)

Timezone Handling

Schedules respect timezones for consistent execution times regardless of server location:

# Runs at 9:00 AM New York time (handles DST automatically)
Postburner::Schedule.create!(
  name: 'ny_morning_job',
  job_class: 'MorningJob',
  anchor: Time.zone.parse('2025-01-01 09:00:00'),
  interval: 1,
  interval_unit: 'days',
  timezone: 'America/New_York'
)

The scheduler calculates next run times in the specified timezone, so jobs run at the expected local time even across daylight saving transitions.

Job Management

Canceling Jobs

For Postburner::Job subclasses:

job = ProcessPayment.create!(args: { 'payment_id' => 123 })
job.queue!(delay: 1.hour)

# Soft delete (recommended) - removes from queue, sets removed_at
job.remove!
job.removed_at  # => 2025-11-19 12:34:56 UTC
job.persisted?  # => true (still in database for audit)

# Hard delete - removes from both queue and database
job.destroy!

# Delete from Beanstalkd only (keeps database record)
job.delete!

For tracked ActiveJob classes:

# Find the TrackedJob record
job = Postburner::TrackedJob.find(123)
job.remove!  # Cancel execution

Retrying Buried Jobs

Jobs that fail repeatedly get "buried" in Beanstalkd:

job = Postburner::Job.find(123)
job.kick!  # Moves buried job back to ready queue

Inspecting Job State

job = Postburner::TrackedJob.find(123)

# Timestamps
job.queued_at      # When queued to Beanstalkd
job.run_at         # Scheduled execution time
job.attempting_at  # First attempt started
job.processing_at  # Current/last processing started
job.processed_at   # Completed successfully
job.removed_at     # Soft deleted

# Statistics
job.lag       # Milliseconds between scheduled and actual execution
job.duration  # Milliseconds to execute
job.attempt_count  # Number of attempts
job.error_count    # Number of errors

# Audit trail
job.logs    # Array of log entries with timestamps
job.errata  # Array of exceptions with backtraces
job.attempts # Array of attempt timestamps

Writing Jobs

Pay attention to the following when writing jobs:

Job Idempotency: Jobs should be designed to be idempotent and safely re-runnable. Like all job queues, Postburner provides at-least-once delivery—in rare errant cases outside of Postburner's control, a job may be executed more than once, i.e. network issues, etc.

TTR (Time-to-Run): If a job exceeds its TTR without completion, Beanstalkd releases it back to the queue while still running—causing duplicate execution. For long-running jobs, call extend! periodically to reset the TTR, or set a sufficiently high TTR value. You must include the Postburner::Beanstalkd or Postburner::Tracked module with ActiveJob to use extend!.

Queue Strategies

Postburner uses different strategies to control job execution. These affect Postburner::Job subclasses (not ActiveJob classes).

Strategy When to Use Behavior Requires Beanstalkd
DefaultQueue Production Async, gracefully requeues premature jobs Yes
StrictQueue Production Async via Beanstalkd, raises error on premature execution Yes
NullQueue Test (usually) Jobs created but not inserted into Beanstalkd, manual execution No
InlineTestQueue Test Inline execution (error raised for delayed jobs) No
TimeTravelTestQueue Test Inline execution, (auto time-travels for delayed jobs) No 
# Switch strategies
Postburner.default_strategy!              # Default production (DefaultQueue)
Postburner.strict_strategy!               # Strict production (StrictQueue)
Postburner.null_strategy!                 # Deferred, or manual execution
Postburner.inline_test_strategy!          # Testing (InlineTestQueue)
Postburner.time_travel_test_strategy!     # Testing inline but also with time travel

Adding a Queue Strategy

There are 4 hook methods that are used with inheritance:

  • insert - Inserts the job into the queue
  • handle_perform! - Handles the job being performed
  • handle_premature_perform - Handles jobs executed before their scheduled run_at time
  • testing - Returns true if the strategy is for testing

Testing

Automatic Test Mode

In Rails test environments, Postburner automatically uses inline execution via the InlineTestQueue strategy:

# test/test_helper.rb - automatic!
Postburner.testing?  # => true in tests

Postburner provides test-friendly execution modes that don't require Beanstalkd.

Testing Default Jobs (ActiveJob)

Use standard ActiveJob test helpers:

class MyTest < ActiveSupport::TestCase
  test "job executes" do
    SendEmail.perform_later(123)
    # Job executes immediately in test mode
  end

  test "job with delay" do
    travel 1.hour do
      SendEmail.set(wait: 1.hour).perform_later(123)
    end
  end
end

Testing Tracked Jobs

Tracked jobs create database records you can inspect:

test "tracked job logs execution" do
  ProcessPayment.perform_later(456)

  job = Postburner::TrackedJob.last
  assert job.processed_at.present?
  assert_includes job.logs.map { |l| l[1]['message'] }, "Processing payment 456"
end

Testing Emails

Standard ActiveJob test assertions (assert_enqueued_emails, assert_enqueued_jobs) work when using the :test ActiveJob adapter in your test environment, which is the Rails default:

class EmailTest < ActiveSupport::TestCase
  # assert_enqueued_emails works — Rails' :test adapter tracks the enqueue
  test "welcome email is enqueued" do
    assert_enqueued_emails 1 do
      UserMailer.welcome(user).deliver_later
    end
  end

  # assert_emails works — the job executes inline, delivering the email
  test "welcome email is sent" do
    assert_emails 1 do
      SendWelcomeEmail.perform_later(user.id)
    end
  end
end

How this works: assert_enqueued_emails and assert_enqueued_jobs are powered by Rails' :test ActiveJob adapter, which tracks enqueued jobs in an in-memory array. Postburner's queue strategies (InlineTestQueue, etc.) control Postburner::Job subclasses separately. The two systems coexist — ActiveJob assertions use the :test adapter, Postburner::Job assertions use database queries.

By default, Rails uses the :test adapter in test mode unless you override it. Since you set :postburner as your adapter in config/application.rb, you should set it back to :test for your test environment:

# config/environments/test.rb
config.active_job.queue_adapter = :test

This is standard practice for any ActiveJob adapter (Sidekiq, SolidQueue, etc.) — the production adapter handles real execution, and the :test adapter enables Rails' built-in assertion helpers. Without this, assert_enqueued_emails will not work because Postburner's adapter executes jobs rather than tracking them in an array.

Postburner::Mailer bypasses ActiveJob entirely, so assert_enqueued_emails cannot see these jobs regardless of adapter. Assert on side effects instead:

test "mailer job sends email" do
  job = Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1)

  assert_emails 1 do
    job.queue!  # Executes inline in test mode, delivering the email
  end

  assert job.reload.processed_at
end

Switching Queue Strategies

For tests requiring specific queue behaviors, use switch_queue_strategy! and restore_queue_strategy!:

class ScheduleTest < ActiveSupport::TestCase
  def setup
    # Use time travel test strategy for inline execution with auto time travel
    switch_queue_strategy! Postburner::TimeTravelTestQueue
  end

  def teardown
    restore_queue_strategy!
  end

  test "scheduled job executes" do
    job = MyJob.create!(args: {})
    job.queue!(delay: 1.hour)
    assert job.reload.processed_at  # Auto time-travels
  end
end

Block form for isolated tests:

test "specific strategy for one test" do
  # Use the NullQueue to not push the job to Beanstalkd, only create the db record.
  use_queue_strategy Postburner::NullQueue do
    job = MyJob.create!(args: {})
    job.queue!
    assert_nil job.bkid  # Not queued to Beanstalkd

    # The manually execute the job, you must use `travel_to` if it is scheduled for the future.
    Postburner::Job.perform(job.id)
    assert job.reload.processed_at # completed time
  end
end

Available strategies:

  • Postburner::InlineTestQueue - Default test mode, raises on delayed jobs
  • Postburner::TimeTravelTestQueue - Auto time-travel for delayed jobs
  • Postburner::NullQueue - Create jobs without queueing
  • Postburner::StrictQueue - Production mode with error on premature execution
  • Postburner::DefaultQueue - Production mode with graceful requeue

Workers

Postburner uses named worker configurations to support different deployment patterns. Each worker can have different fork/thread settings and process different queues, enabling flexible production deployments.

Named Workers Configuration

Configure multiple named workers with different concurrency profiles:

production:  # <- environment config
  beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
  forks: 2
  threads: 10
  gc_limit: 5000

  workers:  # <- worker config (overrides env-level)
    # Heavy, memory-intensive jobs - more processes, fewer threads
    imports:
      forks: 4           # Overrides env-level forks
      threads: 1         # Overrides env-level threads
      gc_limit: 500      # Overrides env-level gc_limit
      queues:
        - imports
        - data_processing

    # General jobs - fewer processes, many threads
    general:
      threads: 100       # Overrides env-level threads (forks uses env-level forks=2)
      queues:
        - default
        - mailers
        - notifications

Puma-Style Architecture

The worker supports two modes based on fork configuration:

  • forks: 0 - Single process with thread pools (development/staging)
  • forks: 1+ - Multiple processes with thread pools (production, Puma-style)

This gives you a natural progression from simple single-threaded development to high-concurrency production.

Configuration Examples

Development (single worker, single-threaded):

development:  # <- environment config
  beanstalk_url: beanstalk://localhost:11300

  workers:  # <- worker config
    default:
      # Uses env defaults: forks=0, threads=1, gc_limit=nil
      queues:
        - default
        - mailers

Staging (single worker, multi-threaded):

staging:  # <- environment config
  beanstalk_url: beanstalk://localhost:11300
  threads: 10
  gc_limit: 5000

  workers:  # <- worker config
    default:
      # Uses env-level: threads=10, gc_limit=5000
      queues:
        - critical
        - default
        - mailers

Production (multiple workers with different profiles):

production:  # <- environment config
  beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
  forks: 2
  threads: 10
  gc_limit: 5000

  workers:  # <- worker config (overrides env-level)
    imports:
      forks: 4           # Overrides env-level forks (4 processes)
      threads: 1         # Overrides env-level threads (1 thread per process = 4 concurrent jobs)
      gc_limit: 500      # Overrides env-level gc_limit
      queues:
        - imports
        - data_processing

    general:
      # forks uses env-level forks=2 (2 processes)
      threads: 100       # Overrides env-level threads (100 threads per process = 200 concurrent jobs)
      # gc_limit uses env-level gc_limit=5000
      queues:
        - default
        - mailers
        - notifications

Running Workers

Single worker (auto-selected):

bin/postburner

If only one worker is defined, it's automatically selected.

Multiple workers (must specify):

bin/postburner --worker imports    # Run the 'imports' worker
bin/postburner --worker general    # Run the 'general' worker

Filter queues (override config):

bin/postburner --worker general --queues default,mailers  # Only process specific queues

Note: Prefer defining separate workers in config/postburner.yml rather than using --queues overrides. Named workers make your deployment configuration explicit and version-controlled. Use --queues only for debugging or temporary overrides.

Rake task:

bundle exec rake postburner:work                          # Auto-select worker
bundle exec rake postburner:work WORKER=general           # Specific worker
bundle exec rake postburner:work WORKER=general QUEUES=default,mailers  # Filter queues
bundle exec rake postburner:work CONFIG=config/custom.yml # Custom config

Programmatic Worker Control

Use Postburner::Runner to start workers programmatically from Ruby code:

# Basic usage
runner = Postburner::Runner.new(
  config: 'config/postburner.yml',
  env: 'production',
  worker: 'default'
)
runner.run

# With queue filtering
runner = Postburner::Runner.new(
  worker: 'general',
  queues: ['default', 'mailers']
)
runner.run

# From environment variables (Rake task pattern)
runner = Postburner::Runner.new(
  config: ENV['CONFIG'] || 'config/postburner.yml',
  env: Rails.env,
  worker: ENV['WORKER'],
  queues: ENV['QUEUES']&.split(',')
)
runner.run

Options:

  • :config - Path to YAML configuration file (default: 'config/postburner.yml')
  • :env - Environment name (default: RAILS_ENV or 'development')
  • :worker - Worker name from config (required if multiple workers defined)
  • :queues - Array of queue names to filter (overrides config queues)

This provides a unified interface used by both bin/postburner and rake postburner:work, making it easy to integrate Postburner workers into custom deployment scripts or process managers.

Running Workers in Separate Processes

For production deployments, run different workers in separate OS processes for isolation and resource allocation:

Docker Compose example:

services:
  # Process 1: Imports worker (forks=4, threads=1)
  worker-imports:
    build: .
    command: bin/postburner --worker imports
    environment:
      RAILS_ENV: production
      BEANSTALK_URL: beanstalk://beanstalkd:11300
    depends_on:
      - beanstalkd
      - postgres

  # Process 2: General worker (forks=2, threads=100)
  worker-general:
    build: .
    command: bin/postburner --worker general
    environment:
      RAILS_ENV: production
      BEANSTALK_URL: beanstalk://beanstalkd:11300
    depends_on:
      - beanstalkd
      - postgres

Procfile example (Heroku/Foreman):

worker_imports: bin/postburner --worker imports
worker_general: bin/postburner --worker general

Worker Architecture

Single Process Mode (forks: 0):

Main Process
├─ Queue 'critical' Thread Pool (1 thread)
├─ Queue 'default' Thread Pool (10 threads)
└─ Queue 'mailers' Thread Pool (5 threads)

Multi-Process Mode (forks: 1+):

Parent Process
├─ Fork 0 (queue: critical)
│   └─ Thread 1
├─ Fork 0 (queue: default)
│   ├─ Thread 1-10
├─ Fork 1 (queue: default)  # Puma-style: multiple forks of same queue
│   ├─ Thread 1-10
├─ Fork 2 (queue: default)
│   └─ Thread 1-10
├─ Fork 3 (queue: default)
│   └─ Thread 1-10
│   └─ Total: 40 concurrent jobs for 'default' (4 forks × 10 threads)
└─ Fork 0 (queue: mailers)
    └─ Thread 1-5

Benefits

Single Process Mode (forks: 0):

  • Simplest deployment
  • Easy debugging
  • Lower memory footprint
  • Best for development and moderate load

Multi-Process Mode (forks: 1+):

  • Horizontal scaling per queue
  • Process isolation
  • Automatic memory cleanup via GC limits
  • Crash isolation (one fork down doesn't affect others)
  • Best for production high-volume workloads

Shutdown Timeout

The shutdown_timeout controls how long workers wait for in-flight jobs to complete during graceful shutdown. This applies to thread pool termination and child process shutdown.

production:
  default_ttr: 300        # Default TTR for jobs
  shutdown_timeout: 300   # Defaults to default_ttr if not specified

  workers:
    imports:
      shutdown_timeout: 600     # Override: wait up to 10 minutes for long imports
      queues:
        - imports

Best practice: Set shutdown_timeout to match or exceed your longest-running job's TTR to avoid force-killing jobs during deployment.

Connection model: Each worker thread maintains its own Beanstalkd connection for thread safety.

GC Limits

Set gc_limit at environment level or per worker to automatically restart after processing N jobs.

  • Worker processes N jobs
  • Worker exits with code 99
  • In single-process mode (forks: 0): process manager restarts the entire process
  • In multi-process mode (forks: 1+): parent process automatically restarts just that fork
production:  # <- environment config
  forks: 2
  threads: 10
  gc_limit: 5000

  workers:  # <- worker config
    imports:
      forks: 4           # Overrides env-level forks
      threads: 1         # Overrides env-level threads
      gc_limit: 500      # Overrides env-level gc_limit (restart after 500 jobs, memory-intensive)
      queues:
        - imports
        - data_processing

    general:
      # Uses env-level forks=2, threads=100, gc_limit=5000 (restart after 5000 jobs)
      threads: 100       # Overrides env-level threads
      queues:
        - default
        - mailers

Why?

  • Prevents memory bloat in long-running workers
  • Automatic cleanup without manual intervention
  • Works seamlessly with Docker and Kubernetes restarts

Configuration

YAML Configuration

# config/postburner.yml
default: &default
  beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
  # default_queue: default       # Queue for jobs (defaults to 'default')
  default_mailer_queue: mailers  # Queue for Postburner::Mailer (defaults to default_queue)

development:  # <- environment config
  <<: *default
  # Env defaults: forks=0, threads=1, gc_limit=nil

  workers:  # <- worker config
    default:
      queues:
        - default
        - mailers

test:  # <- environment config
  <<: *default
  # Env defaults: forks=0, threads=1, gc_limit=nil

  workers:  # <- worker config
    default:
      queues:
        - default

staging:  # <- environment config
  <<: *default
  threads: 10      # Multi-threaded, single process
  gc_limit: 5000

  workers:  # <- worker config
    default:
      # Uses env-level: threads=10, gc_limit=5000
      queues:
        - default
        - mailers

production:  # <- environment config
  <<: *default
  forks: 2         # Inherited by workers
  threads: 10      # Inherited by workers
  gc_limit: 5000   # Inherited by workers

  workers:  # <- worker config (overrides env-level)
    critical:
      forks: 1             # Overrides env-level forks
      threads: 1           # Overrides env-level threads (1 concurrent job)
      gc_limit: 100        # Overrides env-level gc_limit
      queues:
        - critical

    default:
      forks: 4             # Overrides env-level forks
      threads: 10          # Overrides env-level threads (40 total concurrent jobs: 4 × 10)
      gc_limit: 1000       # Overrides env-level gc_limit
      queues:
        - default

    mailers:
      # forks uses env-level forks=2
      threads: 5           # Overrides env-level threads (10 total email senders: 2 × 5)
      gc_limit: 500        # Overrides env-level gc_limit
      queues:
        - mailers

Start workers with:

bin/postburner                    # Auto-select single worker
bin/postburner --worker default   # Specify worker
rake postburner:work              # Rake task (auto-select)
rake postburner:work WORKER=default

Queue Names

Postburner automatically prefixes all queue names with postburner.{env}. to create Beanstalkd tube names. This namespacing prevents collisions when multiple applications share the same Beanstalkd server.

The same base queue name is used everywhere:

Context Syntax Beanstalkd Tube
ActiveJob queue_as :mailers postburner.production.mailers
Postburner::Job queue 'mailers' postburner.production.mailers
postburner.yml queues: [mailers] postburner.production.mailers

Example:

# app/jobs/welcome_email_job.rb
class WelcomeEmailJob < ApplicationJob
  queue_as :mailers  # Uses base name 'mailers'
end

# config/postburner.yml
production:
  workers:
    email_worker:
      queues:
        - mailers  # Same base name 'mailers'

Both refer to the Beanstalkd tube postburner.production.mailers.

Naming convention: Use underscores for multi-word queue names (e.g., background_jobs, high_priority). Avoid hyphens as they can cause issues with some Beanstalkd client libraries.

Important: No need to set config.active_job.queue_name_prefix - Postburner handles prefixing automatically when jobs are enqueued with postburner.

Queue Configuration Methods

For Postburner::Job subclasses (and backwards compatibility):

class CriticalJob < Postburner::Job
  queue 'critical'                  # Beanstalkd tube name
  priority 0                  # Lower = higher priority (0 is highest)
  ttr 300                     # TTR (time-to-run) in seconds
  max_retries 3           # Max retry attempts
  retry_delay 5               # Fixed delay: 5 seconds between retries

  def perform(args)
    # ...
  end
end

Exponential backoff with proc:

class BackgroundTask < Postburner::Job
  queue 'default'
  max_retries 5
  retry_delay ->(retries) { 2 ** retries }  # 2s, 4s, 8s, 16s, 32s

  def perform(args)
    # ...
  end
end

ActiveJob Configuration

For ActiveJob classes, use standard ActiveJob configuration:

class MyJob < ApplicationJob
  include Postburner::Tracked  # Optional: for audit trail

  queue_as :default

  retry_on StandardError, wait: :exponentially_longer, attempts: 5
  retry_on CustomError, wait: 10.seconds

  discard_on ActiveJob::DeserializationError

  def perform(user_id)
    # ...
  end
end

Callbacks

Postburner provides lifecycle callbacks:

class ProcessPayment < ApplicationJob
  include Postburner::Tracked

  before_enqueue :validate_payment
  after_enqueue :send_confirmation

  before_attempt :log_attempt       # Runs on every retry
  after_attempt :update_metrics

  before_processing :acquire_lock
  after_processing :release_lock

  after_processed :send_receipt     # Only on success

  def perform(payment_id)
    # ...
  end

  private

  def validate_payment
    raise "Invalid" unless payment_valid?
  end

  def send_receipt
    PaymentMailer.receipt(arguments.first).deliver_later
  end
end

Available callbacks:

  • before_enqueue, around_enqueue, after_enqueue - When queued
  • before_attempt, around_attempt, after_attempt - Every execution (including retries)
  • before_processing, around_processing, after_processing - During execution
  • after_processed - Only after successful completion

Instrumentation

Postburner emits ActiveSupport::Notifications events following Rails conventions. Use these for monitoring, logging, or alerting.

Job Events

Event When Payload Keys
perform_start.job.postburner Before job execution begins :job, :beanstalk_job_id, :gc_count, :gc_limit
perform.job.postburner Around job execution (includes duration) :job, :beanstalk_job_id, :gc_count, :gc_limit
retry.job.postburner When Postburner::Job is retried :job, :beanstalk_job_id, :error, :wait, :attempt
retry_stopped.job.postburner When Postburner::Job exhausts retries (buried) :job, :beanstalk_job_id, :error
discard.job.postburner When default ActiveJob fails (discarded) :job, :beanstalk_job_id, :error
enqueue.job.postburner When job is enqueued for immediate execution :job
enqueue_at.job.postburner When job is enqueued with delay :job, :scheduled_at

Job Payload Structure:

{
  class: "ProcessPayment",           # Job class name
  id: 123,                           # Postburner job ID (tracked jobs only)
  job_id: "abc-123",                 # ActiveJob UUID
  arguments: { payment_id: 456 },    # Job arguments
  queue_name: "critical",            # Queue name
  beanstalk_job_id: 789,             # Beanstalkd job ID
  tracked: true,                     # Whether job is tracked in PostgreSQL
  gc_count: 42,                      # Jobs processed in this thread (nil if unavailable)
  gc_limit: 5000                     # GC limit for this worker (nil if unlimited)
}

Schedule Events

Event When Payload Keys
create.schedule.postburner When schedule is created :schedule
update.schedule.postburner When schedule is updated :schedule, :changes
audit.schedule.postburner When scheduler audits a schedule :schedule

Schedule Payload Structure:

{
  id: 1,
  name: "daily_cleanup",
  job_class: "CleanupJob",
  enabled: true,
  cron: nil,
  anchor: "2025-01-01T09:00:00Z",
  interval: 1,
  interval_unit: "days"
}

Schedule Execution Events

Event When Payload Keys
create.schedule_execution.postburner When execution is created :schedule, :execution
enqueue.schedule_execution.postburner When execution is enqueued to Beanstalkd :schedule, :execution, :beanstalk_job_id
skip.schedule_execution.postburner When execution is skipped :schedule, :execution

Execution Payload Structure:

{
  id: 42,
  schedule_id: 1,
  status: "scheduled",
  run_at: "2025-01-15T09:00:00Z",
  next_run_at: "2025-01-16T09:00:00Z",
  enqueued_at: "2025-01-14T10:00:00Z",
  beanstalk_job_id: 789,
  job_id: 123
}

Watchdog Events (Worker Level)

Emitted by the worker when executing the watchdog job from Beanstalkd:

Event When Payload Keys
perform_start.watchdog.postburner Before watchdog job execution begins :beanstalk_job_id, :interval, :watchdog, :gc_count, :gc_limit
perform.watchdog.postburner Around watchdog job execution (includes duration) :beanstalk_job_id, :interval, :watchdog, :gc_count, :gc_limit

Watchdog Payload Structure:

{
  beanstalk_job_id: 789,       # Beanstalkd job ID
  interval: 300,               # Scheduler interval in seconds
  watchdog: true,              # Identifier flag
  gc_count: 42,                # Jobs processed (nil if unavailable)
  gc_limit: 5000               # GC limit for worker (nil if unlimited)
}

Scheduler Events (Scheduler Level)

Emitted by the Scheduler class when processing schedules (nested within watchdog execution):

Event When Payload Keys
perform_start.scheduler.postburner Before scheduler processes schedules :interval
perform.scheduler.postburner Around scheduler run (includes summary) :interval, :lock_acquired, :schedules_processed, :schedules_failed, :executions_created, :orphans_enqueued

Scheduler Payload Structure:

{
  interval: 300,               # Scheduler interval in seconds
  lock_acquired: true,         # Whether advisory lock was acquired
  schedules_processed: 5,      # Number of schedules processed
  schedules_failed: 0,         # Number of schedules that failed
  executions_created: 3,       # Number of executions created
  orphans_enqueued: 1          # Number of orphaned executions enqueued
}

Subscribing to Events

# config/initializers/postburner_instrumentation.rb

# Log all job executions
ActiveSupport::Notifications.subscribe('perform.job.postburner') do |name, start, finish, id, payload|
  duration = (finish - start) * 1000
  Rails.logger.info "[Postburner] #{payload[:job][:class]} completed in #{duration.round(2)}ms"
end

# Alert on discarded jobs
ActiveSupport::Notifications.subscribe('discard.job.postburner') do |*args|
  payload = args.last
  Alerting.notify(
    "Job discarded after max retries",
    job_class: payload[:job][:class],
    error: payload[:error].message
  )
end

# Track retry metrics
ActiveSupport::Notifications.subscribe('retry.job.postburner') do |*args|
  payload = args.last
  StatsD.increment('postburner.retry', tags: [
    "job:#{payload[:job][:class]}",
    "attempt:#{payload[:attempt]}"
  ])
end

# Monitor watchdog job execution (worker level)
ActiveSupport::Notifications.subscribe('perform.watchdog.postburner') do |name, start, finish, id, payload|
  duration = (finish - start) * 1000
  Rails.logger.info "[Watchdog] Job completed in #{duration.round(2)}ms " \
                    "(interval: #{payload[:interval]}s, gc_count: #{payload[:gc_count]}/#{payload[:gc_limit]})"
end

# Monitor scheduler processing (scheduler level, nested within watchdog)
ActiveSupport::Notifications.subscribe('perform.scheduler.postburner') do |name, start, finish, id, payload|
  duration = (finish - start) * 1000
  Rails.logger.info "[Scheduler] Processed #{payload[:schedules_processed]} schedules, " \
                    "created #{payload[:executions_created]} executions in #{duration.round(2)}ms"
end

Note: These events complement (don't replace) ActiveJob's built-in instrumentation events like enqueue.active_job and perform.active_job.

Logging

Configuration

Set the log level in your Rails environment configuration:

# config/environments/production.rb
config.log_level = :info  # Default, recommended for production

# config/environments/development.rb
config.log_level = :debug  # Verbose logging for development

Postburner uses Rails.logger, so standard Rails log level configuration applies.

Debugging

Set the log level to debug for verbose logging:

config.log_level = :debug

Custom Job Logging

class ProcessPayment < ApplicationJob
  include Postburner::Tracked

  def perform(payment_id)
    log "Starting payment processing for $#{payment.amount}"  # Stored in database
    payment.charge!
    log! "Payment charged successfully"  # Saved immediately
  end
end

These job-specific logs are stored in the database (for tracked jobs only) and are separate from Rails application logs, providing a complete audit trail for critical operations.

Why Beanstalkd?

Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.

The protocol reads more like a README than a protocol. Check it out and you will instantly understand how it works.

Diagram of the typical job lifecycle:

   put            reserve               delete
  -----> [READY] ---------> [RESERVED] --------> *poof*`

Diagram with all lifecycle:

   put with delay               release with delay
  ----------------> [DELAYED] <------------.
                        |                   |
                        | (time passes)     |
                        |                   |
   put                  v     reserve       |       delete
  -----------------> [READY] ---------> [RESERVED] --------> *poof*
                       ^  ^                |  |
                       |   \  release      |  |
                       |    `-------------'   |
                       |                      |
                       | kick                 |
                       |                      |
                       |       bury           |
                    [BURIED] <---------------'
                       |
                       |  delete
                        `--------> *poof*

Binlogs

Beanstalkd lets you persist jobs to disk in case of a crash or restart. Just restart beanstalkd and the jobs will be back in the queue.

# Create binlog directory
sudo mkdir -p /var/lib/beanstalkd
sudo chown beanstalkd:beanstalkd /var/lib/beanstalkd  # If running as service

# Start beanstalkd with binlog persistence
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd

Other options:

# Basic persistence
beanstalkd -b /var/lib/beanstalkd

# With custom binlog file size (default varies)
beanstalkd -b /var/lib/beanstalkd -s 10485760  # 10MB per binlog file

# Reduce fsync frequency for better performance (trades safety for speed)
beanstalkd -b /var/lib/beanstalkd -f 200  # fsync at most every 200ms (default: 50ms)

# Never fsync (maximum performance, higher risk of data loss on power failure)
beanstalkd -b /var/lib/beanstalkd -F

Beanstalkd switches:

  • -b <dir> - Enable binlog persistence in specified directory
  • -s <bytes> - Maximum size of each binlog file (requires -b)
  • -f <ms> - Call fsync at most once every N milliseconds (requires -b, default: 50ms)
  • -F - Never call fsync (requires -b, maximum performance but higher data loss risk)

Note: The fsync options control the trade-off between performance and safety. A power failure could result in loss of jobs added within the fsync interval.

Priority

Priority controls the order in which jobs are processed from the queue. Beanstalkd always processes the highest priority (lowest number) jobs first.

Priority Range: 0 to 4,294,967,295 (unsigned 32-bit integer)

  • 0 = Highest priority (processed first)
  • 4,294,967,295 = Lowest priority (processed last)
  • Default: 65536 (configurable in config/postburner.yml)

When a worker reserves a job, Beanstalkd returns the highest priority job available in the tube. Jobs with the same priority are processed in FIFO order.

ActiveJob with Postburner::Beanstalkd:

class ProcessPayment < ApplicationJob
  include Postburner::Beanstalkd

  queue_as :critical
  priority 0  # Highest priority - processed immediately
end

class SendEmail < ApplicationJob
  include Postburner::Beanstalkd

  queue_as :mailers
  priority 1000  # Lower priority - processed after critical jobs
end

Postburner::Job:

class CriticalJob < Postburner::Job
  queue 'critical'
  priority 0  # Highest priority

  def perform(args)
    # Critical business logic
  end
end

class BackgroundTask < Postburner::Job
  queue 'default'
  priority 5000  # Lower priority

  def perform(args)
    # Non-urgent background work
  end
end

Example Priority Ranges:

Priority Range Use Case Examples
0-256 Critical business operations Payments, transactions, real-time notifications
256-4096 High priority tasks Password resets, order confirmations
4096-65536 Standard background jobs Email sending, data exports (65536 is default)
65536-262144 Low priority maintenance Cache warming, log cleanup
262144+ Deferred/bulk operations Analytics, batch reports

Configuration:

Set default priority in config/postburner.yml:

production:
  default_queue: default           # Queue for jobs (defaults to 'default')
  default_mailer_queue: mailers    # Queue for Postburner::Mailer (defaults to default_queue)
  default_priority: 65536          # Default without explicit priority set
  default_ttr: 300

Dynamic Priority (Postburner::Job only):

class ProcessOrder < Postburner::Job
  queue 'orders'
  priority 100  # Default

  before_enqueue :adjust_priority

  def perform(args)
    order = Order.find(args['order_id'])
    order.process!
  end

  private

  def adjust_priority
    if args['urgent']
      self.priority = 0  # Override to highest priority
    end
  end
end

# Usage
ProcessOrder.create!(args: { 'order_id' => 123, 'urgent' => true }).queue!

Time to Run (TTR)

TTR (Time-to-Run) is the maximum duration in seconds that a worker has to process a reserved job before Beanstalkd automatically releases it back to the ready queue.

  1. Worker reserves a job from Beanstalkd
  2. TTR countdown begins immediately upon reservation
  3. If job completes within TTR, worker deletes/buries the job (success)
  4. If TTR expires before completion, Beanstalkd automatically releases the job back to ready queue
  5. Another worker can then reserve and process the same job

This mechanism protects against workers crashing or hanging—jobs won't be stuck indefinitely.

TTR Range: 1 to 4,294,967,295 seconds (unsigned 32-bit integer)

  • Minimum: 1 second (Beanstalkd silently increases 0 to 1)
  • Default: 300 seconds (5 minutes, configurable in config/postburner.yml)

ActiveJob with Postburner::Beanstalkd:

class ProcessPayment < ApplicationJob
  include Postburner::Beanstalkd # Or Postburner::Tracked

  queue_as :critical
  priority 0
  ttr 300  # 5 minutes to complete
end

class QuickEmail < ApplicationJob
  include Postburner::Beanstalkd # Or Postburner::Tracked

  queue_as :mailers
  ttr 60  # 1 minute for fast email jobs
end

class LongRunningReport < ApplicationJob
  include Postburner::Beanstalkd # Or Postburner::Tracked

  queue_as :reports
  ttr 3600  # 1 hour for complex reports
end

Postburner::Job:

class DataImport < Postburner::Job
  queue 'imports'
  ttr 1800  # 30 minutes

  def perform(args)
    # Long-running import logic
  end
end

Extending TTR with extend! (Touch Command):

For jobs that may take longer than expected, you can extend the TTR dynamically using extend!. This calls Beanstalkd's touch command, which resets the TTR countdown.

Available for:

  • Postburner::Job subclasses (via extend! method)
  • Postburner::Tracked ActiveJob classes (via extend! method)
class ProcessImport < ApplicationJob
  include Postburner::Tracked  # Includes Beanstalkd and enables extend!

  queue_as :imports
  ttr 300  # 5 minutes initial TTR

  def perform(file_id)
    file = ImportFile.find(file_id)

    file.each_batch(size: 100) do |batch|
      batch.each do |row|
        process_row(row)
      end

      extend!  # Reset TTR to 300 seconds from now
      log "Processed batch, extended TTR"
    end
  end
end

For Postburner::Job (via bk accessor):

class LargeDataProcessor < Postburner::Job
  queue 'processing'
  ttr 600  # 10 minutes

  def perform(args)
    dataset = Dataset.find(args['dataset_id'])

    dataset.each_chunk do |chunk|
      process_chunk(chunk)

      extend!  # Reset TTR (calls bk.touch internally)
      log "Chunk processed, TTR extended"
    end
  end
end

extend! and Beanstalkd touch

  • Calls Beanstalkd's touch command on the reserved job
  • Resets the TTR countdown to the original TTR value from the current moment
  • Example: Job has ttr 300. After 200 seconds, calling extend! gives you another 300 seconds (not just 100)
  • Can be called multiple times during job execution
  • Useful for iterative processing where each iteration is quick but total time is unpredictable

Configuration:

Set default TTR in config/postburner.yml:

production:
  default_priority: 65536
  default_ttr: 300  # 5 minutes default for all jobs

Recommended TTR Values:

Job Type TTR Reasoning
Email sending 60-120s Network operations, should be fast
API calls 30-60s External services, fast or timeout
File processing 600-1800s Variable based on file size, use extend!
Reports 1800-3600s Complex queries and aggregations
Imports/Exports 3600s+ Large datasets, use extend! in loops

Best Practices:

  1. Set realistic TTR values based on expected job duration plus buffer
  2. Use extend! for iterative work where total time is unpredictable but each iteration is bounded
  3. Don't set TTR too high - it delays recovery from crashed workers
  4. Monitor TTR timeouts - frequent timeouts indicate jobs need more time or optimization
  5. For Default jobs (ActiveJob without Postburner::Beanstalkd), include the module to set custom TTR

What happens on TTR timeout:

# Job starts processing
job = ProcessImport.perform_later(file_id)

# Worker crashes or hangs after 200 seconds
# At 300 seconds (TTR), Beanstalkd automatically:
# 1. Releases job back to ready queue
# 2. Increments the job's reserve count
# 3. Makes job available for another worker

# Another worker picks up the job and retries
# (ActiveJob retry_on/discard_on handlers still apply)

Beanstalkd Integration

Postburner uses Beaneater as the Ruby client for Beanstalkd. You can access the underlying Beaneater connection directly for advanced operations.

Connection Methods

Postburner uses thread-local connections — each thread gets its own Beanstalkd socket, matching the pattern used by worker threads. This is safe under multi-threaded servers like Puma.

# Get the thread-local Beaneater connection (returns Beaneater instance)
conn = Postburner.connection
conn.tubes.to_a      # List all tubes
conn.beanstalk.stats # Server statistics

# Block form - yields thread-local connection
Postburner.connected do |conn|
  conn.tubes.to_a  # List all tubes
  conn.tubes['postburner.production.critical'].stats
  conn.tubes['postburner.production.critical'].kick(10)  # Kick 10 buried jobs
end

Shutdown cleanup:

On process shutdown, call disconnect_all! to close every thread's connection cleanly:

# config/puma.rb
on_worker_shutdown do
  Postburner.disconnect_all!
end

This is optional — Ruby closes sockets on process exit — but gives you clean logs and explicit teardown.

Job-Level Access

# Get Beanstalkd job ID
job.bkid  # => 12345

# Access Beaneater job object
job.bk.stats
# => {"id"=>12345, "tube"=>"critical", "state"=>"ready", ...}

Beaneater API

The connection object is a standard Beaneater instance. See the Beaneater documentation for full API details:

Postburner.connected do |conn|
  # Tubes
  conn.tubes.to_a                    # List all tube names
  conn.tubes['my-tube'].stats        # Tube statistics
  conn.tubes['my-tube'].peek(:ready) # Peek at next ready job
  conn.tubes['my-tube'].kick(10)     # Kick 10 buried jobs

  # Server
  conn.stats                         # Server statistics

  # Jobs
  conn.jobs.find(12345)              # Find job by ID
end

Tube Statistics and Management

Postburner provides methods to inspect and manage Beanstalkd tubes.

Note: Beanstalkd tubes are created lazily - they only exist when jobs have been put into them. This means configured queues (e.g., default, mailers) won't appear in stats until at least one job has been enqueued to them. The scheduler tube appears immediately because the watchdog job is created on worker startup.

Example worker startup output:

[Postburner::Worker] Starting worker 'default'...
[Postburner::Worker] Queues: default, mailers
[Postburner::Worker] Startup stats: ready=1 delayed=0 buried=0 reserved=0 total=1
  postburner.development.scheduler: ready=1 delayed=0 buried=0 reserved=0 total=1

Note that default and mailers queues are configured but don't appear in stats yet - they'll appear once jobs are enqueued to them.

View tube statistics:

# View all tubes on the Beanstalkd server
stats = Postburner.stats
# => {
#   tubes: [
#     { name: "postburner.production.default", ready: 10, delayed: 5, buried: 0, reserved: 2, total: 17 },
#     { name: "postburner.production.critical", ready: 0, delayed: 0, buried: 0, reserved: 1, total: 1 }
#   ],
#   totals: { ready: 10, delayed: 5, buried: 0, reserved: 3, total: 18 }
# }

# View specific tubes only
stats = Postburner.stats(['postburner.production.critical'])
# => { tubes: [...], totals: {...} }

Understanding lazy tube creation:

# Fresh Beanstalkd, no jobs queued yet - only scheduler tube exists
stats = Postburner.stats
# => {
#   tubes: [
#     { name: "postburner.development.scheduler", ready: 1, delayed: 0, ... }
#   ],
#   totals: { ready: 1, delayed: 0, buried: 0, reserved: 0, total: 1 }
# }
# Note: 'default' and 'mailers' tubes don't appear yet!

# After queueing a job to 'default':
MyJob.perform_later(123)
stats = Postburner.stats
# => {
#   tubes: [
#     { name: "postburner.development.default", ready: 1, ... },
#     { name: "postburner.development.scheduler", ready: 1, ... }
#   ],
#   totals: { ready: 2, ... }
# }

Clear jobs from tubes:

For safety, clear_jobs! requires you to explicitly specify which tubes to clear. This prevents accidentally clearing tubes from other applications sharing the same Beanstalkd server.

# Collect stats only (no clearing)
result = Postburner.clear_jobs!
# => { tubes: [...], totals: {...}, cleared: false }

# Clear specific tubes (must be in config/postburner.yml)
result = Postburner.clear_jobs!(['postburner.production.default'])
# => { tubes: [...], totals: {...}, cleared: true }

# Pretty-print JSON output
Postburner.clear_jobs!(['postburner.production.default'], silent: false)
# OR
Postburner.clear_jobs!(['postburner.production.default'])
# Outputs formatted JSON to stdout

# Silent mode (no output, just return data)
result = Postburner.clear_jobs!(['postburner.production.default'], silent: true)

Shortcut using watched_tube_names or scheduler_tube_name:

Clear all configured tubes at once:

Postburner.clear_jobs!(Postburner.watched_tube_names)
# => { tubes: [...], totals: {...}, cleared: true }

Or clear scheduler tube, specifically:

Postburner.clear_jobs!(Postburner.scheduler_tube_name)
# => { tubes: [...], totals: {...}, cleared: true }

These shortcuts are useful for testing and development, but not recommended for production.

Safety validation:

Only tubes defined in your loaded configuration can be cleared. This prevents mistakes in multi-tenant Beanstalkd environments:

# Error: trying to clear tube not in config
Postburner.clear_jobs!(['postburner.production.other-app'])
# => ArgumentError: Cannot clear tubes not in configuration.
#    Invalid tubes: postburner.production.other-app
#    Configured tubes: postburner.production.default, postburner.production.critical

Low-level Connection API:

For programmatic use without output formatting, use Connection#clear_tubes!:

Postburner.connected do |conn|
  # Returns data only (no puts)
  result = conn.clear_tubes!(Postburner.watched_tube_names)
  # => { tubes: [...], totals: {...}, cleared: true }

  # Same validation - must be in configuration
  result = conn.clear_tubes!(['postburner.production.default'])
end

Web UI - v2 Coming Soon

Mount the inspection interface:

# config/routes.rb
mount Postburner::Engine => "/postburner"

# Only in development
mount Postburner::Engine => "/postburner" if Rails.env.development?

Add your own authentication:

# config/routes.rb
authenticate :user, ->(user) { user.admin? } do
  mount Postburner::Engine => "/postburner"
end

Installation

1. Install Beanstalkd

First install beanstalkd:

sudo apt-get install beanstalkd
brew install beanstalkd # for macOS/Linux

# Start beanstalkd (in-memory only)
beanstalkd -l 127.0.0.1 -p 11300

# OR with persistence (recommended for production)
mkdir -p /var/lib/beanstalkd
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd

2. Add Gem

# Gemfile
gem 'postburner', '~> 1.0.0.rc.2

bundle install

3. Install Migration

rails generate postburner:install
rails db:migrate

This creates the postburner_jobs table for tracked jobs.

4. Configure ActiveJob

# config/application.rb
config.active_job.queue_adapter = :postburner

5. Create Worker Configuration

cp config/postburner.yml.example config/postburner.yml

Edit config/postburner.yml for your environment (see Configuration).

Migration from v0.x

Key changes in v1.0:

Changed

  • Postburner::Job#backburner_job -> Postburner::Job#bk
  • queue_priority → priority
  • queue_ttr → ttr
  • queue_max_job_retries → max_retries
  • queue_retry_delay → retry_delay

Removed

  • Backburner dependency
  • config/initializers/backburner.rb
  • Backburner worker (bundle exec backburner)

Added

  • ActiveJob adapter (first-class Rails integration)
  • Default jobs (Beanstalkd only, no PostgreSQL)
  • bin/postburner executable
  • config/postburner.yml configuration
  • Multiple worker types (Simple, Forking, ThreadsOnFork)

Migration Steps

  1. Update Gemfile:

    gem 'postburner', '~> 1.0.0.pre.18'

  2. Remove Backburner config:

    rm config/initializers/backburner.rb

  3. Create Postburner config:

    cp config/postburner.yml.example config/postburner.yml

  4. Update ActiveJob adapter:

    # config/application.rb
config.active_job.queue_adapter = :postburner  # was :backburner

  5. Update worker command:

    # Old
bundle exec backburner

# New bundle exec postburner --config config/postburner.yml


6. **Existing Jobs:** Direct `Postburner::Job` subclasses continue to work without changes!

## Contributing

Submit a pull request. Follow project conventions. Be nice.

There is a CLAUDE.md file for guidance when using Claude Code. Please use it or contribute to it. Do NOT reference AI tools in commits or code. Do not co-author with AI tools. Pull requests will be rejected if they violate these rules.

We encourage AI tools, but do not vibe, as the code must look like it was written by a human. Code that contains AI agent idioms will be rejected. Code that doesn't follow the project conventions will be rejected.


### Testing

```bash
bundle install
bundle exec rails test # must have beanstalkd on 11300 by default
bundle exec rails app:postburner:work # run worker from engine root (uses test/dummy app)

License

The gem is available as open source under the terms of the MIT License.