Module: Sidekiq

Defined in:
lib/sidekiq/systemd.rb,
lib/sidekiq.rb,
lib/sidekiq/api.rb,
lib/sidekiq/cli.rb,
lib/sidekiq/job.rb,
lib/sidekiq/web.rb,
lib/sidekiq/delay.rb,
lib/sidekiq/fetch.rb,
lib/sidekiq/rails.rb,
lib/sidekiq/client.rb,
lib/sidekiq/logger.rb,
lib/sidekiq/worker.rb,
lib/sidekiq/manager.rb,
lib/sidekiq/testing.rb,
lib/sidekiq/version.rb,
lib/sidekiq/job_util.rb,
lib/sidekiq/launcher.rb,
lib/sidekiq/component.rb,
lib/sidekiq/job_retry.rb,
lib/sidekiq/paginator.rb,
lib/sidekiq/processor.rb,
lib/sidekiq/scheduled.rb,
lib/sidekiq/sd_notify.rb,
lib/sidekiq/job_logger.rb,
lib/sidekiq/web/action.rb,
lib/sidekiq/web/router.rb,
lib/sidekiq/ring_buffer.rb,
lib/sidekiq/web/helpers.rb,
lib/sidekiq/web/application.rb,
lib/sidekiq/middleware/chain.rb,
lib/sidekiq/redis_connection.rb,
lib/sidekiq/middleware/modules.rb,
lib/sidekiq/web/csrf_protection.rb,
lib/sidekiq/redis_client_adapter.rb,
lib/generators/sidekiq/job_generator.rb,
lib/sidekiq/extensions/action_mailer.rb,
lib/sidekiq/extensions/active_record.rb,
lib/sidekiq/extensions/class_methods.rb,
lib/sidekiq/extensions/generic_proxy.rb,
lib/sidekiq/transaction_aware_client.rb,
lib/sidekiq/transaction_aware_client.rb,
lib/sidekiq/middleware/current_attributes.rb

Overview

Use `Sidekiq.transactional_push!` in your sidekiq.rb initializer

Defined Under Namespace

Modules: Component, Context, CurrentAttributes, Extensions, Generators, Job, JobUtil, LoggingUtils, Middleware, Paginator, Queues, RedisConnection, Scheduled, SdNotify, ServerMiddleware, TestingClient, TestingExtensions, WebHelpers, WebRouter, Worker Classes: BasicFetch, CLI, Client, DeadSet, EmptyQueueError, JobLogger, JobRecord, JobRetry, JobSet, Launcher, Logger, Manager, Monitor, Process, ProcessSet, Processor, Queue, Rails, RedisClientAdapter, RetrySet, RingBuffer, ScheduledSet, Shutdown, SortedEntry, SortedSet, Stats, Testing, TransactionAwareClient, Web, WebAction, WebApplication, WebRoute, WorkSet

Constant Summary collapse

NAME = 
"Sidekiq"
LICENSE = 
"See LICENSE and the LGPL-3.0 for licensing details."
DEFAULTS = 
{
  queues: [],
  labels: [],
  concurrency: 10,
  require: ".",
  strict: true,
  environment: nil,
  timeout: 25,
  poll_interval_average: nil,
  average_scheduled_poll_interval: 5,
  on_complex_arguments: :warn,
  error_handlers: [],
  death_handlers: [],
  lifecycle_events: {
    startup: [],
    quiet: [],
    shutdown: [],
    heartbeat: []
  },
  dead_max_jobs: 10_000,
  dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months
  reloader: proc { |&block| block.call }
}
FAKE_INFO = 
{
  "redis_version" => "9.9.9",
  "uptime_in_days" => "9999",
  "connected_clients" => "9999",
  "used_memory_human" => "9P",
  "used_memory_peak_human" => "9P"
}
Workers =

Since “worker” is a nebulous term, we've deprecated the use of this class name. Is “worker” a process, a type of job, a thread? Undefined! WorkSet better describes the data.

WorkSet
VERSION = 
"6.5.0"
ClientMiddleware =

no difference for now

ServerMiddleware

Class Method Summary collapse

Class Method Details

.[](key) ⇒ Object 



98
99
100
 
# File 'lib/sidekiq.rb', line 98

def self.[](key)
  @config[key]
end

.[]=(key, val) ⇒ Object 



102
103
104
 
# File 'lib/sidekiq.rb', line 102

def self.[]=(key, val)
  @config[key] = val
end

.average_scheduled_poll_interval=(interval) ⇒ Object

How frequently Redis should be checked by a random Sidekiq process for scheduled and retriable jobs. Each individual process will take turns by waiting some multiple of this value.

See sidekiq/scheduled.rb for an in-depth explanation of this value



295
296
297
 
# File 'lib/sidekiq.rb', line 295

def self.average_scheduled_poll_interval=(interval)
  self[:average_scheduled_poll_interval] = interval
end

.client_middleware {|@client_chain| ... } ⇒ Object

Yields:

  • (@client_chain) 


201
202
203
204
205
 
# File 'lib/sidekiq.rb', line 201

def self.client_middleware
  @client_chain ||= Middleware::Chain.new(self)
  yield @client_chain if block_given?
  @client_chain
end

.concurrency=(val) ⇒ Object

config.concurrency = 5



57
58
59
 
# File 'lib/sidekiq.rb', line 57

def self.concurrency=(val)
  self[:concurrency] = Integer(val)
end

.configure_client {|_self| ... } ⇒ Object

Configuration for Sidekiq client, use like:

Sidekiq.configure_client do |config|
  config.redis = { size: 1, url: 'redis://myhost:8877/0' }
end

Yields:

  • (_self)

Yield Parameters:

  • _self (Sidekiq) —

    the object that the method was called on



143
144
145
 
# File 'lib/sidekiq.rb', line 143

def self.configure_client
  yield self unless server?
end

.configure_server {|_self| ... } ⇒ Object

Configuration for Sidekiq server, use like:

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add MyServerHook
  end
end

Yields:

  • (_self)

Yield Parameters:

  • _self (Sidekiq) —

    the object that the method was called on



133
134
135
 
# File 'lib/sidekiq.rb', line 133

def self.configure_server
  yield self if server?
end

.death_handlers ⇒ Object

Death handlers are called when all retries for a job have been exhausted and the job dies. It's the notification to your application that this job will not succeed without manual intervention.

Sidekiq.configure_server do |config|

config.death_handlers << ->(job, ex) do
end

end



242
243
244
 
# File 'lib/sidekiq.rb', line 242

def self.death_handlers
  self[:death_handlers]
end

.default_error_handler(ex, ctx) ⇒ Object

Private APIs



81
82
83
84
85
 
# File 'lib/sidekiq.rb', line 81

def self.default_error_handler(ex, ctx)
  logger.warn(dump_json(ctx)) unless ctx.empty?
  logger.warn("#{ex.class.name}: #{ex.message}")
  logger.warn(ex.backtrace.join("\n")) unless ex.backtrace.nil?
end

.default_job_options ⇒ Object 



229
230
231
 
# File 'lib/sidekiq.rb', line 229

def self.default_job_options
  @default_job_options ||= {"retry" => true, "queue" => "default"}
end

.default_job_options=(hash) ⇒ Object 



221
222
223
 
# File 'lib/sidekiq.rb', line 221

def self.default_job_options=(hash)
  @default_job_options = default_job_options.merge(hash.transform_keys(&:to_s))
end

.default_server_middleware ⇒ Object 



213
214
215
 
# File 'lib/sidekiq.rb', line 213

def self.default_server_middleware
  Middleware::Chain.new(self)
end

.default_worker_options ⇒ Object

deprecated



225
226
227
 
# File 'lib/sidekiq.rb', line 225

def self.default_worker_options # deprecated
  @default_job_options ||= {"retry" => true, "queue" => "default"}
end

.default_worker_options=(hash) ⇒ Object

deprecated



217
218
219
 
# File 'lib/sidekiq.rb', line 217

def self.default_worker_options=(hash) # deprecated
  @default_job_options = default_job_options.merge(hash.transform_keys(&:to_s))
end

.dump_json(object) ⇒ Object 



250
251
252
 
# File 'lib/sidekiq.rb', line 250

def self.dump_json(object)
  JSON.generate(object)
end

.ent? ⇒ Boolean

Returns:

  • (Boolean) 


286
287
288
 
# File 'lib/sidekiq.rb', line 286

def self.ent?
  defined?(Sidekiq::Enterprise)
end

.error_handlers ⇒ Object

Register a proc to handle any error which occurs within the Sidekiq process.

Sidekiq.configure_server do |config|
  config.error_handlers << proc {|ex,ctx_hash| MyErrorService.notify(ex, ctx_hash) }
end

The default error handler logs errors to Sidekiq.logger.



306
307
308
 
# File 'lib/sidekiq.rb', line 306

def self.error_handlers
  self[:error_handlers]
end

.fetch(*args, &block) ⇒ Object 



110
111
112
 
# File 'lib/sidekiq.rb', line 110

def self.fetch(*args, &block)
  @config.fetch(*args, &block)
end

.handle_exception(ex, ctx = {}) ⇒ Object 



114
115
116
117
118
119
120
121
122
 
# File 'lib/sidekiq.rb', line 114

def self.handle_exception(ex, ctx = {})
  self[:error_handlers].each do |handler|
    handler.call(ex, ctx)
  rescue => ex
    logger.error "!!! ERROR HANDLER THREW AN ERROR !!!"
    logger.error ex
    logger.error ex.backtrace.join("\n") unless ex.backtrace.nil?
  end
end

.load_json(string) ⇒ Object 



246
247
248
 
# File 'lib/sidekiq.rb', line 246

def self.load_json(string)
  JSON.parse(string)
end

.log_formatter ⇒ Object 



254
255
256
257
258
259
260
 
# File 'lib/sidekiq.rb', line 254

def self.log_formatter
  @log_formatter ||= if ENV["DYNO"]
    Sidekiq::Logger::Formatters::WithoutTimestamp.new
  else
    Sidekiq::Logger::Formatters::Pretty.new
  end
end

.log_formatter=(log_formatter) ⇒ Object 



262
263
264
265
 
# File 'lib/sidekiq.rb', line 262

def self.log_formatter=(log_formatter)
  @log_formatter = log_formatter
  logger.formatter = log_formatter
end

.logger ⇒ Object 



267
268
269
 
# File 'lib/sidekiq.rb', line 267

def self.logger
  @logger ||= Sidekiq::Logger.new($stdout, level: :info)
end

.logger=(logger) ⇒ Object 



271
272
273
274
275
276
277
278
279
280
 
# File 'lib/sidekiq.rb', line 271

def self.logger=(logger)
  if logger.nil?
    self.logger.level = Logger::FATAL
    return self.logger
  end

  logger.extend(Sidekiq::LoggingUtils)

  @logger = logger
end

.merge!(hash) ⇒ Object 



106
107
108
 
# File 'lib/sidekiq.rb', line 106

def self.merge!(hash)
  @config.merge!(hash)
end

.on(event, &block) ⇒ Object

Register a block to run at a point in the Sidekiq lifecycle. :startup, :quiet or :shutdown are valid events.

Sidekiq.configure_server do |config|
  config.on(:shutdown) do
    puts "Goodbye cruel world!"
  end
end

Raises:

  • (ArgumentError) 


318
319
320
321
322
 
# File 'lib/sidekiq.rb', line 318

def self.on(event, &block)
  raise ArgumentError, "Symbols only please: #{event}" unless event.is_a?(Symbol)
  raise ArgumentError, "Invalid event name: #{event}" unless self[:lifecycle_events].key?(event)
  self[:lifecycle_events][event] << block
end

.options ⇒ Object 



88
89
90
91
 
# File 'lib/sidekiq.rb', line 88

def self.options
  logger.warn "`config.options[:key] = value` is deprecated, use `config[:key] = value`: #{caller(1..2)}"
  @config
end

.options=(opts) ⇒ Object 



93
94
95
96
 
# File 'lib/sidekiq.rb', line 93

def self.options=(opts)
  logger.warn "config.options = hash` is deprecated, use `config.merge!(hash)`: #{caller(1..2)}"
  @config = opts
end

.pro? ⇒ Boolean

Returns:

  • (Boolean) 


282
283
284
 
# File 'lib/sidekiq.rb', line 282

def self.pro?
  defined?(Sidekiq::Pro)
end

.queues=(val) ⇒ Object

config.queues = %w( high default low ) # strict config.queues = %w( high,3 default,2 low,1 ) # weighted config.queues = %w( feature1,1 feature2,1 feature3,1 ) # random

With weighted priority, queue will be checked first (weight / total) of the time. high will be checked first (3/6) or 50% of the time. I'd recommend setting weights between 1-10. Weights in the hundreds or thousands are ridiculous and unnecessarily expensive. You can get random queue ordering by explicitly setting all weights to 1.



70
71
72
73
74
75
76
77
78
 
# File 'lib/sidekiq.rb', line 70

def self.queues=(val)
  self[:queues] = Array(val).each_with_object([]) do |qstr, memo|
    name, weight = qstr.split(",")
    self[:strict] = false if weight.to_i > 0
    [weight.to_i, 1].max.times do
      memo << name
    end
  end
end

.redis ⇒ Object

Raises:

  • (ArgumentError) 


151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/sidekiq.rb', line 151

def self.redis
  raise ArgumentError, "requires a block" unless block_given?
  redis_pool.with do |conn|
    retryable = true
    begin
      yield conn
    rescue RedisConnection.adapter::BaseError => ex
      # 2550 Failover can cause the server to become a replica, need
      # to disconnect and reopen the socket to get back to the primary.
      # 4495 Use the same logic if we have a "Not enough replicas" error from the primary
      # 4985 Use the same logic when a blocking command is force-unblocked
      # The same retry logic is also used in client.rb
      if retryable && ex.message =~ /READONLY|NOREPLICAS|UNBLOCKED/
        conn.disconnect!
        retryable = false
        retry
      end
      raise
    end
  end
end

.redis=(hash) ⇒ Object 



193
194
195
196
197
198
199
 
# File 'lib/sidekiq.rb', line 193

def self.redis=(hash)
  @redis = if hash.is_a?(ConnectionPool)
    hash
  else
    RedisConnection.create(hash)
  end
end

.redis_info ⇒ Object 



173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
 
# File 'lib/sidekiq.rb', line 173

def self.redis_info
  redis do |conn|
    # admin commands can't go through redis-namespace starting
    # in redis-namespace 2.0
    if conn.respond_to?(:namespace)
      conn.redis.info
    else
      conn.info
    end
  rescue RedisConnection.adapter::CommandError => ex
    # 2850 return fake version when INFO command has (probably) been renamed
    raise unless /unknown command/.match?(ex.message)
    FAKE_INFO
  end
end

.redis_pool ⇒ Object 



189
190
191
 
# File 'lib/sidekiq.rb', line 189

def self.redis_pool
  @redis ||= RedisConnection.create
end

.server? ⇒ Boolean

Returns:

  • (Boolean) 


147
148
149
 
# File 'lib/sidekiq.rb', line 147

def self.server?
  defined?(Sidekiq::CLI)
end

.server_middleware {|@server_chain| ... } ⇒ Object

Yields:

  • (@server_chain) 


207
208
209
210
211
 
# File 'lib/sidekiq.rb', line 207

def self.server_middleware
  @server_chain ||= default_server_middleware
  yield @server_chain if block_given?
  @server_chain
end

.start_watchdog ⇒ Object 



8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
# File 'lib/sidekiq/systemd.rb', line 8

def self.start_watchdog
  usec = Integer(ENV["WATCHDOG_USEC"])
  return Sidekiq.logger.error("systemd Watchdog too fast: " + usec) if usec < 1_000_000

  sec_f = usec / 1_000_000.0
  # "It is recommended that a daemon sends a keep-alive notification message
  # to the service manager every half of the time returned here."
  ping_f = sec_f / 2
  Sidekiq.logger.info "Pinging systemd watchdog every #{ping_f.round(1)} sec"
  Thread.new do
    loop do
      sleep ping_f
      Sidekiq::SdNotify.watchdog
    end
  end
end

.strict_args!(mode = :raise) ⇒ Object 



324
325
326
 
# File 'lib/sidekiq.rb', line 324

def self.strict_args!(mode = :raise)
  self[:on_complex_arguments] = mode
end

.transactional_push! ⇒ Object 



33
34
35
36
37
38
39
40
41
42
43
44
 
# File 'lib/sidekiq/transaction_aware_client.rb', line 33

def self.transactional_push!
  begin
    require "after_commit_everywhere"
  rescue LoadError
    Sidekiq.logger.error("You need to add after_commit_everywhere to your Gemfile to use Sidekiq's transactional client")
    raise
  end

  default_job_options["client_class"] = Sidekiq::TransactionAwareClient
  Sidekiq::JobUtil::TRANSIENT_ATTRIBUTES << "client_class"
  true
end

.❨╯°□°❩╯︵┻━┻ ⇒ Object 



52
53
54
 
# File 'lib/sidekiq.rb', line 52

def self.❨╯°□°❩╯︵┻━┻
  puts "Calm down, yo."
end