Class: Deimos::Utils::DeadlockRetry

Inherits:
Object
  • Object
show all
Defined in:
lib/deimos/utils/deadlock_retry.rb,
sig/defs.rbs

Overview

Utility class to retry a given block if a a deadlock is encountered. Supports Postgres and MySQL deadlocks and lock wait timeouts.

Constant Summary collapse

RETRY_COUNT =

Maximum number of times to retry the block after encountering a deadlock

Returns:

  • (Integer) 
2
DEADLOCK_MESSAGES =

Need to match on error messages to support older Rails versions

Returns:

  • (Array<String>) 
[
  # MySQL
  'Deadlock found when trying to get lock',
  'Lock wait timeout exceeded',

  # Postgres
  'deadlock detected'
].freeze

Class Method Summary collapse

Class Method Details

.deadlock?(error) ⇒ Boolean

Whether the given exception is a deadlock or lock wait timeout, i.e. transient contention on the database rather than a problem with the data being written.

Parameters:

  • error (Exception)

Returns:

  • (Boolean) 


28
29
30
31
 
# File 'lib/deimos/utils/deadlock_retry.rb', line 28

def deadlock?(error)
  error.is_a?(ActiveRecord::StatementInvalid) &&
    DEADLOCK_MESSAGES.any? { |m| error.message.include?(m) }
end

.wrap(tags = []) ⇒ void

This method returns an undefined value.

Retry the given block when encountering a deadlock. For any other exceptions, they are reraised. This is used to handle cases where the database may be busy but the transaction would succeed if retried later. Note that your block should be idempotent and it will be wrapped in a transaction. Sleeps for a random number of seconds to prevent multiple transactions from retrying at the same time.

@param tags — Tags to attach when logging and reporting metrics.

Parameters:

  • tags (::Array[untyped]) (defaults to: []) 


44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 
# File 'lib/deimos/utils/deadlock_retry.rb', line 44

def wrap(tags=[])
  # Normalize to an Array so downstream logging/metrics (and dogstatsd's
  # `tags.to_a`) never receive a bare String, which would raise
  # NoMethodError inside this rescue and defeat the retry mechanism.
  tags = Array(tags)
  count = RETRY_COUNT

  begin
    ActiveRecord::Base.transaction do
      yield
    end
  rescue ActiveRecord::StatementInvalid => e
    # Reraise if not a known deadlock
    raise unless deadlock?(e)

    # Reraise if all retries exhausted
    raise if count <= 0

    Deimos::Logging.log_warn(
      message: 'Deadlock encountered when trying to execute query. ' \
               "Retrying. #{count} attempt(s) remaining",
      tags: tags
    )

    Deimos.config.metrics&.increment(
      'deadlock',
      tags: tags
    )

    count -= 1

    # Sleep for a random amount so that if there are multiple
    # transactions deadlocking, they don't all retry at the same time
    sleep(Random.rand(5.0) + 0.5)

    retry
  end
end