Module: Restate::Middleware::DeadlockDetection

Defined in:
lib/restate/middleware/deadlock_detection.rb

Overview

Detects VirtualObject deadlocks caused by re-entrant calls to a VO whose exclusive handler is still running higher up the call chain.

The problem

Restate VirtualObjects serialize exclusive handler access per key. If handler A on VO key “x” calls handler B on the same VO key “x”, the call will block forever — the key is already locked by A. This is a deadlock.

How it works

This middleware tracks which VO keys are held by the current call chain and propagates that information via a header on every outbound call.

Inbound side

  1. Reads the held-locks header from the incoming request.

  2. If the current handler is an exclusive VO handler targeting a key already in the set → deadlock. Raises a DeadlockError immediately.

  3. If this handler is an exclusive VO handler, appends its lock to the set so further downstream calls propagate it.

Outbound side

Injects the held-locks header into every outbound service call. When handler metadata is available (the target service class is known), only raises for exclusive handlers — shared handler calls are safe. Falls back to raising for any same-service call when metadata is unavailable (e.g., calling by string name to an external service).

Wire format

Lock entries are encoded as base64url(service).base64url(key) and separated by commas. Base64url encoding ensures arbitrary service names and keys (including those containing ., ,, or non-ASCII characters) are handled correctly.

Journal determinism

The held-locks header is deterministic across replays: its value depends only on the execution path, which Restate’s journal guarantees is identical on every replay.

Usage

endpoint = Restate.endpoint(MyVirtualObject)
endpoint.use(Restate::Middleware::DeadlockDetection::Inbound)
endpoint.use_outbound(Restate::Middleware::DeadlockDetection::Outbound)

Defined Under Namespace

Classes: DeadlockError, Inbound, Outbound

Constant Summary collapse

HEADER = 
'x-restate-held-locks'
ENTRY_SEPARATOR = 
','
FIELD_SEPARATOR = 
'.'
DEADLOCK_STATUS_CODE = 
409
THREAD_KEY = 
:restate_held_exclusive_locks

Class Method Summary collapse

Class Method Details

.decode_header(raw) ⇒ Object

Deserializes a header value into a Set of [service, key] pairs.



106
107
108
109
110
111
112
113
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 106

def decode_header(raw)
  return Set.new if raw.nil? || raw.to_s.empty?

  entries = raw.to_s.split(ENTRY_SEPARATOR).filter_map do |entry|
    decode_lock(entry.strip)
  end
  Set.new(entries)
end

.decode_lock(encoded) ⇒ Object

Decodes a wire-format lock string into [service, key]. Returns nil if the format is invalid.



89
90
91
92
93
94
95
96
97
98
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 89

def decode_lock(encoded)
  parts = encoded.split(FIELD_SEPARATOR, 2)
  return nil unless parts.length == 2

  svc = Base64.urlsafe_decode64(parts[0]).force_encoding('UTF-8')
  key = Base64.urlsafe_decode64(parts[1]).force_encoding('UTF-8')
  [svc, key]
rescue ArgumentError
  nil
end

.encode_header(locks) ⇒ Object

Serializes a set of [service, key] lock pairs into a header value.



101
102
103
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 101

def encode_header(locks)
  locks.map { |svc, key| encode_lock(svc, key) }.join(ENTRY_SEPARATOR)
end

.encode_lock(service, key) ⇒ Object

Encodes a [service, key] pair into a wire-safe string.



81
82
83
84
85
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 81

def encode_lock(service, key)
  b64_svc = Base64.urlsafe_encode64(service, padding: false)
  b64_key = Base64.urlsafe_encode64(key, padding: false)
  "#{b64_svc}#{FIELD_SEPARATOR}#{b64_key}"
end

.held_locks ⇒ Set<Array<String>>

Returns the current set of held exclusive locks for this fiber. Each entry is a two-element array: [service_name, key].

Returns:

  • (Set<Array<String>>) 


71
72
73
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 71

def held_locks
  Thread.current[THREAD_KEY] || Set.new
end

.held_locks=(locks) ⇒ Object

Parameters:

  • locks (Set<Array<String>>) 


76
77
78
 
# File 'lib/restate/middleware/deadlock_detection.rb', line 76

def held_locks=(locks)
  Thread.current[THREAD_KEY] = locks
end