Class: Wurk::Swarm::Restart

Inherits:
Object
  • Object
show all
Defined in:
lib/wurk/swarm/restart.rb

Overview

Non-blocking rolling-restart / recycle state machine. One slot is in flight at a time; advance moves it a single phase per supervise tick so the supervisor never blocks on a restart and keeps honoring TERM:

spawn_replacement → await_heartbeat(heartbeat_wait) → term_old
→ await_exit(drain_timeout) → done

The reaper reports child exits via claim_exit, so a replacement that dies before it heartbeats is seen as dead (not "slow"): the old child is kept, a per-slot backoff applied, and the slot retried. abort drops everything — the swarm's TERM handler then drains the in-flight replacement + old as ordinary children.

Collaborators are injected (Config) so the machine is decoupled from the swarm's fork/kill/Redis internals and unit-testable against fakes.

Defined Under Namespace

Classes: Config

Instance Method Summary collapse

Constructor Details

#initialize(config) ⇒ Restart

Returns a new instance of Restart.



24
25
26
27
28
29
30
31
32
33
34
35
36
 
# File 'lib/wurk/swarm/restart.rb', line 24

def initialize(config)
  @spawn = config.spawn            # ->(slot, idx) => replacement pid (registered by the swarm)
  @kill = config.kill              # ->(pid, sig)
  @heartbeat = config.heartbeat    # ->(pid) => truthy once the child has beaten
  @describe = config.describe      # ->(pid) => { slot:, index: } | nil
  @now = config.now                # -> monotonic seconds
  @logger = config.logger
  @heartbeat_wait = config.heartbeat_wait
  @drain_timeout = config.drain_timeout
  @backoff = config.backoff
  @queue = []
  @current = nil
end

Instance Method Details

#abort ⇒ Object

TERM/INT: forget queued + in-flight work. The replacement and old child are ordinary children the swarm's shutdown TERMs and drains.



82
83
84
85
 
# File 'lib/wurk/swarm/restart.rb', line 82

def abort
  @queue.clear
  @current = nil
end

#advance ⇒ Object 



70
71
72
73
74
75
76
77
78
 
# File 'lib/wurk/swarm/restart.rb', line 70

def advance
  start_next if @current.nil?
  return if @current.nil?

  case @current[:phase]
  when :await_heartbeat then advance_await_heartbeat
  when :await_exit then advance_await_exit
  end
end

#claim_exit(pid) ⇒ Object

Reaper hook. Returns true when pid belonged to the in-flight restart so the swarm skips crash-respawn for it. A replacement is only claimed while awaiting its heartbeat — once it takes over the slot (await_exit), its death is an ordinary crash the swarm respawns.



56
57
58
59
60
61
62
63
64
65
66
67
68
 
# File 'lib/wurk/swarm/restart.rb', line 56

def claim_exit(pid) # rubocop:disable Naming/PredicateMethod
  return false unless @current

  if pid == @current[:old_pid]
    @current[:old_exited] = true
    true
  elsif pid == @current[:replacement] && @current[:phase] == :await_heartbeat
    @current[:replacement_dead] = true
    true
  else
    false
  end
end

#enqueue(pids) ⇒ Object

Queue live child PIDs for restart, skipping any already queued or in flight so recycle + rolling restart can't double up on one slot.



44
45
46
47
48
49
50
 
# File 'lib/wurk/swarm/restart.rb', line 44

def enqueue(pids)
  pids.each do |pid|
    next if in_flight?(pid) || @queue.include?(pid)

    @queue << pid
  end
end

#idle? ⇒ Boolean

Returns:

  • (Boolean) 


38
39
40
 
# File 'lib/wurk/swarm/restart.rb', line 38

def idle?
  @current.nil? && @queue.empty?
end