Class: Wurk::Swarm

Inherits:
Object
  • Object
show all
Includes:
Component
Defined in:
lib/wurk/swarm.rb,
lib/wurk/swarm/backoff.rb,
lib/wurk/swarm/restart.rb,
lib/wurk/swarm/child_boot.rb,
lib/wurk/swarm/orphan_guard.rb

Overview

Parent supervisor. Forks N children per the worker topology, monitors PIDs, relays signals, respawns crashed children with per-slot exponential backoff, handles rolling restart on SIGUSR1, recycles RSS-bloated children.

The supervise loop never sleeps on behalf of a respawn or a restart: crash backoff is tracked as per-slot due-times (Swarm::Backoff) and rolling restart / recycle run as a non-blocking state machine (Swarm::Restart) advanced one phase per tick. TERM/INT is therefore honored within a tick regardless of restart or backoff state.

Boot ordering (must be exact — see docs/idea/03-process-model.md):

1. Host app boots fully; eager loads done.
2. Railtie `after_initialize` fires.
3. `boot` closes parent-side connections (Redis, ActiveRecord).
4. `boot` forks N children.
5. Each child reconnects DB + opens a fresh Redis pool, then
 installs its own signal handlers and starts the Launcher.
6. Parent calls `supervise` to enter the wait/relay loop.

Signals (see docs/idea/04-signals.md):

TERM/INT  → `shutdown`           (graceful drain; aborts any restart)
TSTP      → relay TSTP           (quiet — stop fetching; one-way, no resume)
USR1      → `rolling_restart`    (zero-downtime cycle)

Defined Under Namespace

Classes: Backoff, ChildBoot, OrphanGuard, Restart

Constant Summary collapse

SUPERVISE_TICK = 
0.2
RESPAWN_BACKOFF = 
1.0
HEARTBEAT_WAIT = 
30
MEMORY_CHECK_INTERVAL = 
10
DEFAULT_SHUTDOWN_TIMEOUT = 
25
SHUTDOWN_GRACE =

Children each hard_shutdown after their own drain deadline (bulk_requeue

  • a 3s ensure window + heartbeat cleanup); the parent must not SIGKILL them mid-tail, so its own wait always extends past theirs by this much.
5
SWARM_SIGNALS =

USR2 is relayed (log reopen) — without a trap, a logrotate config that signals the master pid would hit USR2's default disposition and kill the whole swarm.

{ 'TERM' => :term, 'INT' => :term, 'TSTP' => :tstp, 'USR1' => :usr1, 'USR2' => :usr2 }.freeze

Constants included from Component

Component::DEFAULT_THREAD_PRIORITY, Component::LEADER_CACHE_TTL_MS, Component::PROCESS_NONCE

Instance Attribute Summary collapse

Attributes included from Component

#config

Instance Method Summary collapse

Methods included from Component

#default_tag, #fire_event, #handle_exception, #hostname, #identity, #leader?, #logger, #mono_ms, #process_nonce, #real_ms, #redis, #safe_thread, #tid, #watchdog

Constructor Details

#initialize(topology:, config: Wurk.configuration, memory_limit: config.memory_limit_kb, shutdown_timeout: DEFAULT_SHUTDOWN_TIMEOUT) ⇒ Swarm

Returns a new instance of Swarm.



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/wurk/swarm.rb', line 57

def initialize(topology:, config: Wurk.configuration, memory_limit: config.memory_limit_kb,
               shutdown_timeout: DEFAULT_SHUTDOWN_TIMEOUT)
  @topology = topology
  @config = config
  @memory_limit = memory_limit
  @shutdown_timeout = shutdown_timeout
  @children = {}
  @assignments = []
  @stopping = false
  @quieted = false
  @last_memory_check = 0
  @signal_read = nil
  @signal_write = nil
  @respawn_backoff = Backoff.new(base: RESPAWN_BACKOFF)
  @restart = build_restart
end

Instance Attribute Details

#children ⇒ Object (readonly)

Returns the value of attribute children.



55
56
57
 
# File 'lib/wurk/swarm.rb', line 55

def children
  @children
end

#topology ⇒ Object (readonly)

Returns the value of attribute topology.



55
56
57
 
# File 'lib/wurk/swarm.rb', line 55

def topology
  @topology
end

Instance Method Details

#boot(install_signals: true) ⇒ Object

install_signals: is false in tests so the integration suite can drive shutdown / rolling_restart directly without poisoning the test process's signal handlers.

Traps go in BEFORE fork_children: a TERM landing in the (previously post-fork) window between fork and trap installation left the parent on its default disposition — it died instantly and orphaned live, fetching children. Installed first, the trap queues the TERM and the supervise loop drains it (relaying to children) even if it arrives mid-boot.

Raises:

  • (ArgumentError) 


83
84
85
86
87
88
89
90
91
92
 
# File 'lib/wurk/swarm.rb', line 83

def boot(install_signals: true)
  raise 'Wurk::Swarm already booted' unless @assignments.empty?
  raise ArgumentError, 'Topology has no slots' if @topology.empty?

  @assignments = @topology.assignments.freeze
  install_signal_handlers if install_signals
  close_parent_sockets
  fork_children
  @children.keys
end

#quiet_swarm ⇒ Object

TSTP quiet is one-way and GLOBAL (spec §21.3): it must survive respawns and memory recycles, or a quieted-but-crashed child's replacement would resume fetching mid-maintenance. The flag makes every future fork boot already-quieted (see ChildBoot start_quiet).



117
118
119
120
 
# File 'lib/wurk/swarm.rb', line 117

def quiet_swarm
  @quieted = true
  relay_signal('TSTP')
end

#rolling_restart ⇒ Object

SIGUSR1: queue every live child for the rolling-restart state machine, which replaces one slot at a time (spawn replacement → await its heartbeat → TERM the old child → await its drain) without blocking the supervise thread, so TERM stays responsive throughout the cycle.



126
127
128
 
# File 'lib/wurk/swarm.rb', line 126

def rolling_restart
  @restart.enqueue(@children.keys)
end

#shutdown(timeout: @shutdown_timeout) ⇒ Object 



105
106
107
108
109
110
111
 
# File 'lib/wurk/swarm.rb', line 105

def shutdown(timeout: @shutdown_timeout)
  @stopping = true
  @restart.abort
  relay_signal('TERM')
  wait_for_children(timeout + SHUTDOWN_GRACE)
  hard_kill_stragglers
end

#supervise ⇒ Object 



94
95
96
97
98
99
100
101
102
103
 
# File 'lib/wurk/swarm.rb', line 94

def supervise
  until done?
    drain_signals
    reap_children
    spawn_due_respawns
    @restart.advance unless @stopping
    check_memory_pressure
    sleep SUPERVISE_TICK
  end
end