Class: SolidLoop::LeaseRenewer

Inherits:
Object
  • Object
show all
Defined in:
lib/solid_loop/lease_renewer.rb

Overview

A SINGLE per-process background service that renews ALL in-flight LLM-turn leases from ONE dedicated DB connection (docs/decisions/durable_attempt_lease.md).

WHY a shared renewer (not a heartbeat-per-turn): the previous design spawned one background thread PER in-flight turn, and each renew tick checked out its own connection via with_connection. At full worker-pool occupancy — the STANDARD Rails deployment shape, where the DB pool size == worker concurrency — every pooled connection is held by a busy worker, so EVERY heartbeat blocks in with_connection waiting for a connection that never frees. After checkout_timeout each heartbeat treats pool exhaustion as a lost lease → the turn yields uncommitted → the reaper requeues into the same saturated pool → livelock. No generation could complete while saturated.

The fix makes renewal O(1) in connections, independent of worker concurrency: ONE renewer thread owns ONE dedicated connection (checked out for the process lifetime, NOT returned to the shared pool between ticks) and batch-renews every registered lease each tick. A LlmCompletionJob REGISTERS its (loop_id, execution_token, lease_duration) on start and DEREGISTERS in ensure; renewal never competes with workers for a connection again.

Leak safety (does NOT depend on ensure running): every registration also carries a HARD CEILING derived from the OWNING AGENT'S legal turn duration — once it has been renewed for longer than max_duration + config.lease_leak_grace, the renewer DROPS it and stops renewing. A leaked registration (a turn that somehow skipped its ensure) therefore self-expires shortly after max_duration, its lease lapses, and the reaper reclaims the loop — so a leaked registration can never strand a loop forever. Crucially, a LEGITIMATE long turn is NEVER dropped: the agent already caps a legal turn at max_duration (its own contract), so any turn within that bound completes before the ceiling. (The earlier lease_duration × K ceiling was WRONG: a healthy turn stays registered for its whole life, so any legitimate stream longer than K lease-widths — e.g. a tiny read_timeout with steady chunks — was dropped and reaper-reclaimed mid-flight, and the replacement hit the same ceiling, so no long turn could ever complete.)

Ownership safety is unchanged from the old per-turn heartbeat: each registered lease is renewed with the SAME conditional UPDATE keyed on loop_id + execution_token, gated by lease_expires_at IS NOT NULL so a canonical commit that NULLed the lease (tool phase) is never resurrected. A 0-row renew that is genuinely lost (rotated away / reclaimed) flips a per- registration lost flag; a 0-row renew because the lease was legitimately cleared to NULL is NOT a loss. The main path checks the per-registration flag (check!) before any canonical write, exactly as before.

Isolation & lifecycle: the renewer connection is checked out from the SHARED pool ONCE (one extra long-lived connection for the whole process — size the pool as worker_concurrency + 1 so it never contends with a full worker set; documented in the README) and released only when the renewer stops. Drawing from the shared pool (rather than a separate one) keeps the renewer's view of the data identical to the workers' — including under transactional test fixtures, which tie every pooled connection to one test transaction. Each renew runs inside the Rails executor so it participates in query-cache/reload/ connection management. The thread is lazily started on the first registration and INTENTIONALLY runs for the whole process lifetime (there is nothing to tear down between turns — registrations come and go, the one thread + one connection persist). Its single connection is released back to the pool at process exit when the thread unwinds. stop!/reset_instance! (used by tests to prove a clean exit) are idempotent, join the thread, and let the held connection return to the pool — no thread or connection is leaked (verified by the specs' clean-exit / baseline-thread-count assertions).

Defined Under Namespace

Classes: Registration

Constant Summary collapse

MIN_INTERVAL =

Never renew slower than this, even for a tiny ttl, so a pathologically small lease does not spin the DB. Mirrors the old LeaseHeartbeat::MIN_INTERVAL.

1.0
MUTEX =

Guards singleton creation AND teardown so two concurrent first-turns can never construct DISTINCT renewers (BLOCKER — the class-var ||= is not atomic: two threads could both see @instance == nil, each new a renewer, each start a thread and try to check out the one process-lifetime connection; whichever lost the class-var assignment would be unreachable from instance/reset_instance! yet keep its thread — and possibly the ONLY reserved connection — running forever, reintroducing the livelock). All access to @instance goes through this constant mutex; the actual stop! runs OUTSIDE the lock so we never hold it across a thread join.

Mutex.new

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(logger: nil) ⇒ LeaseRenewer

Returns a new instance of LeaseRenewer.



132
133
134
135
136
137
138
139
140
 
# File 'lib/solid_loop/lease_renewer.rb', line 132

def initialize(logger: nil)
  @logger        = logger || (defined?(Rails) ? Rails.logger : nil)
  @registrations = {} # key => Registration
  @mutex         = Mutex.new
  @wake          = Concurrent::Event.new
  @stopping      = Concurrent::AtomicBoolean.new(false)
  @thread        = nil
  @tick_interval = MIN_INTERVAL
end

Class Method Details

.instance ⇒ Object 



114
115
116
 
# File 'lib/solid_loop/lease_renewer.rb', line 114

def instance
  MUTEX.synchronize { @instance ||= new }
end

.reset_instance! ⇒ Object

Reset the process singleton (tests only): stop the running renewer and drop the memoized instance so the next registration starts a fresh one. Idempotent. The swap-to-nil is atomic under MUTEX; the (blocking, thread-joining) stop! runs OUTSIDE the lock so a concurrent instance never waits on a join.



122
123
124
125
126
127
128
129
 
# File 'lib/solid_loop/lease_renewer.rb', line 122

def reset_instance!
  inst = MUTEX.synchronize do
    taken = @instance
    @instance = nil
    taken
  end
  inst&.stop!
end

Instance Method Details

#deregister(key) ⇒ Object

Deregister an in-flight turn (called from the job's ensure). Idempotent.



171
172
173
174
175
176
 
# File 'lib/solid_loop/lease_renewer.rb', line 171

def deregister(key)
  @mutex.synchronize do
    @registrations.delete(key)
    recompute_interval
  end
end

#lost?(key) ⇒ Boolean

True once a conditional renew for this registration touched 0 rows and the lease is no longer ours. Thread-safe; readable from the main path between chunks. Missing key (already deregistered / ceiling-dropped) reads as not-lost.

Returns:

  • (Boolean) 


181
182
183
 
# File 'lib/solid_loop/lease_renewer.rb', line 181

def lost?(key)
  @mutex.synchronize { @registrations[key]&.lost? } || false
end

#register(loop_id:, execution_token:, lease_duration:, max_duration: nil) ⇒ Object

Register an in-flight LLM turn. Returns an opaque key used to query the per-registration lost flag and to deregister. Starts the shared renewer thread on first use. The cadence is driven by the SMALLEST registered lease (~ttl/4), so a short-lived lease is still renewed in time.

max_duration (the owning agent's already-enforced legal turn bound) sizes the per-registration leak ceiling (max_duration + config.lease_leak_grace). A legitimate turn — bounded by max_duration — always finishes before it; a genuine leak self-expires shortly after. Defaults to config.default_max_duration so an out-of-band caller that omits it still gets a bounded ceiling.



152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/solid_loop/lease_renewer.rb', line 152

def register(loop_id:, execution_token:, lease_duration:, max_duration: nil)
  key = [ loop_id, execution_token ]
  reg = Registration.new(
    loop_id: loop_id, execution_token: execution_token,
    lease_duration: lease_duration.to_f,
    max_duration: (max_duration || SolidLoop.config.default_max_duration).to_f,
    registered_at: Time.current,
    lost: Concurrent::AtomicBoolean.new(false)
  )
  @mutex.synchronize do
    @registrations[key] = reg
    recompute_interval
  end
  ensure_running
  @wake.set # renew promptly so a just-registered lease is refreshed without waiting a full tick
  key
end

#registered?(key) ⇒ Boolean

Whether key is still actively registered (renewed each tick). Drops to false on deregister OR once the hard ceiling drops a leaked registration. Thread-safe; used by specs to assert the ceiling behavior.

Returns:

  • (Boolean) 


188
189
190
 
# File 'lib/solid_loop/lease_renewer.rb', line 188

def registered?(key)
  @mutex.synchronize { @registrations.key?(key) }
end

#stop! ⇒ Object

Idempotent. Stops the renewer thread, joins it, and releases the dedicated connection. Called at process exit; also used by tests to prove a clean exit.



194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
 
# File 'lib/solid_loop/lease_renewer.rb', line 194

def stop!
  @stopping.make_true
  @wake.set
  thread = nil
  @mutex.synchronize do
    thread   = @thread
    @thread  = nil
  end
  begin
    thread&.join
  rescue Exception => e # rubocop:disable Lint/RescueException
    @logger&.error("[SolidLoop] lease renewer died: #{e.class}: #{e.message}")
  end
  # The held connection is released back to the shared pool when the renewer
  # thread's `with_connection` block unwinds on stop — no connection leaks.
  nil
end