Module: SolidLoop::ToolFailureReconciler

Defined in:
app/services/solid_loop/tool_failure_reconciler.rb

Overview

ONE shared fenced-transition helper for the tool path's TWO failure reconcilers (the tool-write failure path) so their semantics cannot drift:

* `ToolExecutionJob#reconcile_failure!` — a non-graceful StandardError that
escaped the pipeline.
* `ToolMiddlewares::ErrorHandling#handle_exception` — a graceful error
(McpError) the middleware terminalizes in-line.

WHY it cannot use Loop#transition_status's block to gate ownership: that method performs the loop update! AFTER yielding REGARDLESS — a next inside the block aborts only the block, not the status write. So a stale worker whose per-tool lease was rotated away (its lease expired while it was still alive and the reaper's case 2 reclaimed the ToolCall as a fresh lease, siblings unfenced) would still terminalize a loop it no longer owns: loop.execution_token is the SHARED generation token and still matches, so the transition fires, the token is cleared, and the CURRENT owner (worker B) is killed. The nested lease_held_by? check only skips the ToolCall update, never the loop write.

The fix: do the ownership gate OURSELVES under the loop→tool_call row locks and only terminalize when BOTH tokens validate. Three post-error shapes:

POST-CLAIM error (`lease_token` present, tool NOT executed): the classic
both-token case. Acquire loop → tool_call (the established lock order),
validate `loop.execution_token == mine` AND `tool_call.lease_held_by?(mine)`
BEFORE either write, then do the ToolCall failure mark AND the loop
terminalization in that one locked transaction. A ROTATED lease (A holds
LA, reaper reclaimed as LB) → stale/no-op, the loop is NOT failed.

EXECUTED continuation, no lease (`lease_token` nil, `executed_at` present):
the tool result is already the terminal exactly-once checkpoint; the error
is in the continuation (canonical response / successor). Fail the loop under
the GENERATION fence only — never dirty the already-committed ToolCall.

PRE-LEASE setup error (`lease_token` nil, NOT executed): no tool lease was
ever acquired (agent resolution / sequential-mode eval / builder /
configure_tool_middlewares). Generation-only fenced failure keeps the
job-level reconciliation contract intact. No ToolCall write (none is owned).

Returns TRUE when reconciliation was handled here (the caller swallows the error — the failure is durable OR we are a proven-stale no-op that must not retry); FALSE only means "not our concern" (the caller re-raises).

Class Method Summary collapse

Class Method Details

.fail_loop_generation_fenced(loop_model, execution_token, error_msg) ⇒ Object

Generation-only fenced loop failure (pre-lease setup / executed continuation). No ToolCall write — no per-tool lease is owned here. Reuses the loop's status CAS: the block is empty, so transition_status's "update after yield" behaviour is exactly what we want (a bare generation-fenced transition).



111
112
113
114
115
116
117
118
119
120
 
# File 'app/services/solid_loop/tool_failure_reconciler.rb', line 111

def fail_loop_generation_fenced(loop_model, execution_token, error_msg)
  loop_model.transition_status(
    from: :running,
    to: :failed,
    expected_execution_token: execution_token,
    error_message: error_msg,
    execution_token: nil,
    lease_expires_at: nil
  )
end

.reconcile!(loop_model:, tool_call:, execution_token:, lease_token:, error:) ⇒ Object 



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'app/services/solid_loop/tool_failure_reconciler.rb', line 46

def reconcile!(loop_model:, tool_call:, execution_token:, lease_token:, error:)
  # A cancellation fires precisely when this generation already lost the token
  # (pause/stop): the new generation owns both loop and call. Nothing to fail
  # or dirty — swallow (handled) without touching the DB.
  return true if error.is_a?(SolidLoop::CancellationError)

  error_msg = "#{error.class}: #{error.message}"
  Rails.logger.error(
    "ToolFailureReconciler: #{error_msg}\n#{error.backtrace&.take(10)&.join("\n")}"
  )

  if lease_token.present?
    reconcile_post_claim(loop_model, tool_call, execution_token, lease_token, error_msg)
  elsif tool_call&.reload&.executed_at?
    # Executed continuation with no lease: terminal checkpoint already written;
    # fail the loop under the generation fence, leave the ToolCall untouched.
    fail_loop_generation_fenced(loop_model, execution_token, error_msg)
  else
    # Pre-lease setup error: generation-only fenced failure.
    fail_loop_generation_fenced(loop_model, execution_token, error_msg)
  end

  # Reconciliation was handled on the tool path in every branch above — either
  # we terminalized under a validated fence, or we proved ourselves stale and
  # must NOT retry (a stale worker that keeps retrying is worse than a no-op).
  true
end

.reconcile_post_claim(loop_model, tool_call, execution_token, lease_token, error_msg) ⇒ Object

POST-CLAIM: the BOTH-token gate, done under the loop→tool_call row locks so loop.execution_token and the ToolCall lease columns are frozen between the validation and the writes. If EITHER token fails to match → no-op: we do NOT terminalize a loop we no longer own (the rotated-lease stale-worker case).



78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
 
# File 'app/services/solid_loop/tool_failure_reconciler.rb', line 78

def reconcile_post_claim(loop_model, tool_call, execution_token, lease_token, error_msg)
  loop_model.with_lock do
    # Loop-generation fence (first token) under the loop row lock.
    next unless loop_model.running? && loop_model.execution_token == execution_token

    tool_call.with_lock do
      now = Time.current
      # Per-tool lease fence (second token) under the tool_call row lock.
      # BOTH must validate BEFORE either write — a rotated/expired lease means
      # a live successor (worker B) owns this call; failing the loop would kill
      # it. Stale → no-op. Note: an EXPIRED lease with no rotation yet (still
      # our generation, no worker B) also no-ops here — an over-TTL worker is
      # by definition no longer authoritative, so its failure surfaces via the
      # reaper's case 2 re-invoke rather than terminalizing the loop directly.
      next unless tool_call.lease_held_by?(lease_token, now: now)

      # Both tokens validated under the locks: do BOTH writes in this one
      # locked transaction.
      tool_call.update!(is_success: false, error_message: error_msg, lease_token: nil, leased_until: nil)
      loop_model.update!(
        status: :failed,
        error_message: error_msg,
        execution_token: nil,
        lease_expires_at: nil
      )
    end
  end
end