Class: Insika::CircuitState

Inherits:
Object
  • Object
show all
Defined in:
lib/insika/circuit_state.rb

Overview

The circuit breaker's durable state (WS3): one cell per (tenant, provider/model ref) holding the recent failures and when the circuit opened. The read-modify-write rides @store.transaction — the same discipline as the dispatch claim (A1): two concurrent failures of the same cell serialize, so the "10 failures in 60s" count is never lost to a race.

States, standard semantics:

:closed    fewer than `after` failures in the `within` window — attempts flow.
:open      the window count is met AND the cooldown hasn't elapsed — the
         edge FAIL-FASTS (no provider call) with retry_after = remaining
         cooldown. Reached at the `after`-th failure, which stamps
         opened_at.
:half_open the cooldown elapsed — the next attempt is the TRIAL: a success
         closes (record_success clears the cell), a failure reopens
         (a new failure count / opened_at).

Defined Under Namespace

Classes: Record

Constant Summary collapse

SCOPE = 
"circuit_state"
COUNT_LIMIT =

Timestamps are pruned to COUNT_LIMIT — a pathological loop cannot grow the cell unboundedly (the window is 60s-wide; 100 reads and writes bounded).

100

Instance Method Summary collapse

Constructor Details

#initialize(store:) ⇒ CircuitState

Returns a new instance of CircuitState.



30
31
32
 
# File 'lib/insika/circuit_state.rb', line 30

def initialize(store:)
  @store = store
end

Instance Method Details

#record_failure(tenant:, ref:, after: 10, within: 60, now: Time.now) ⇒ Object

Records ONE failure for (tenant, ref). If this failure makes the window reach after and the circuit is not already open, it stamps opened_at (the instant the breaker trips). -> :closed (still closed) | :open (JUST tripped — closed->open, the ONLY transition that alerts) | :reopened (an already-tripped cell re-stamped: a half-open trial failed; the circuit is open again but the node already told the operator it is sick — no NEW alert (WS3)).



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
 
# File 'lib/insika/circuit_state.rb', line 41

def record_failure(tenant:, ref:, after: 10, within: 60, now: Time.now)
  key = key_for(tenant, ref)
  @store.transaction do
    record = load(key)
    cutoff = (now.to_i - within)
    retained = record.failures.select { |t| t > cutoff }.last(COUNT_LIMIT)
    failures = (retained + [now.to_i]).last(COUNT_LIMIT)
    opened_at = record.opened_at
    tripped = :closed
    if opened_at.nil?
      if failures.size >= after
        opened_at = now.to_i
        tripped = :open
      end
    else
      # the circuit was tripped before; a failure here can only be a HALF-OPEN
      # trial that failed. Re-stamp opened_at so it RE-OPENS — a breaker that
      # never restamps is one-shot: after the first cooldown every later turn
      # is an unlocked half-open trial (WS3).
      opened_at = now.to_i
      tripped = :reopened
    end
    @store.set(SCOPE, key, { "failures" => failures, "opened_at" => opened_at })
    tripped
  end
end

#record_success(tenant:, ref:) ⇒ Object

A successful attempt CLOSES the circuit: the cell is cleared so the failure window starts fresh (half-open trial success included).



70
71
72
 
# File 'lib/insika/circuit_state.rb', line 70

def record_success(tenant:, ref:)
  @store.delete(SCOPE, key_for(tenant, ref))
end

#retry_after(tenant:, ref:, cooldown: 300, now: Time.now) ⇒ Object

Seconds until the circuit can be retried (the remaining cooldown). nil while closed.



92
93
94
95
96
97
98
99
100
101
 
# File 'lib/insika/circuit_state.rb', line 92

def retry_after(tenant:, ref:, cooldown: 300, now: Time.now)
  record = load(key_for(tenant, ref))
  return nil if record.failures.empty?

  opened = record.opened_at.to_i
  return nil if opened.zero?

  remaining = cooldown - (now.to_i - opened)
  remaining.positive? ? remaining : nil
end

#state(tenant:, ref:, after: 10, within: 60, cooldown: 300, now: Time.now) ⇒ Object

-> :closed | :open | :half_open



75
76
77
78
79
80
81
82
83
84
85
86
87
88
 
# File 'lib/insika/circuit_state.rb', line 75

def state(tenant:, ref:, after: 10, within: 60, cooldown: 300, now: Time.now)
  record = load(key_for(tenant, ref))
  opened = record.opened_at.to_i
  # :closed needs BOTH doors shut: the window count is under `after` AND the
  # circuit never tripped. A surviving opened_at tombstone keeps the circuit
  # governed by the cooldown (open, then half-open) even after its original
  # failures age out of the rolling window — only a SUCCESS clears it (WS3:
  # a failed half-open trial must reopen, not silently close).
  return :closed if record.failures.size < after && opened.zero?

  return :open if opened.zero? || (now.to_i - opened) < cooldown

  :half_open
end