Class: RubyLLM::Resilience::Breaker

Inherits:
Object
  • Object
show all
Defined in:
lib/ruby_llm/resilience/breaker.rb

Overview

Cache-backed circuit breaker for one service.

State machine: CLOSED → OPEN → HALF_OPEN → CLOSED

Three keys per service in the configured store:

{service}:failures    — consecutive-failure counter (windowed TTL)
{service}:open_until  — epoch float; presence means open/half-open
{service}:probe_lock  — SETNX lock so exactly one caller probes

API purity contract (learned the hard way in production):

allow_request?  — the MUTATING gate. In half-open it CONSUMES the
                probe slot. Call it exactly once per real request.
open?/closed?/state/failure_count/seconds_until_probe — PURE reads,
                safe for dashboards, logging, and health checks.

Fail-open everywhere: if the store is unreachable, the breaker reports closed and records nothing. The breaker must never take the app down when Redis blips — the API call itself is the thing being protected.

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(service) ⇒ Breaker

Returns a new instance of Breaker.



69
70
71
72
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 69

def initialize(service)
  @service = service.to_s
  self.class.register(@service)
end

Instance Attribute Details

#service ⇒ Object (readonly)

Returns the value of attribute service.



26
27
28
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 26

def service
  @service
end

Class Method Details

.dashboard_status(services: nil) ⇒ Object 



46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 46

def dashboard_status(services: nil)
  config = Resilience.config
  (services || known_services).map do |service|
    breaker = new(service)
    settings = config.settings_for(service)
    {
      service: service,
      state: breaker.state,
      failure_count: breaker.failure_count,
      seconds_until_probe: breaker.seconds_until_probe,
      metadata: config.metadata_for(service),
      failure_threshold: settings.failure_threshold,
      cooldown_seconds: settings.cooldown_seconds,
      overridden: config.services.key?(service)
    }
  end
end

.known_services ⇒ Object

Per-process registry of breakers seen since boot. A cache-store contract can't enumerate keys, so this (plus an explicit list) is how dashboards discover services.



42
43
44
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 42

def known_services
  REGISTRY_MUTEX.synchronize { REGISTRY.to_a.sort }
end

.register(service) ⇒ Object 



35
36
37
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 35

def register(service)
  REGISTRY_MUTEX.synchronize { REGISTRY.add(service) }
end

.reset_registry! ⇒ Object 



64
65
66
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 64

def reset_registry!
  REGISTRY_MUTEX.synchronize { REGISTRY.clear }
end

Instance Method Details

#allow_request? ⇒ Boolean

The mutating gate: true if this request may proceed. In half-open, acquires the atomic probe lock — exactly one caller across all processes gets true; everyone else is treated as open.

Returns:

  • (Boolean) 


77
78
79
80
81
82
83
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 77

def allow_request?
  case current_state
  when :closed    then true
  when :open      then false
  when :half_open then acquire_probe_lock
  end
end

#closed? ⇒ Boolean

Returns:

  • (Boolean) 


91
92
93
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 91

def closed?
  !open?
end

#failure_count ⇒ Object 



99
100
101
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 99

def failure_count
  safely(0) { cache.read(failures_key) }.to_i
end

#open? ⇒ Boolean

Pure: true only when fully open. Half-open reports false (a request MAY be allowed). Never consumes the probe slot — dashboard-safe.

Returns:

  • (Boolean) 


87
88
89
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 87

def open?
  current_state == :open
end

#record_failure ⇒ Object

Increment the failure counter; trip at threshold. In half-open, a single probe failure re-opens immediately (force: the open_until key still exists in half-open and must be overwritten, not skipped).



123
124
125
126
127
128
129
130
131
132
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 123

def record_failure
  if current_state == :half_open
    trip!(force: true)
  else
    count = safely do
      cache.increment(failures_key, 1, expires_in: settings.failures_window_seconds)
    end
    trip! if count && count >= settings.failure_threshold
  end
end

#record_success ⇒ Object

Reset failure count and close the breaker. Fires on_status with :closed on EVERY success — gauge semantics (idempotent), matching the production original. It is not a once-per-transition event.



115
116
117
118
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 115

def record_success
  safely { cache.delete_multi([ failures_key, open_until_key, probe_lock_key ]) }
  notify_status(:closed)
end

#reset! ⇒ Object

Force-close (admin/console use).



135
136
137
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 135

def reset!
  record_success
end

#seconds_until_probe ⇒ Object

Seconds until the breaker will allow a probe (nil if closed).



104
105
106
107
108
109
110
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 104

def seconds_until_probe
  open_until = safely { cache.read(open_until_key) }
  return nil unless open_until

  remaining = open_until.to_f - Time.now.to_f
  remaining.positive? ? remaining.ceil : 0
end

#state ⇒ Object 



95
96
97
 
# File 'lib/ruby_llm/resilience/breaker.rb', line 95

def state
  current_state
end