Class: Findbug::Storage::CircuitBreaker

Inherits:
Object
  • Object
show all
Defined in:
lib/findbug/storage/circuit_breaker.rb

Overview

CircuitBreaker prevents cascading failures when Redis is down.

THE PROBLEM IT SOLVES

Imagine Redis goes down during peak traffic:

Without circuit breaker:
- 1000 requests/second
- Each tries to write to Redis
- Each waits 1 second for timeout
- Your app becomes unusable

With circuit breaker:
- After 5 failures, circuit "opens"
- Next 1000 requests skip Redis immediately
- Your app stays fast
- After 30 seconds, we try again

THE THREE STATES

┌─────────────────────────────────────────────────────────┐
│                                                         │
│   ┌──────────┐    failures >= 5    ┌──────────┐        │
│   │  CLOSED  │ ─────────────────── │   OPEN   │        │
│   │ (normal) │                     │ (tripped)│        │
│   └──────────┘                     └──────────┘        │
│        ▲                                 │             │
│        │ success                         │ 30 seconds  │
│        │                                 ▼             │
│        │                          ┌───────────┐        │
│        └───────────────────────── │ HALF-OPEN │        │
│                                   │ (testing) │        │
│                                   └───────────┘        │
│                                         │              │
│                                         │ failure      │
│                                         ▼              │
│                                   ┌──────────┐         │
│                                   │   OPEN   │         │
│                                   └──────────┘         │
└─────────────────────────────────────────────────────────┘

THREAD SAFETY

This class uses Monitor (a reentrant mutex) to ensure thread safety. Multiple threads can check/update the circuit state without races.

Constant Summary collapse

FAILURE_THRESHOLD =

How many failures before we trip the circuit

5
RECOVERY_TIME =

How long to wait before trying again (in seconds)

30

Class Method Summary collapse

Class Method Details

.allow? ⇒ Boolean

Check if requests are allowed through

Examples:

if CircuitBreaker.allow?
  # try Redis operation
else
  # skip and log
end

Returns:

  • (Boolean) —

    true if we should attempt the operation



75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 75

def allow?
  synchronize do
    case state
    when :closed
      # Normal operation - allow all requests
      true
    when :open
      if recovery_period_elapsed?
        # Time to test if Redis is back
        transition_to(:half_open)
        true
      else
        # Still in cooldown - reject immediately
        false
      end
    when :half_open
      # We're testing - allow this one request through
      true
    end
  end
end

.execute { ... } ⇒ Object?

Execute a block with circuit breaker protection

This is a convenience method that combines allow?/record_success/record_failure.

Examples:

result = CircuitBreaker.execute do
  redis.lpush("key", "value")
end

Yields:

  • the operation to protect

Returns:

  • (Object, nil) —

    the block’s return value, or nil if rejected



166
167
168
169
170
171
172
173
174
175
176
177
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 166

def execute
  return nil unless allow?

  begin
    result = yield
    record_success
    result
  rescue StandardError => e
    record_failure
    raise e
  end
end

.failure_count ⇒ Integer

Get current failure count (for monitoring)

Returns:

  • (Integer) —

    number of consecutive failures



141
142
143
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 141

def failure_count
  @failures || 0
end

.record_failure ⇒ Object

Record a failed operation

Call this when a Redis operation fails. After enough failures, the circuit opens.



114
115
116
117
118
119
120
121
122
123
124
125
126
127
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 114

def record_failure
  synchronize do
    @failures = (@failures || 0) + 1

    if state == :half_open
      # Failed during testing - back to open
      transition_to(:open)
    elsif @failures >= FAILURE_THRESHOLD
      # Too many failures - trip the circuit
      transition_to(:open)
      log_circuit_opened
    end
  end
end

.record_success ⇒ Object

Record a successful operation

Call this after a successful Redis operation. This resets the failure count and closes the circuit.



102
103
104
105
106
107
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 102

def record_success
  synchronize do
    @failures = 0
    transition_to(:closed)
  end
end

.reset! ⇒ Object

Reset the circuit breaker (for testing)



146
147
148
149
150
151
152
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 146

def reset!
  synchronize do
    @state = :closed
    @failures = 0
    @opened_at = nil
  end
end

.state ⇒ Symbol

Get current state (for monitoring/debugging)

Returns:

  • (Symbol) —

    :closed, :open, or :half_open



133
134
135
 
# File 'lib/findbug/storage/circuit_breaker.rb', line 133

def state
  @state || :closed
end