Class: RailsErrorDashboard::Services::StormProtection::CircuitBreaker

Inherits:
Object
  • Object
show all
Defined in:
lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb

Overview

Per-process circuit breaker for the error capture path.

Counts capture attempts in fixed 10-second buckets and transitions between states based on the completed bucket’s rate:

:closed    — normal operation, per-fingerprint buckets decide fidelity
:shedding  — elevated rate: context shed, notifications suppressed
:open      — storm: count-only mode, zero per-event I/O
:half_open — post-cooldown probe: small sample admitted, watching rate

Hysteresis: opens FAST (a single hot bucket, or mid-bucket fast-trip), closes SLOW (two consecutive calm buckets) to prevent flapping.

Concurrency: the hot path is one AtomicFixnum increment plus a float comparison. The mutex is taken only on bucket roll (once per 10s) and for state transitions — never per event.

Constant Summary collapse

BUCKET_SECONDS = 
10
CALM_BUCKETS_TO_CLOSE = 
2

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }) ⇒ CircuitBreaker

Returns a new instance of CircuitBreaker.

Parameters:

  • clock (#call) (defaults to: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }) —

    returns monotonic seconds; injectable for tests



29
30
31
32
33
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 29

def initialize(clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
  @clock = clock
  @mutex = Mutex.new
  reset!
end

Instance Attribute Details

#state ⇒ Object (readonly)

Returns the value of attribute state.



26
27
28
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 26

def state
  @state
end

Instance Method Details

#clear_closed_episode! ⇒ Object

Forget a closed episode once it has been persisted by the flush job.



71
72
73
74
75
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 71

def clear_closed_episode!
  @mutex.synchronize do
    @episode = nil if @episode && @episode[:ended_at]
  end
end

#episode_snapshot ⇒ Hash?

Episode metadata for the honesty layer (storm_events row).

Returns:

  • (Hash, nil) —

    nil when no episode is active or recently closed



66
67
68
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 66

def episode_snapshot
  @mutex.synchronize { @episode&.dup }
end

#record! ⇒ Object

Count one capture attempt and return the state that should govern it. Called on EVERY capture — must stay allocation-free on the fast path.



48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 48

def record!
  now = @clock.call
  roll!(now) if now - @bucket_start >= BUCKET_SECONDS

  count = @bucket_count.increment

  # Fast-trip: don't wait for the bucket to complete if it's already
  # over the open threshold — at 50k errors/min a full 10s bucket
  # would let ~8k events through before reacting.
  if count >= open_threshold * BUCKET_SECONDS && @state != :open
    trip_open!(now, count)
  end

  @state
end

#reset! ⇒ Object 



35
36
37
38
39
40
41
42
43
44
 
# File 'lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb', line 35

def reset!
  @mutex.synchronize do
    @state = :closed
    @bucket_start = @clock.call
    @bucket_count = Concurrent::AtomicFixnum.new(0)
    @calm_buckets = 0
    @opened_at = nil
    @episode = nil
  end
end