Module: RailsNexus::StormProtection

Defined in:
lib/rails_nexus/storm_protection.rb

Overview

Storm Protection: Circuit breaker + adaptive sampling Prevents error floods from crashing the app with DB writes

Constant Summary collapse

DEFAULT_PER_PROCESS_THRESHOLD =

Default thresholds

50
DEFAULT_GLOBAL_THRESHOLD =

errors per second per process

200
DEFAULT_COOLDOWN_SECONDS =

errors per second globally

60
DEFAULT_SAMPLE_RATE_CALM =

circuit breaker cooldown

1.0
DEFAULT_SAMPLE_RATE_STORM =

100% capture when calm

0.1

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.circuit_open ⇒ Object (readonly)

Returns the value of attribute circuit_open.



15
16
17
 
# File 'lib/rails_nexus/storm_protection.rb', line 15

def circuit_open
  @circuit_open
end

.error_counts ⇒ Object (readonly)

Returns the value of attribute error_counts.



15
16
17
 
# File 'lib/rails_nexus/storm_protection.rb', line 15

def error_counts
  @error_counts
end

.last_reset ⇒ Object (readonly)

Returns the value of attribute last_reset.



15
16
17
 
# File 'lib/rails_nexus/storm_protection.rb', line 15

def last_reset
  @last_reset
end

Class Method Details

.allow_capture? ⇒ Boolean

Check if we should capture this error Returns true if allowed, false if shed

Returns:

  • (Boolean) 


31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
# File 'lib/rails_nexus/storm_protection.rb', line 31

def allow_capture?
  return true unless enabled?

  reset_counts_if_needed

  # If circuit is open (storm detected), use sampling
  if circuit_open?
    return should_sample?
  end

  # Increment count and check threshold
  @error_counts[:total] += 1
  @error_counts[:current_second] += 1

  if @error_counts[:current_second] >= threshold
    open_circuit!
    return should_sample?
  end

  true
end

.cooldown ⇒ Object 



25
26
27
 
# File 'lib/rails_nexus/storm_protection.rb', line 25

def cooldown
  RailsNexus.configuration.storm_cooldown_seconds || DEFAULT_COOLDOWN_SECONDS
end

.enabled? ⇒ Boolean

Returns:

  • (Boolean) 


17
18
19
 
# File 'lib/rails_nexus/storm_protection.rb', line 17

def enabled?
  RailsNexus.configuration.storm_protection_enabled
end

.record_captured ⇒ Object

Record that an error was captured



54
55
56
 
# File 'lib/rails_nexus/storm_protection.rb', line 54

def record_captured
  @error_counts[:captured] += 1
end

.record_shed ⇒ Object

Record that an error was shed



59
60
61
 
# File 'lib/rails_nexus/storm_protection.rb', line 59

def record_shed
  @error_counts[:shed] += 1
end

.reset! ⇒ Object

Reset circuit (manual or after cooldown)



80
81
82
83
84
85
 
# File 'lib/rails_nexus/storm_protection.rb', line 80

def reset!
  @circuit_open = false
  @error_counts = { total: 0, current_second: 0, captured: 0, shed: 0 }
  @last_reset = Time.now
  @second_window = Time.now.to_i
end

.stats ⇒ Object

Get current stats



64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/rails_nexus/storm_protection.rb', line 64

def stats
  reset_counts_if_needed
  {
    enabled: enabled?,
    circuit_open: circuit_open?,
    errors_this_second: @error_counts[:current_second] || 0,
    threshold: threshold,
    total_errors: @error_counts[:total] || 0,
    total_captured: @error_counts[:captured] || 0,
    total_shed: @error_counts[:shed] || 0,
    last_reset: @last_reset,
    sample_rate: circuit_open? ? DEFAULT_SAMPLE_RATE_STORM : DEFAULT_SAMPLE_RATE_CALM
  }
end

.threshold ⇒ Object 



21
22
23
 
# File 'lib/rails_nexus/storm_protection.rb', line 21

def threshold
  RailsNexus.configuration.storm_threshold_per_second || DEFAULT_PER_PROCESS_THRESHOLD
end