Class: Crawlora::RateLimiter

Inherits:
Object
  • Object
show all
Defined in:
lib/crawlora/client.rb

Overview

Optional client-side throttle: caps concurrency and spaces requests to a maximum rate (requests per second).

Instance Method Summary collapse

Constructor Details

#initialize(rps, concurrency) ⇒ RateLimiter

Returns a new instance of RateLimiter.



79
80
81
82
83
84
85
86
 
# File 'lib/crawlora/client.rb', line 79

def initialize(rps, concurrency)
  @interval = rps&.positive? ? 1.0 / rps : 0.0
  @slots = concurrency&.positive? ? concurrency : nil
  @available = @slots
  @mutex = Mutex.new
  @cond = ConditionVariable.new
  @next_at = 0.0
end

Instance Method Details

#run ⇒ Object 



88
89
90
91
92
93
94
95
96
 
# File 'lib/crawlora/client.rb', line 88

def run
  acquire
  begin
    space
    yield
  ensure
    release
  end
end