Class: Ask::WebFetch::Backends::AttachedBrowser::Page::Network

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/web_fetch/backends/attached_browser.rb

Overview

Real network-idle detection for the attached browser, via CDP's Network domain events. The launched-browser path gets idle waiting from Ferrum's network; the attached path previously skipped it entirely (a no-op), which read lazy-loading SPAs too early — reddit's post list renders in waves, and the fetch saw only the first 2.6k of an 8k+ body. Tracks requestStarted / requestFinished events on the page session and waits until the network has been quiet for duration, capped at timeout.

Instance Method Summary collapse

Constructor Details

#initialize(page) ⇒ Network

Returns a new instance of Network.



147
148
149
150
151
152
153
154
 
# File 'lib/ask/web_fetch/backends/attached_browser.rb', line 147

def initialize(page)
  @page = page
  @session = page.instance_variable_get(:@session)
  @in_flight = 0
  @quiet_since = Process.clock_gettime(Process::CLOCK_MONOTONIC)
  @mutex = Mutex.new
  subscribe
end

Instance Method Details

#status ⇒ Object 



156
157
158
159
160
161
162
163
 
# File 'lib/ask/web_fetch/backends/attached_browser.rb', line 156

def status
  @page.evaluate(
    "performance.getEntriesByType('navigation')[0] ? " \
    "performance.getEntriesByType('navigation')[0].responseStatus : 0"
  ) || 0
rescue Ferrum::Error
  0
end

#wait_for_idle(duration:, timeout:) ⇒ Object 



165
166
167
168
169
170
171
172
173
174
175
176
 
# File 'lib/ask/web_fetch/backends/attached_browser.rb', line 165

def wait_for_idle(duration:, timeout:)
  deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout
  loop do
    quiet = @mutex.synchronize { @in_flight.zero? }
    if quiet && Process.clock_gettime(Process::CLOCK_MONOTONIC) - @quiet_since >= duration
      return
    end
    raise Ferrum::TimeoutError, "network did not go idle within #{timeout}s" if Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline

    sleep 0.1
  end
end