Class: Ask::WebFetch::Backends::Browser

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/browser.rb

Overview

Last-resort backend: a real Chrome driven over CDP via Ferrum. Renders JavaScript, so it reads SPAs and client-side pages that Local's plain HTTP cannot, and it lets Cloudflare-style managed challenges that auto-solve for real browsers complete themselves — neither Local (no JS engine) nor Jina (known datacenter renderer) can do either.

Two modes:

  • Launched — a fresh Chrome the backend starts itself (default when a binary is found). Honest limitation, measured in the wild: sites running aggressive bot protection (patronview.com, npmjs.com, stackoverflow.com all soft-block a freshly launched browser from a datacenter IP) never clear their invisible challenge for a fresh automation profile, however real the Chrome.
  • Attached — connects to an already-running Chrome via CDP (ASK_WEB_FETCH_CDP_URL, e.g. http://127.0.0.1:9222). That browser is a trusted context: long-lived, mature profile, any cookies it has already earned (cf_clearance). Sites that soft-block fresh profiles load normally there. See AttachedBrowser.

Opt-in, like Crawl4AI: joins the chain only when a Chrome/Chromium binary is found or ASK_WEB_FETCH_CDP_URL is set. Configure the binary with ASK_WEB_FETCH_CHROME_PATH and a persistent profile directory with ASK_WEB_FETCH_PROFILE (a profile keeps solved cookies across restarts; within one process the browser is reused anyway).

HTML is converted through the same Markdown pipeline as Local, with the same default adaptive ContentFilter.

Constant Summary collapse

CHALLENGE_TIMEOUT =

Seconds to let a Cloudflare-style challenge auto-solve before giving up (tunable via Browser.challenge_timeout).

30
IDLE_TIMEOUT =

Seconds to wait for the network to go quiet after the page loads, so lazy-loaded content is in before we read the DOM.

5
POLL_INTERVAL =

How often to poll for the challenge to clear.

0.5
DEFAULT_PATHS = 
[
  '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
  '/Applications/Chromium.app/Contents/MacOS/Chromium',
  '/usr/bin/google-chrome',
  '/usr/bin/google-chrome-stable',
  '/usr/bin/chromium',
  '/usr/bin/chromium-browser',
  '/opt/google/chrome/chrome'
].freeze

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::PARKED_DOMAIN_MARKERS, Ask::WebFetch::Backend::USER_AGENT

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name, #guard_page!, #markdown_outlinks, #outlink_urls

Class Attribute Details

.browser ⇒ Object

A shared browser, built once and reused. Ferrum is loaded lazily so consumers who never hit this backend pay nothing for it.



93
94
95
96
97
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 93

def browser
  return @browser if @browser

  browser_mutex.synchronize { @browser ||= build_browser }
end

.cdp_url ⇒ Object 



107
108
109
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 107

def cdp_url
  @cdp_url || ENV['ASK_WEB_FETCH_CDP_URL']
end

.challenge_timeout ⇒ Object 



111
112
113
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 111

def challenge_timeout
  @challenge_timeout || CHALLENGE_TIMEOUT
end

.content_filter ⇒ Object 



99
100
101
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 99

def content_filter
  @content_filter ||= ContentFilter.default
end

.path ⇒ Object 



103
104
105
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 103

def path
  @path || ENV['ASK_WEB_FETCH_CHROME_PATH'] || DEFAULT_PATHS.find { |p| File.exist?(p) }
end

.poll_interval ⇒ Object 



115
116
117
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 115

def poll_interval
  @poll_interval || POLL_INTERVAL
end

Class Method Details

.configured? ⇒ Boolean

Returns:

  • (Boolean) 


119
120
121
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 119

def configured?
  !path.to_s.empty? || !cdp_url.to_s.empty?
end

.warmed_domains ⇒ Object

Domains already given a warm pass this process — a failed warm (DataDome-class wall) is not re-paid at CHALLENGE_TIMEOUT on every fetch of that domain.



87
88
89
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 87

def warmed_domains
  @warmed_domains ||= Set.new
end

.ws_url_for(cdp_url) ⇒ Object

Turns an ASK_WEB_FETCH_CDP_URL into the browser-level WebSocket URL. Accepts a ws:// URL as-is, or an HTTP endpoint ("http://127.0.0.1:9222" or "…/json/version") which is probed for its webSocketDebuggerUrl — the same discovery puppeteer's connect does.



128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 128

def ws_url_for(cdp_url)
  return cdp_url if cdp_url.start_with?('ws://', 'wss://')

  version_url = cdp_url.end_with?('/json/version') ? cdp_url : "#{cdp_url.chomp('/')}/json/version"
  body = Net::HTTP.get(URI(version_url))
  JSON.parse(body)['webSocketDebuggerUrl']
rescue Errno::ECONNREFUSED, SocketError => e
  raise FetchError, "cannot reach CDP endpoint #{version_url}: #{e.message}"
rescue JSON::ParserError => e
  raise FetchError, "bad CDP version response from #{version_url}: #{e.message}"
end

Instance Method Details

#domain_root(url) ⇒ Object 



233
234
235
236
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 233

def domain_root(url)
  uri = URI(url)
  "#{uri.scheme}://#{uri.host}"
end

#fetch(url) ⇒ Object 



158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 158

def fetch(url)
  page = nil
  raise FetchError, 'no Chrome/Chromium found and no CDP endpoint set' unless self.class.configured?
  raise FetchError, 'ferrum gem unavailable' unless self.class.browser

  page = self.class.browser.create_page
  fetch_attempt(page, url)
rescue Ferrum::TimeoutError, Ferrum::ProcessTimeoutError, Ferrum::DeadBrowserError => e
  raise TimeoutError, "#{e.class}: #{e.message}"
rescue Ferrum::StatusError => e
  raise FetchError, "browser could not load #{url}: #{e.message}"
rescue Ferrum::Error => e
  raise ServerError, "browser #{e.class}: #{e.message}"
rescue Errno::ECONNREFUSED, SocketError => e
  raise TimeoutError, "browser connection #{e.class}: #{e.message}"
ensure
  page&.close
end

#fetch_attempt(page, url, warmed: false) ⇒ Object

One fetch of url on the page, with the warm-and-retry pass: a challenge page means this domain hasn't issued the profile a clearance cookie yet. Visiting the DOMAIN ROOT first (where a managed challenge auto-solves for a trusted browser) earns the cookie — pinned to this browser + IP and persisted in the profile — then the URL is retried once. Subsequent fetches for the same domain find the cookie and never warm again. Bounded: one warm per domain per process (warmed_domains), one retry per fetch — a still-challenged URL fails as before, never wedging the queue on a DataDome-class wall.

Raises:



187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 187

def fetch_attempt(page, url, warmed: false)
  page.go_to(url)
  wait_for_challenge(page)
  wait_for_idle(page)

  status = page.network.status
  raise FetchError, "got #{status} at #{url}" if status && status >= 400

  body = page.body
  if challenge_page?(body)
    raise FetchError, "challenge page at #{url}" if warmed
    raise FetchError, "challenge page at #{url}" unless warm_domain(page, url)

    return fetch_attempt(page, url, warmed: true)
  end
  # Browser renders the parked page a JS redirect lands on (the
  # server shell hands /lander to JS) — Local never sees it. The
  # shared guard catches it on the rendered HTML (raw_body) and
  # the converted content; the distinct ParkedDomainError lets the
  # pipeline classify (never retry) it.
  result = Markdown.generate(body, base_url: url, filter: self.class.content_filter)
  result[:outlinks] = outlink_urls(body, url)
  guard_page!(url, result[:content], raw_body: body)

  result
end

#warm_domain(page, url) ⇒ Object

Earns the domain's clearance cookie: navigates to the domain root (the challenge JS runs there, not on the deep URL), waits for the managed challenge to auto-solve, and lets the cookie land in the profile. Returns true when the domain is now warm. Domains already attempted this process are skipped — a failed warm is not re-paid at CHALLENGE_TIMEOUT per fetch.



220
221
222
223
224
225
226
227
228
229
230
231
 
# File 'lib/ask/web_fetch/backends/browser.rb', line 220

def warm_domain(page, url)
  domain = URI(url).host
  return false if self.class.warmed_domains.include?(domain)

  self.class.warmed_domains << domain
  page.go_to(domain_root(url))
  wait_for_challenge(page)
  wait_for_idle(page)
  true
rescue Ferrum::Error, URI::InvalidURIError
  false
end