Class: Ask::WebFetch::Backends::Crawl4Ai

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/crawl4ai.rb

Overview

Self-hosted Crawl4AI (https://docs.crawl4ai.com) — a headless Chromium crawler that renders JavaScript and returns clean markdown. Runs as its own Docker service (default http://localhost:11235), the same self-hosted pattern as ask-web-search's SearXNG. No API key; configure via CRAWL4AI_URL (and CRAWL4AI_TOKEN for 0.9+ JWT-protected servers).

Kept FIRST in the default chain: when the service is present it handles the JS-rendered pages the Local backend can't. When it isn't configured — or is unreachable — it fails fast and the chain falls through to Local, with Jina as the last resort.

Constant Summary collapse

DEFAULT_URL = 
'http://localhost:11235'
OPEN_TIMEOUT = 
5
READ_TIMEOUT =

Browser rendering (plus first-request pool warmup) is slow — the crawl itself gets crawler_config.timeout, so the HTTP read must allow that plus headroom, unlike the plain-HTML backends.

90
CRAWL_TIMEOUT = 
60

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::PARKED_DOMAIN_MARKERS, Ask::WebFetch::Backend::USER_AGENT

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name, #guard_page!, #markdown_outlinks, #outlink_urls

Class Attribute Details

.token ⇒ Object 



39
40
41
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 39

def token
  @token || ENV['CRAWL4AI_TOKEN']
end

.url ⇒ Object 



35
36
37
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 35

def url
  @url || ENV['CRAWL4AI_URL']
end

Class Method Details

.configured? ⇒ Boolean

Presence = configuration. The tool's default chain only includes this backend when CRAWL4AI_URL is set, so consumers without a Crawl4AI service see zero behavior change (Local -> Jina).

Returns:

  • (Boolean) 


46
47
48
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 46

def configured?
  !url.to_s.empty?
end

Instance Method Details

#fetch(url) ⇒ Object 



51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 51

def fetch(url)
  raise FetchError, 'Crawl4AI not configured (set CRAWL4AI_URL)' if self.class.url.to_s.empty?

  body = crawl(url)
  raise FetchError, "challenge page at #{url}" if challenge_page?(body)

  page = to_page(body, url)
  # A registrar parking page renders fine in headless Chrome too —
  # the shared guard's prose markers catch it (the HTML-only
  # markers never reach a markdown-only backend), so the ad is
  # rejected, not returned as the site's content. Crawl4AI leads
  # the default chain, so this guard is what keeps parked domains
  # out of every result.
  guard_page!(url, page[:content])

  page
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
       Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  raise TimeoutError, "Crawl4AI #{e.class}: #{e.message}"
end