Class: Ask::WebFetch::Backends::Crawl4Ai

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/crawl4ai.rb

Overview

Self-hosted Crawl4AI (https://docs.crawl4ai.com) — a headless Chromium crawler that renders JavaScript and returns clean markdown. Runs as its own Docker service (default http://localhost:11235), the same self-hosted pattern as ask-web-search's SearXNG. No API key; configure via CRAWL4AI_URL (and CRAWL4AI_TOKEN for 0.9+ JWT-protected servers).

Kept FIRST in the default chain: when the service is present it handles the JS-rendered pages the Local backend can't. When it isn't configured — or is unreachable — it fails fast and the chain falls through to Local, with Jina as the last resort.

Constant Summary collapse

DEFAULT_URL = 
'http://localhost:11235'
OPEN_TIMEOUT = 
5
READ_TIMEOUT =

Browser rendering (plus first-request pool warmup) is slow — the crawl itself gets crawler_config.timeout, so the HTTP read must allow that plus headroom, unlike the plain-HTML backends.

90
CRAWL_TIMEOUT = 
60

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::USER_AGENT

Class Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name

Class Attribute Details

.token ⇒ Object 



38
39
40
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 38

def token
  @token || ENV['CRAWL4AI_TOKEN']
end

.url ⇒ Object 



34
35
36
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 34

def url
  @url || ENV['CRAWL4AI_URL']
end

Class Method Details

.configured? ⇒ Boolean

Presence = configuration. The tool's default chain only includes this backend when CRAWL4AI_URL is set, so consumers without a Crawl4AI service see zero behavior change (Local -> Jina).

Returns:

  • (Boolean) 


45
46
47
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 45

def configured?
  !url.to_s.empty?
end

Instance Method Details

#fetch(url) ⇒ Object 



50
51
52
53
54
55
56
57
58
59
60
61
62
63
 
# File 'lib/ask/web_fetch/backends/crawl4ai.rb', line 50

def fetch(url)
  raise FetchError, 'Crawl4AI not configured (set CRAWL4AI_URL)' if self.class.url.to_s.empty?

  body = crawl(url)
  raise FetchError, "challenge page at #{url}" if challenge_page?(body)

  page = to_page(body, url)
  raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])

  page
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
       Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  raise FetchError, "Crawl4AI #{e.class}: #{e.message}"
end