Class: Ask::Tools::WebFetch

Inherits:
Ask::Tool
  • Object
show all
Defined in:
lib/ask/web_fetch/tool.rb

Overview

Fetches a URL and returns its content as clean markdown for LLM consumption. Tries each configured backend in order and returns the first success.

Chain: Crawl4AI first (self-hosted headless-Chromium renderer; used when the CRAWL4AI_URL service is present, fails fast when it isn't), then the local fetcher, with Jina Reader as the last resort.

Constant Summary collapse

DEFAULT_MAX_CHARS = 
20_000

Class Attribute Summary collapse

Instance Method Summary collapse

Class Attribute Details

.backends ⇒ Object

Backend chain, tried in order. Crawl4AI leads when configured (CRAWL4AI_URL), so a present self-hosted renderer is preferred; otherwise Local, with Jina as the last resort. Swap or extend for future backends; each must subclass Ask::WebFetch::Backend and implement #fetch(url).



26
27
28
29
30
31
32
33
34
 
# File 'lib/ask/web_fetch/tool.rb', line 26

def self.backends
  @backends ||= begin
    chain = [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
    if Ask::WebFetch::Backends::Crawl4Ai.configured?
      chain.unshift(Ask::WebFetch::Backends::Crawl4Ai)
    end
    chain
  end
end

Instance Method Details

#execute(url:, max_chars: DEFAULT_MAX_CHARS) ⇒ Object 



52
53
54
55
56
57
 
# File 'lib/ask/web_fetch/tool.rb', line 52

def execute(url:, max_chars: DEFAULT_MAX_CHARS)
  page = fetch_page(url)
  markdown = format(page, url)
  markdown = truncate(markdown, max_chars) if max_chars&.positive?
  Ask::Result.ok(data: markdown)
end