Class: Ask::Tools::WebFetch

Inherits:
Ask::Tool
  • Object
show all
Defined in:
lib/ask/web_fetch/tool.rb

Overview

Fetches a URL and returns its content as clean markdown for LLM consumption. Tries each configured backend in order and returns the first success, so a blocked or JS-rendered page falls through from the local fetcher to Jina Reader.

Constant Summary collapse

DEFAULT_MAX_CHARS = 
20_000

Class Attribute Summary collapse

Instance Method Summary collapse

Class Attribute Details

.backends ⇒ Object

Backend chain, tried in order. Swap or extend for future backends (e.g. a self-hosted crawler); each must subclass Ask::WebFetch::Backend and implement #fetch(url).



20
21
22
 
# File 'lib/ask/web_fetch/tool.rb', line 20

def self.backends
  @backends ||= [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
end

Instance Method Details

#execute(url:, max_chars: DEFAULT_MAX_CHARS) ⇒ Object 



40
41
42
43
44
45
 
# File 'lib/ask/web_fetch/tool.rb', line 40

def execute(url:, max_chars: DEFAULT_MAX_CHARS)
  page = fetch_page(url)
  markdown = format(page, url)
  markdown = truncate(markdown, max_chars) if max_chars&.positive?
  Ask::Result.ok(data: markdown)
end