Module: Ask::WebFetch

Defined in:
lib/ask/web_fetch.rb,
lib/ask/web_fetch/http.rb,
lib/ask/web_fetch/backend.rb,
lib/ask/web_fetch/version.rb,
lib/ask/web_fetch/markdown.rb,
lib/ask/web_fetch/noise_filter.rb,
lib/ask/web_fetch/backends/jina.rb,
lib/ask/web_fetch/backends/local.rb,
lib/ask/web_fetch/content_filter.rb,
lib/ask/web_fetch/backends/browser.rb,
lib/ask/web_fetch/backends/crawl4ai.rb,
lib/ask/web_fetch/backends/attached_browser.rb

Overview

Fetches a URL and returns its content as clean markdown for LLM consumption. The capability layer: a pluggable backend chain, a failure collapse, and one entry point. Tool framing — name, parameter schema, result wrapping — lives with the consumers (the MCP servers, the agents) that call this library, not here.

Defined Under Namespace

Modules: Backends, Markdown Classes: Backend, ContentFilter, EmptyContentError, Error, FetchError, Http, NoiseFilter, ParkedDomainError, ServerError, TimeoutError

Constant Summary collapse

DEFAULT_MAX_CHARS = 
20_000
DETERMINISTIC =

Errors that mean "the URL is dead" — no amount of retrying changes the answer. When EVERY backend failed this way, the aggregate re-raises as FetchError so callers can fail fast; any transient failure in the mix (timeout, 5xx, empty render) keeps the base Error, which recovers on retry.

[Ask::WebFetch::FetchError, Ask::WebFetch::EmptyContentError].freeze
VERSION = 
'0.7.1'

Class Attribute Summary collapse

Class Method Summary collapse

Class Attribute Details

.backends ⇒ Object

Backend chain, tried in order. Crawl4AI leads when configured (CRAWL4AI_URL), so a present self-hosted renderer is preferred; otherwise Local, with Jina as the last resort, and Browser appended when Chrome is available. Swap or extend for future backends; each must subclass Ask::WebFetch::Backend and implement #fetch(url).



47
48
49
50
51
52
53
54
55
56
 
# File 'lib/ask/web_fetch.rb', line 47

def self.backends
  @backends ||= begin
    chain = [Ask::WebFetch::Backends::Local, Ask::WebFetch::Backends::Jina]
    if Ask::WebFetch::Backends::Crawl4Ai.configured?
      chain.unshift(Ask::WebFetch::Backends::Crawl4Ai)
    end
    chain << Ask::WebFetch::Backends::Browser if Ask::WebFetch::Backends::Browser.configured?
    chain
  end
end

Class Method Details

.collapse(failures, url) ⇒ Object

Collapses every backend's failure into ONE error whose class carries the best explanation. Precedence, most definitive first: a parked domain beats an empty shell (the shell IS the parking ad's shell — Local sees the JS redirect stub, Browser the lander), empty beats a dead 4xx (the page existed, it just had no content), and any deterministic explanation beats a transient one (transient keeps the retryable base Error). Clients read the class: ParkedDomainError / EmptyContentError / FetchError are terminal — retrying never changes the answer; Error may recover on retry.

Raises:



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
 
# File 'lib/ask/web_fetch.rb', line 85

def self.collapse(failures, url)
  detail = failures.map { |backend, e| "#{backend.backend_name}: #{e.message}" }.join('; ')
  message = "all web fetch backends failed for #{url} (#{detail})"

  classes = failures.map { |_, e| e.class }
  if classes.any? { |k| k <= Ask::WebFetch::ParkedDomainError }
    raise Ask::WebFetch::ParkedDomainError, message
  end
  if classes.any? { |k| k <= Ask::WebFetch::EmptyContentError }
    raise Ask::WebFetch::EmptyContentError, message
  end

  deterministic = failures.all? { |_, e| DETERMINISTIC.any? { |klass| e.is_a?(klass) } }
  raise(deterministic ? Ask::WebFetch::FetchError : Ask::WebFetch::Error, message)
end

.fetch(url, max_chars: DEFAULT_MAX_CHARS) ⇒ Object

Fetches url and returns LLM-ready markdown — "# Title\n\nSource: url\n\ncontent" — capped at max_chars (default 20000; pass nil to skip the cap). The single entry point for "give me this page as markdown"; the raw page hash is #fetch_page.



105
106
107
108
109
110
 
# File 'lib/ask/web_fetch.rb', line 105

def self.fetch(url, max_chars: DEFAULT_MAX_CHARS)
  page = fetch_page(url)
  markdown = format(page, url)
  markdown = truncate(markdown, max_chars) if max_chars&.positive?
  markdown
end

.fetch_page(url) ⇒ Object

Fetches url through the chain and returns the first success as { title:, description:, content:, outlinks:, redirected: }. Raises when every backend fails; the raised class carries the verdict (see #collapse) and the message lists every backend and what it said.



66
67
68
69
70
71
72
73
74
 
# File 'lib/ask/web_fetch.rb', line 66

def self.fetch_page(url)
  failures = []
  backends.each do |backend_class|
    return backend_class.new.fetch(url)
  rescue Ask::WebFetch::Error => e
    failures << [backend_class, e]
  end
  collapse(failures, url)
end