Class: Ask::WebFetch::Backend

Inherits:

Object

• Object
• Ask::WebFetch::Backend

Defined in:

lib/ask/web_fetch/backend.rb

Overview

Base class for fetch backends, plus the errors they raise.

A backend turns a URL into LLM-ready markdown. To add a new backend:

1. subclass Backend and implement #fetch(url)
2. #fetch must return { title: String|nil, content: String }
3. #fetch must raise FetchError (hard failure) or
 EmptyContentError (page fetched but nothing usable) on failure
4. register the class in Ask::Tools::WebFetch.backends

The tool tries each backend in order and returns the first success.

Direct Known Subclasses

Ask::WebFetch::Backends::Jina, Ask::WebFetch::Backends::Local

Constant Summary

USER_AGENT =

Identity sent on every request, browser-like plus a gem tag.

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
"ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze

MIN_CONTENT_LENGTH =

Content shorter than this is treated as a page with no usable content (e.g. a JS-rendered shell with nothing server-side).

100

CHALLENGE_RE =

Cloudflare-style anti-bot signatures. Deliberately narrow: challenge/interstitial pages carry these markers, while legitimate pages can contain the word "captcha" in unrelated config/JS (e.g. Wikipedia embeds an hcaptcha edit-config flag on every page).

/just a moment|checking your browser|cf-chl/i

Class Method Summary

• .backend_name ⇒ Object

Instance Method Summary

• #fetch(url) ⇒ Object

  Fetches url and returns { title: String|nil, content: String }.

Class Method Details

.backend_name ⇒ Object

# File 'lib/ask/web_fetch/backend.rb', line 45

def self.backend_name
  name.split('::').last
end

Instance Method Details

#fetch(url) ⇒ Object

Fetches url and returns { title: String|nil, content: String }. Raises FetchError or EmptyContentError on failure.

Raises:

• (NotImplementedError)

# File 'lib/ask/web_fetch/backend.rb', line 51

def fetch(url)
  raise NotImplementedError, "#{self.class} must implement #fetch(url)"
end