Class: Ask::WebFetch::Backend

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/web_fetch/backend.rb

Overview

Base class for fetch backends, plus the errors they raise.

A backend turns a URL into LLM-ready markdown. To add a new backend:

1. subclass Backend and implement #fetch(url)
2. #fetch must return { title: String|nil, content: String }
3. #fetch must raise FetchError (hard failure) or
 EmptyContentError (page fetched but nothing usable) on failure
4. run the page's markdown through Ask::WebFetch::Markdown.clean
 before returning it — backends that hold HTML get this from
 Markdown.generate, backends fed pre-converted markdown (Jina,
 Crawl4AI) must call it explicitly so the shared noise removal
 and whitespace normalization apply everywhere
5. run the extracted page through #guard_page! — the parked-domain
 and empty-content verdicts are identical in every backend
6. register the class in Ask::WebFetch.backends

The tool tries each backend in order and returns the first success.

Constant Summary collapse

USER_AGENT =

Identity sent on every request, browser-like plus a gem tag.

'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' \
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36 ' \
"ask-web-fetch/#{Ask::WebFetch::VERSION}".freeze
MIN_CONTENT_LENGTH =

Content shorter than this is treated as a page with no usable content (e.g. a JS-rendered shell with nothing server-side). Low on purpose: the content hub keeps everything that is a real page, and the crawler's soft-404 detection (not length) is what separates pages from error shells.

40
CHALLENGE_RE =

Cloudflare-style anti-bot signatures. Deliberately narrow: challenge/interstitial pages carry these markers, while legitimate pages can contain the word "captcha" in unrelated config/JS (e.g. Wikipedia embeds an hcaptcha edit-config flag on every page).

/just a moment|checking your browser|cf-chl/i
PARKED_DOMAIN_MARKERS =

Registrar parking-page markers: the page is an ad for a parked (for-sale) domain, not the site's content. A content company must never store these as if they were the site. Observed live on the CC list, three shapes: (a) GoDaddy's parking-lander JS app (ap:"parking" flag, parking-lander asset, LANDER_SYSTEM="PW") served at /lander, (b) Namecheap's parking app (utm_campaign= nc_market + parkingpage links), and (c) static registrar pages ("is parked free, courtesy of GoDaddy.com", "is registered at Namecheap"). Deliberately specific — generic terms like "domain" or "for sale" appear on real pages.

/ap:"parking"|parking-lander|LANDER_SYSTEM="PW"|utm_campaign=nc_market|utm_source=parkingpage|is parked free, courtesy of GoDaddy|is available on GoDaddy Auctions|is registered at Namecheap/i

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.backend_name ⇒ Object 



83
84
85
 
# File 'lib/ask/web_fetch/backend.rb', line 83

def self.backend_name
  name.split('::').last
end

Instance Method Details

#fetch(url) ⇒ Object

Fetches url and returns { title:, description:, content:, redirected:, licenses:, outlinks: } — licenses being the page's declared license signals (hrefs/values) and outlinks the page's raw link set ([] when the backend can't see any), both consumed by the crawler's classification/discovery layers. Raises FetchError or EmptyContentError on failure.

Raises:

  • (NotImplementedError) 


93
94
95
 
# File 'lib/ask/web_fetch/backend.rb', line 93

def fetch(url)
  raise NotImplementedError, "#{self.class} must implement #fetch(url)"
end

#guard_page!(url, content, raw_body: nil) ⇒ Object

The shared page guard, run by EVERY backend at the same point in its flow — after extraction, before returning: a registrar parking page raises ParkedDomainError, content below the minimum raises EmptyContentError. Same verdicts, same messages, everywhere; a backend's only job is to pass the strings it has.

raw_body: the raw HTML the backend saw, where it saw it (Local, Browser) — the HTML-only markers (ap:"parking", parking-lander, LANDER_SYSTEM="PW") live in scripts and assets that never survive conversion to markdown. content: what the backend would return (all four) — the prose markers survive conversion, so a backend that only ever sees rendered text (Jina, Crawl4AI) still rejects the ad.

Parked is checked BEFORE the content minimum on purpose: a parking page can render above it (puncta.ai: 395c of Namecheap auction ads) and must still be rejected.

Raises:

 


114
115
116
117
118
 
# File 'lib/ask/web_fetch/backend.rb', line 114

def guard_page!(url, content, raw_body: nil)
  raise ParkedDomainError,
        "parked domain at #{url} — registrar parking page, not site content" if parked_domain?(raw_body) || parked_domain?(content)
  raise EmptyContentError, "no readable content at #{url}" unless usable_content?(content)
end

#markdown_outlinks(content, base_url) ⇒ Object

Fallback for backends that only see rendered markdown (Jina, Crawl4AI's markdown output): the markdown's text links, resolved and scheme-filtered. Same shape as #outlink_urls, one implementation for every backend that lacks the raw HTML.



145
146
147
148
149
150
151
152
153
154
155
156
157
 
# File 'lib/ask/web_fetch/backend.rb', line 145

def markdown_outlinks(content, base_url)
  content.to_s.scan(/\]\(([^)\s]+)\)/).filter_map do |match|
    dest = match[0]
    next if dest.start_with?('javascript:', 'mailto:', 'tel:', '#', 'data:')

    uri = URI.join(base_url, dest)
    next unless %w[http https].include?(uri.scheme)

    uri.to_s
  rescue URI::InvalidURIError
    next
  end.uniq
end

#outlink_urls(html, base_url) ⇒ Object

The page's raw outlinks from HTML: every resolved against the base URL and scheme-filtered to absolute http(s). Nav and footer are included — a crawler's discovery reads the full link set even when the stored content is pruned by the ContentFilter. Shared by every backend that holds the page's HTML.



127
128
129
130
131
132
133
134
135
136
137
138
139
 
# File 'lib/ask/web_fetch/backend.rb', line 127

def outlink_urls(html, base_url)
  Nokogiri::HTML(html).css('a[href]').filter_map do |anchor|
    href = anchor['href'].to_s.strip
    next if href.empty? || href.start_with?('javascript:', 'mailto:', 'tel:', '#', 'data:')

    uri = URI.join(base_url, href)
    next unless %w[http https].include?(uri.scheme)

    uri.to_s
  rescue URI::InvalidURIError
    next
  end.uniq
end