Class: Ask::WebFetch::Backends::Local

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/local.rb

Overview

Default backend: pure Ruby httpx + Nokogiri + reverse_markdown. No external service or API key; mirrors ask-web-search's self-hosted SearXNG approach.

HTML is converted through Ask::WebFetch::Markdown with a default ContentFilter, so the "fit" content — pruned by text density rather than by keyword — is what comes back.

Constant Summary collapse

MAX_REDIRECTS = 
5
JS_APP_SHELL_MARKERS =

Client-rendered framework markers: the server HTML is (at least partly) a shell awaiting JS. Deliberately narrow — generic terms like "script" or "root" appear on every page; these are the specific footprints of React/Vue/Next/Nuxt app shells.

/id=["'](?:root|app|__next|site-content)["']|__NEXT_DATA__|window\.__NUXT__|ng-app|data-reactroot/
SHELL_DEFER_THRESHOLD =

A framework marker (React/Vue/Next) alone is NOT emptiness: a marked page can server-render real content (careers.abb job pages: 2,162 chars of job description) and must be stored as-is when the rendering backends cannot do better — dropping it lost real pages (2026-08-14). The shell deferral fires only when the extraction is genuinely little: below this, the page is a true shell (airbnb: 613KB HTML -> 143 chars). Tunable; the chain turns the signal into "prefer Browser for this URL".

500
SHELL_CONTENT_THRESHOLD =

The ratio detector's own bar, kept for compatibility with the comment below (large HTML + near-empty text is a shell whatever the framework).

4_000
SHELL_HTML_BYTES =

A page whose server HTML is large but yields almost no text is a shell whatever framework it uses (airbnb: 613KB HTML -> 143 chars of markdown). Framework markers miss these; the size ratio catches them. nytimes (1.4MB -> 7.5k text) stays above the text floor and is correctly left to Local.

20_000

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::PARKED_DOMAIN_MARKERS, Ask::WebFetch::Backend::USER_AGENT

Class Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name, #guard_page!, #markdown_outlinks, #outlink_urls

Class Attribute Details

.content_filter ⇒ Object 



27
28
29
 
# File 'lib/ask/web_fetch/backends/local.rb', line 27

def content_filter
  @content_filter ||= ContentFilter.default
end

.http ⇒ Object 



36
37
38
 
# File 'lib/ask/web_fetch/backends/local.rb', line 36

def http
  @http ||= Http
end

Instance Method Details

#fetch(url) ⇒ Object 



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/ask/web_fetch/backends/local.rb', line 41

def fetch(url)
  body, content_type, redirect = fetch_html(url)
  raise FetchError, "expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
  raise FetchError, "challenge page at #{url}" if challenge_page?(body)

  page = to_markdown(body, url)
  page[:redirected] = redirect
  # Parked-domain pages are not content: the domain owner parked it
  # with a registrar and the page is an ad for buying the domain
  # (GoDaddy/Namecheap/Sedo parking). A content company must never
  # store these as if they were the site. The shared guard checks
  # the raw server HTML first (parked pages are fully
  # server-rendered — the HTML-only markers live in scripts and
  # assets), then the content minimum, then the JS-shell
  # completeness signal below.
  guard_page!(url, page[:content], raw_body: body)
  # The completeness signal: a JS-app shell whose server HTML
  # renders little is a TRUNCATED page, not a complete one — the
  # real content awaits client-side JS that Local cannot run.
  # Storing the shell as success would silently under-deliver
  # (airbnb: 613KB server HTML -> 143 chars of markdown); failing
  # through lets the chain prefer a rendering backend (Browser),
  # and keeps a partial page from ever being stored as the real
  # thing. Two detectors: known framework markers, or a large
  # HTML page with almost no server-rendered text.
  if js_app_shell?(body) && page[:content].length < SHELL_DEFER_THRESHOLD
    raise EmptyContentError,
      "JS-app shell at #{url} — server HTML renders only #{page[:content].length} chars; a rendering backend is required"
  end

  page
rescue Errno::ECONNREFUSED, Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  # The transport maps its own failures to TimeoutError; this guard
  # keeps the "only Ask::WebFetch errors escape" invariant even if
  # a transport bug lets a raw socket error through.
  raise TimeoutError, "#{e.class}: #{e.message}"
end

#js_app_shell?(body) ⇒ Boolean

Returns:

  • (Boolean) 


107
108
109
110
 
# File 'lib/ask/web_fetch/backends/local.rb', line 107

def js_app_shell?(body)
  body.to_s.match?(JS_APP_SHELL_MARKERS) ||
    (body.to_s.bytesize > SHELL_HTML_BYTES && markdown_visible_chars(body) < SHELL_CONTENT_THRESHOLD)
end

#markdown_visible_chars(body) ⇒ Object

A cheap text estimate from the raw HTML (tags stripped) — the "how much did the server actually render" number. Deliberately rough: it only feeds a shell-vs-page heuristic.



115
116
117
118
119
120
 
# File 'lib/ask/web_fetch/backends/local.rb', line 115

def markdown_visible_chars(body)
  body.to_s.gsub(/<script[\s\S]*?<\/script>/i, "")
    .gsub(/<style[\s\S]*?<\/style>/i, "")
    .gsub(/<[^>]+>/, " ")
    .gsub(/\s+/, " ").strip.length
end

#to_markdown(html, url) ⇒ Object

Parses html and returns { title:, description:, content:, licenses:, outlinks: } where content is clean markdown (pruned by the ContentFilter), licenses are the page's declared license signals ([] when it declares none), and outlinks are the page's RAW hrefs, resolved and scheme-filtered — nav and footer included, because a crawler's discovery layer reads these even when the stored content is pruned.



129
130
131
132
 
# File 'lib/ask/web_fetch/backends/local.rb', line 129

def to_markdown(html, url)
  Markdown.generate(html, base_url: url, filter: self.class.content_filter)
    .merge(licenses: license_signals(html), outlinks: outlink_urls(html, url))
end