Class: Ask::WebFetch::Backends::Local

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/local.rb

Overview

Default backend: pure Ruby httpx + Nokogiri + reverse_markdown. No external service or API key; mirrors ask-web-search's self-hosted SearXNG approach.

HTML is converted through Ask::WebFetch::Markdown with a default ContentFilter, so the "fit" content — pruned by text density rather than by keyword — is what comes back.

Constant Summary collapse

MAX_REDIRECTS = 
5
JS_APP_SHELL_MARKERS =

Client-rendered framework markers: the server HTML is (at least partly) a shell awaiting JS. Deliberately narrow — generic terms like "script" or "root" appear on every page; these are the specific footprints of React/Vue/Next/Nuxt app shells.

/id=["'](?:root|app|__next|site-content)["']|__NEXT_DATA__|window\.__NUXT__|ng-app|data-reactroot/
SHELL_CONTENT_THRESHOLD =

Markdown below this from a JS-app shell is "server sent a shell", not "page is genuinely short" — a real page (even a short one) is usually server-rendered above this. Tunable; the chain turns the signal into "prefer Browser for this URL".

4_000
SHELL_HTML_BYTES =

A page whose server HTML is large but yields almost no text is a shell whatever framework it uses (airbnb: 613KB HTML -> 143 chars of markdown). Framework markers miss these; the size ratio catches them. nytimes (1.4MB -> 7.5k text) stays above the text floor and is correctly left to Local.

20_000

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::PARKED_DOMAIN_MARKERS, Ask::WebFetch::Backend::USER_AGENT

Class Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name, #guard_page!, #markdown_outlinks, #outlink_urls

Class Attribute Details

.content_filter ⇒ Object 



27
28
29
 
# File 'lib/ask/web_fetch/backends/local.rb', line 27

def content_filter
  @content_filter ||= ContentFilter.default
end

.http ⇒ Object 



36
37
38
 
# File 'lib/ask/web_fetch/backends/local.rb', line 36

def http
  @http ||= Http
end

Instance Method Details

#fetch(url) ⇒ Object 



41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/ask/web_fetch/backends/local.rb', line 41

def fetch(url)
  body, content_type, redirect = fetch_html(url)
  raise FetchError, "expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
  raise FetchError, "challenge page at #{url}" if challenge_page?(body)

  page = to_markdown(body, url)
  page[:redirected] = redirect
  # Parked-domain pages are not content: the domain owner parked it
  # with a registrar and the page is an ad for buying the domain
  # (GoDaddy/Namecheap/Sedo parking). A content company must never
  # store these as if they were the site. The shared guard checks
  # the raw server HTML first (parked pages are fully
  # server-rendered — the HTML-only markers live in scripts and
  # assets), then the content minimum, then the JS-shell
  # completeness signal below.
  guard_page!(url, page[:content], raw_body: body)
  # The completeness signal: a JS-app shell whose server HTML
  # renders little is a TRUNCATED page, not a complete one — the
  # real content awaits client-side JS that Local cannot run.
  # Storing the shell as success would silently under-deliver
  # (airbnb: 613KB server HTML -> 143 chars of markdown); failing
  # through lets the chain prefer a rendering backend (Browser),
  # and keeps a partial page from ever being stored as the real
  # thing. Two detectors: known framework markers, or a large
  # HTML page with almost no server-rendered text.
  if js_app_shell?(body) && page[:content].length < SHELL_CONTENT_THRESHOLD
    raise EmptyContentError,
      "JS-app shell at #{url} — server HTML renders only #{page[:content].length} chars; a rendering backend is required"
  end

  page
rescue Errno::ECONNREFUSED, Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  # The transport maps its own failures to TimeoutError; this guard
  # keeps the "only Ask::WebFetch errors escape" invariant even if
  # a transport bug lets a raw socket error through.
  raise TimeoutError, "#{e.class}: #{e.message}"
end

#js_app_shell?(body) ⇒ Boolean

Returns:

  • (Boolean) 


98
99
100
101
 
# File 'lib/ask/web_fetch/backends/local.rb', line 98

def js_app_shell?(body)
  body.to_s.match?(JS_APP_SHELL_MARKERS) ||
    (body.to_s.bytesize > SHELL_HTML_BYTES && markdown_visible_chars(body) < SHELL_CONTENT_THRESHOLD)
end

#markdown_visible_chars(body) ⇒ Object

A cheap text estimate from the raw HTML (tags stripped) — the "how much did the server actually render" number. Deliberately rough: it only feeds a shell-vs-page heuristic.



106
107
108
109
110
111
 
# File 'lib/ask/web_fetch/backends/local.rb', line 106

def markdown_visible_chars(body)
  body.to_s.gsub(/<script[\s\S]*?<\/script>/i, "")
    .gsub(/<style[\s\S]*?<\/style>/i, "")
    .gsub(/<[^>]+>/, " ")
    .gsub(/\s+/, " ").strip.length
end

#to_markdown(html, url) ⇒ Object

Parses html and returns { title:, description:, content:, licenses:, outlinks: } where content is clean markdown (pruned by the ContentFilter), licenses are the page's declared license signals ([] when it declares none), and outlinks are the page's RAW hrefs, resolved and scheme-filtered — nav and footer included, because a crawler's discovery layer reads these even when the stored content is pruned.



120
121
122
123
 
# File 'lib/ask/web_fetch/backends/local.rb', line 120

def to_markdown(html, url)
  Markdown.generate(html, base_url: url, filter: self.class.content_filter)
    .merge(licenses: license_signals(html), outlinks: outlink_urls(html, url))
end