Class: Ask::WebFetch::Http

Inherits:
Object
  • Object
show all
Defined in:
lib/ask/web_fetch/http.rb

Overview

Minimal pooled HTTP client for the Local backend. Wraps a per-thread httpx session so every page fetch in a thread reuses its keep-alive connection to the host — no fresh TCP+TLS handshake per page (the thing that made the Net::HTTP crawler cost ~1.25s per page) — with HTTP/2 when the server negotiates it, retries with backoff on transient failures, and automatic gzip/deflate decoding.

Deliberately single-hop: the backend follows redirects itself, so the hop-by-hop chain it reports is exactly what happened. Error responses (4xx/5xx) are not errors to the transport — the backend decides what they mean. Transport-level failures (timeout, refused, reset, DNS, TLS) all surface as Ask::WebFetch::TimeoutError, the transient bucket, whatever their underlying class.

Defined Under Namespace

Classes: Response

Constant Summary collapse

CONNECT_TIMEOUT = 
5
READ_TIMEOUT = 
15
WRITE_TIMEOUT = 
15
OPERATION_TIMEOUT =

Whole-request cap. A page that can't be read in a minute is a problem page, not a stall worth a crawl worker.

60
MAX_RETRIES =

Retries per request, on top of the crawl ledger's own auto-heal rounds. GETs are idempotent; a couple of cheap retries beat a full ledger round-trip for transient flakiness.

2
RETRY_ON_STATUS = 
[429, 500, 502, 503, 504].freeze

Class Method Summary collapse

Class Method Details

.get(url, headers: {}) ⇒ Object 



37
38
39
40
41
42
43
44
45
46
47
 
# File 'lib/ask/web_fetch/http.rb', line 37

def self.get(url, headers: {})
  response = session.get(url, headers: headers)
  return raise_timeout(response) if response.is_a?(HTTPX::ErrorResponse)

  Response.new(
    status: response.status,
    body: response.body.to_s,
    content_type: response.headers['content-type'].to_s,
    location: response.headers['location'].to_s
  )
end

.session ⇒ Object

One pooled session per thread — httpx sessions are not thread-safe, and a crawl worker thread reusing its session across every page it fetches is what keeps timeouts and retries configured once. Sessions idle-close themselves after keep-alive timeout, so nothing to reap.



54
55
56
 
# File 'lib/ask/web_fetch/http.rb', line 54

def self.session
  Thread.current[SESSION_KEY] ||= build_session
end