Class: Ask::WebFetch::Backends::Jina

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/jina.rb

Overview

Jina Reader free tier: GET https://r.jina.ai/.

Free without a key (~20 req/min per IP). Set JINA_API_KEY for higher rate limits. The endpoint runs headless Chromium, so it renders JS pages that the Local backend cannot.

Constant Summary collapse

BASE_URL = 
'https://r.jina.ai'
OPEN_TIMEOUT = 
5
READ_TIMEOUT = 
30

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::PARKED_DOMAIN_MARKERS, Ask::WebFetch::Backend::USER_AGENT

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name, #guard_page!, #markdown_outlinks, #outlink_urls

Instance Method Details

#fetch(url) ⇒ Object 



21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/ask/web_fetch/backends/jina.rb', line 21

def fetch(url)
  uri = URI("#{BASE_URL}/#{url}")
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  http.open_timeout = OPEN_TIMEOUT
  http.read_timeout = READ_TIMEOUT
  req = Net::HTTP::Get.new(uri)
  req['User-Agent'] = USER_AGENT
  req['Accept'] = 'text/markdown, text/plain, text/html'
  req['Authorization'] = "Bearer #{ENV['JINA_API_KEY']}" if ENV['JINA_API_KEY']

  res = http.request(req)
  case res.code
  when '200'
    body = res.body.to_s
    raise FetchError, 'challenge page from Jina' if challenge_page?(body)

    content = Markdown.clean(body)
    # Jina only sees rendered markdown — outlinks come from its
    # links, resolved against the requested URL. Content runs
    # through the same Markdown.clean as the converting backends,
    # so decorative symbol noise is stripped here too; a page
    # whose only "content" was noise falls through as empty. A
    # registrar parking page renders fine through Jina — the
    # shared guard's prose markers catch it (the HTML-only
    # markers never reach a markdown-only backend), so the ad is
    # rejected, not returned as the site's content.
    guard_page!(url, content)

    { title: nil, description: nil, content: content, outlinks: markdown_outlinks(body, url) }
  when '429'
    raise ServerError, 'rate limited by Jina (429)'
  when '401', '403'
    raise FetchError, "Jina access error (#{res.code})"
  else
    # 5xx = Jina-side blip (transient); other 4xx = the URL is dead.
    raise(res.code.to_i >= 500 ? ServerError : FetchError, "Jina returned #{res.code}")
  end
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
       Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  raise TimeoutError, "Jina #{e.class}: #{e.message}"
end