Class: Ask::WebFetch::Backends::Local

Inherits:
Ask::WebFetch::Backend show all
Defined in:
lib/ask/web_fetch/backends/local.rb

Overview

Default backend: pure Ruby Net::HTTP + Nokogiri + reverse_markdown. No external service or API key; mirrors ask-web-search's self-hosted SearXNG approach.

Constant Summary collapse

MAX_REDIRECTS = 
5
OPEN_TIMEOUT = 
5
READ_TIMEOUT = 
15
/
  (^|[\s_-])(nav|menu|toolbar|breadcrumb|sidebar|toc|footer|header|
  banner|pagination|search|cookie|modal|popup)([\s_-]|$)
/ix

Constants inherited from Ask::WebFetch::Backend

Ask::WebFetch::Backend::CHALLENGE_RE, Ask::WebFetch::Backend::MIN_CONTENT_LENGTH, Ask::WebFetch::Backend::USER_AGENT

Instance Method Summary collapse

Methods inherited from Ask::WebFetch::Backend

backend_name

Instance Method Details

#fetch(url) ⇒ Object 



27
28
29
30
31
32
33
34
35
36
37
38
39
 
# File 'lib/ask/web_fetch/backends/local.rb', line 27

def fetch(url)
  body, content_type = fetch_html(url)
  raise FetchError, "expected HTML from #{url}, got #{content_type}" unless content_type.include?('html')
  raise FetchError, "challenge page at #{url}" if challenge_page?(body)

  page = to_markdown(body, url)
  raise EmptyContentError, "no readable content at #{url}" unless usable_content?(page[:content])

  page
rescue Net::OpenTimeout, Net::ReadTimeout, Errno::ECONNREFUSED,
       Errno::ECONNRESET, SocketError, URI::InvalidURIError => e
  raise FetchError, "#{e.class}: #{e.message}"
end

#to_markdown(html, _url) ⇒ Object

Parses html and returns { title:, content: } where content is clean markdown.



43
44
45
46
47
48
49
50
51
 
# File 'lib/ask/web_fetch/backends/local.rb', line 43

def to_markdown(html, _url)
  doc = Nokogiri::HTML(html)
  candidate = extract_main(doc)
  scrub(candidate)
  markdown = ReverseMarkdown.convert(candidate.to_html, unknown_tags: :bypass, github_flavored: true)
  markdown = clean(markdown)
  title = doc.at('title')&.text&.strip
  { title: title, content: markdown }
end