Class: Pikuri::Tool::Search::DuckDuckGo

Inherits:
Object
  • Object
show all
Defined in:
lib/pikuri/tool/search/duckduckgo.rb

Overview

Performs a DuckDuckGo search by scraping html.duckduckgo.com, returning hits as Result rows. Split into a thin HTTP fetch (#search) and a pure parser (.parse) so tests exercise the parser against fixture HTML. Engines#search owns the final Markdown rendering. Constructed with no arguments, sharing the uniform +#search+/+#label+ provider shape with the keyed Brave / Exa.

Privacy posture

DDG doesn't save your IP or tie a per-user profile to queries, and proxies requests so downstream providers can't build a search history — real, but DDG relays web results over Bing, so query content still reaches Microsoft (who has no no-training pledge). Better than Exa for sensitive queries, worse than Brave; for anything genuinely embarrassing, don't search the web at all.

Constant Summary collapse

ENDPOINT =

Returns HTML search endpoint.

Returns:

  • (String) —

    HTML search endpoint

'https://html.duckduckgo.com/html/'
USER_AGENT =

Returns User-Agent sent with each request; DDG often rejects a missing or obviously-bot UA.

Returns:

  • (String) —

    User-Agent sent with each request; DDG often rejects a missing or obviously-bot UA.

'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 ' \
'(KHTML, like Gecko) Chrome/120.0 Safari/537.36'
DEFAULT_MAX_RESULTS =

Returns default number of results returned.

Returns:

  • (Integer) —

    default number of results returned.

10
LIMITER =

Returns paces calls (DDG bans IPs that hammer the HTML endpoint) and circuit-breaks 5 min on Engines::Unavailable.

Returns:

RateLimiter.new(min_interval: 2.0, cooldown: 300.0)

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.extract_url(href) ⇒ String

Decode DDG's //duckduckgo.com/l/?uddg=<encoded> redirect wrapper to the real target URL.

Parameters:

  • href (String, nil) —

    href from the result page

Returns:

  • (String) —

    the decoded target, or href unchanged when not a recognized DDG redirect or unparseable



161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/pikuri/tool/search/duckduckgo.rb', line 161

def self.extract_url(href)
  return href if href.nil? || href.empty?

  uri = URI.parse(href.start_with?('//') ? "https:#{href}" : href)
  return href unless uri.host&.end_with?('duckduckgo.com') && uri.path == '/l/'

  params = URI.decode_www_form(uri.query.to_s).to_h
  params['uddg'] || href
rescue URI::InvalidURIError
  href
end

.parse(html, max_results: DEFAULT_MAX_RESULTS) ⇒ Array<Result>

Parse a html.duckduckgo.com result page into Result rows (++ highlights stripped). On zero result nodes: a genuine "no results" page returns [] (so Engines#search renders its stub); the anomaly/CAPTCHA modal raises Engines::Unavailable; anything else raises RuntimeError (likely a layout change) — so an IP soft-block surfaces rather than masquerading as an empty search.

Parameters:

  • html (String) —

    HTML from html.duckduckgo.com

  • max_results (Integer) (defaults to: DEFAULT_MAX_RESULTS) —

    max result entries

Returns:

  • (Array<Result>) —

    hits, possibly empty on a genuine no-results page

Raises:

  • (Engines::Unavailable) —

    on the anomaly/CAPTCHA modal (IP soft-block)

  • (RuntimeError) —

    on zero results with an unrecognized layout



85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
 
# File 'lib/pikuri/tool/search/duckduckgo.rb', line 85

def self.parse(html, max_results: DEFAULT_MAX_RESULTS)
  doc = Nokogiri::HTML(html)
  results = doc.css('div.result.web-result').take(max_results).filter_map do |node|
    title_link = node.at_css('a.result__a')
    next nil if title_link.nil?

    snippet = node.at_css('a.result__snippet')
    Result.new(
      url: extract_url(title_link['href']),
      title: title_link.text.strip,
      body: snippet&.text&.strip.to_s
    )
  end

  if results.empty?
    return [] if genuine_no_results?(doc)

    message = diagnose_empty(doc)
    raise(anomaly_modal?(doc) ? Engines::Unavailable : RuntimeError, message)
  end

  results
end

Instance Method Details

#label ⇒ String

Returns short provider label for Engines.

Returns:

  • (String) —

    short provider label for Engines.



39
40
41
 
# File 'lib/pikuri/tool/search/duckduckgo.rb', line 39

def label
  'DuckDuckGo'
end

#search(query, max_results: DEFAULT_MAX_RESULTS, cancellable: Pikuri::Agent::Control::Cancellable::NEVER) ⇒ Array<Result>

Fetch results for query as an Array<Result>. Throttled to one every 2s and circuit-broken 5 min after a soft-block (see LIMITER). The caller (Engines#search) normalizes the query and wraps this in a cache.

Parameters:

  • query (String) —

    search query (already normalized)

  • max_results (Integer) (defaults to: DEFAULT_MAX_RESULTS) —

    max result entries

  • cancellable (Pikuri::Agent::Control::Cancellable) (defaults to: Pikuri::Agent::Control::Cancellable::NEVER) —

    makes LIMITER's pacing wait interruptible; see RateLimiter#call.

Returns:

  • (Array<Result>) —

    hits, possibly empty when DDG matched nothing

Raises:

  • (Engines::Unavailable) —

    on a soft-block (anomaly/CAPTCHA page) or HTTP 429/5xx (the cascade falls back), or if LIMITER is in cooldown.

  • (RuntimeError) —

    on other HTTP failures or an unrecognized empty-results layout. A genuine empty-results page is not an error; see parse.



57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
 
# File 'lib/pikuri/tool/search/duckduckgo.rb', line 57

def search(query, max_results: DEFAULT_MAX_RESULTS,
           cancellable: Pikuri::Agent::Control::Cancellable::NEVER)
  LIMITER.call(cancellable: cancellable) do
    response = Faraday.get(ENDPOINT, { q: query }, { 'User-Agent' => USER_AGENT })
    unless response.success?
      if response.status == 429 || response.status >= 500
        raise Engines::Unavailable, "HTTP #{response.status}"
      end

      raise "DuckDuckGo request failed: #{response.status} #{response.body}"
    end

    self.class.parse(response.body, max_results: max_results)
  end
end