Module: Portage::Cli::SearchBackends

Defined in:
lib/portage/cli/search_backends.rb

Overview

Where "which stores might sell this?" comes from when the caller never named a store.

Every backend here talks to a documented machine interface and returns bare candidate URLs. None of them parse a results page: scraping a search engine's HTML is the same class of ToS violation Buy already refuses to commit against a merchant, and it would be odd to be scrupulous about the shop and cavalier about the index. That rules out the usual html.duckduckgo.com/html/?q= trick — see DuckDuckGo below for what we use instead and what it costs us.

Defined Under Namespace

Classes: Allowlist, Brave, DuckDuckGo, GoogleCse

Constant Summary collapse

OPEN_TIMEOUT = 
5
READ_TIMEOUT = 
5
USER_AGENT = 
"portage-find".freeze
NON_STORE_HOSTS =

Reference works and marketplaces-of-links that a search backend will happily return for a product query but that are never themselves a UCP store — cheap to skip, and each one skipped is one fewer host we probe.

%w[
  wikipedia.org wikimedia.org duckduckgo.com google.com bing.com
  reddit.com youtube.com facebook.com x.com twitter.com pinterest.com
].freeze

Class Method Summary collapse

Class Method Details

.default ⇒ Object

Ordered cheapest/most-trusted first: your own allowlist costs no network call and needs no key, DuckDuckGo needs no key, the keyed engines only participate when their credentials are actually present.



34
35
36
 
# File 'lib/portage/cli/search_backends.rb', line 34

def self.default
  [Allowlist.new, DuckDuckGo.new, Brave.new, GoogleCse.new].select(&:available?)
end

.get_json(uri, params: {}, headers: {}) ⇒ Object 



38
39
40
41
42
43
44
45
 
# File 'lib/portage/cli/search_backends.rb', line 38

def self.get_json(uri, params: {}, headers: {})
  uri = uri.dup
  uri.query = URI.encode_www_form(params) unless params.empty?
  response = request(uri, headers)
  response.is_a?(Net::HTTPSuccess) ? JSON.parse(response.body) : nil
rescue StandardError
  nil
end

.store_candidate?(url) ⇒ Boolean

Returns true when the URL is worth spending a manifest probe on.

Returns:

  • (Boolean) —

    true when the URL is worth spending a manifest probe on.



56
57
58
59
60
61
 
# File 'lib/portage/cli/search_backends.rb', line 56

def self.store_candidate?(url)
  host = URI.parse(url.to_s).host
  !!host && NON_STORE_HOSTS.none? { |bad| host == bad || host.end_with?(".#{bad}") }
rescue URI::InvalidURIError
  false
end