Class: ScrapeUnblocker::Client

Inherits:
Object
  • Object
show all
Defined in:
lib/scrapeunblocker/client.rb

Overview

Client for the ScrapeUnblocker API.

su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
html = su.get_page_source("https://example.com")

Constant Summary collapse

DEFAULT_BASE_URL = 
"https://api.scrapeunblocker.com"
API_KEY_HEADER = 
"x-scrapeunblocker-key"
RETRYABLE = 
[429, 502, 503, 504].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 180, max_retries: 2, transport: nil) ⇒ Client

Returns a new instance of Client.



25
26
27
28
29
30
31
32
33
34
35
36
37
 
# File 'lib/scrapeunblocker/client.rb', line 25

def initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 180, max_retries: 2, transport: nil)
  @api_key = api_key || ENV["SCRAPEUNBLOCKER_KEY"]
  if @api_key.nil? || @api_key.empty?
    raise Error, "No API key provided. Pass api_key: or set the " \
                 "SCRAPEUNBLOCKER_KEY environment variable. Get your key " \
                 "at https://app.scrapeunblocker.com"
  end
  @base_url = base_url.sub(%r{/+\z}, "")
  @timeout = timeout
  @max_retries = max_retries
  @transport = transport || method(:net_http_transport)
  @skyscanner = Skyscanner.new(self)
end

Instance Attribute Details

#skyscanner ⇒ Skyscanner (readonly)

Returns the Skyscanner plugin endpoints.

Returns:

  • (Skyscanner) —

    the Skyscanner plugin endpoints



23
24
25
 
# File 'lib/scrapeunblocker/client.rb', line 23

def skyscanner
  @skyscanner
end

Instance Method Details

#amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil) ⇒ Object

Scrape one Amazon product by ASIN or URL and return it as a Hash.

Returns title, brand, numeric price and currency, list price and savings, availability, rating, review count, seller, feature bullets, categories and images. Prices come back in the marketplace's own currency: proxy_country defaults to the marketplace's home country (amazon.com -> US), pinning the exit over the ISP pool. Pass either asin (with marketplace) or a full product url.



153
154
155
156
157
 
# File 'lib/scrapeunblocker/client.rb', line 153

def amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil)
  post_json("/marketplace/amazon-product",
            asin: asin, url: url, marketplace: marketplace,
            proxy_country: proxy_country)
end

#amazon_search(keyword, marketplace: "amazon.com", page: 1, sort: "featured", min_price: nil, max_price: nil, proxy_country: nil) ⇒ Object

Search Amazon and return the result cards as an Array of Hashes.

Each card carries asin, title, numeric price and currency, list price, rating, review count, a clean product URL, image and the sponsored / prime flags. sort is "featured" (default), "price_asc", "price_desc", "avg_review" or "newest". Prices are in the marketplace's own currency; proxy_country defaults to the marketplace's home country.



166
167
168
169
170
171
172
 
# File 'lib/scrapeunblocker/client.rb', line 166

def amazon_search(keyword, marketplace: "amazon.com", page: 1, sort: "featured",
                  min_price: nil, max_price: nil, proxy_country: nil)
  post_json("/marketplace/amazon-search",
            keyword: keyword, marketplace: marketplace, page: page,
            sort: sort, min_price: min_price, max_price: max_price,
            proxy_country: proxy_country)
end

#ebay_search(keyword, marketplace: "ebay.com", page: 1, page_size: 60, condition: nil, sort: "best_match", listing_type: "all", min_price: nil, max_price: nil, free_shipping: false, seller: nil, category: nil, proxy_country: nil) ⇒ Object

Search eBay and return the listings as a Hash.

Each listing carries title, numeric price and currency, condition (with a normalised conditionCode), seller username and feedback, shipping cost, sold/watcher/bid counts, image and a clean item URL.

marketplace is a regional eBay host such as "ebay.com" (default) or "ebay.de"; condition is one of "new", "open_box", "refurbished", "used" or "for_parts"; sort is one of "best_match" (default), "newly_listed", "ending_soon", "price_asc" or "price_desc"; listing_type is "all" (default), "buy_it_now" or "auction"; page_size is 60, 120 or 240.

When eBay finds no exact match it still serves a page of loosely related suggestions, and the response then carries exactMatches: false.



131
132
133
134
135
136
137
138
139
140
141
142
143
 
# File 'lib/scrapeunblocker/client.rb', line 131

def ebay_search(keyword, marketplace: "ebay.com", page: 1, page_size: 60,
                condition: nil, sort: "best_match", listing_type: "all",
                min_price: nil, max_price: nil, free_shipping: false,
                seller: nil, category: nil, proxy_country: nil)
  post_json("/marketplace/ebay-search",
            keyword: keyword, marketplace: marketplace, page: page,
            page_size: page_size, condition: condition, sort: sort,
            listing_type: listing_type, min_price: min_price,
            max_price: max_price,
            free_shipping: free_shipping ? true : nil,
            seller: seller, category: category,
            proxy_country: proxy_country)
end

#get_image(url, proxy_country: nil) ⇒ Object

Fetch an image URL through the bypass chain and return its raw bytes.



175
176
177
 
# File 'lib/scrapeunblocker/client.rb', line 175

def get_image(url, proxy_country: nil)
  request("/getImage", url: url, proxy_country: proxy_country)[:body]
end

#get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil, steps: nil, list_elements: nil) ⇒ Object

Fetch a URL and return the fully rendered HTML.

steps is an ordered Array of browser-action Hashes the API runs in the real browser after the page loads (each Hash carries an action and its fields): wait_for selector_type?, timeout_ms?, wait_for_text timeout_ms?, wait value, click selector_type?, timeout_ms?, type selector_type?, value, clear?, timeout_ms?, select selector_type?, value, timeout_ms?, press_key value, scroll value. selector_type is one of "css" (default), "xPath", "className" or "tagName". The steps run once and are not idempotent; if a step fails the API answers HTTP 422 with a JSON body naming the failed step, which surfaces here as a ScrapeUnblocker::ValidationError.

list_elements, when true, makes the API return a JSON summary of the matched elements ("count", "elements") instead of HTML. This method then returns that parsed Hash rather than an HTML String.



55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/scrapeunblocker/client.rb', line 55

def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
                    steps: nil, list_elements: nil)
  body = request("/getPageSource",
                 url: url, proxy_country: proxy_country, time_sleep: time_sleep,
                 method: method, value: value, method_timeout: method_timeout,
                 steps: (steps ? JSON.generate(steps) : nil),
                 list_elements: (list_elements ? true : nil))[:body]
  return JSON.parse(body) if list_elements

  body
end

#get_page_with_cookies(url, proxy_country: nil, time_sleep: nil) ⇒ Object

Fetch a URL and also return the cookies and proxy that served it.



78
79
80
81
82
83
 
# File 'lib/scrapeunblocker/client.rb', line 78

def get_page_with_cookies(url, proxy_country: nil, time_sleep: nil)
  body = request("/getPageSource",
                 url: url, get_cookies: true, proxy_country: proxy_country,
                 time_sleep: time_sleep)[:body]
  PageResult.from_hash(JSON.parse(body))
end

#get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil) ⇒ Object

Fetch a URL and return structured JSON instead of HTML.



68
69
70
71
72
73
74
75
 
# File 'lib/scrapeunblocker/client.rb', line 68

def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
  body = request("/getPageSource",
                 url: url, parsed_data: true, proxy_country: proxy_country,
                 time_sleep: time_sleep,
                 refresh_rules: (refresh_rules ? true : nil),
                 rules_hint: rules_hint)[:body]
  ParsedPage.from_hash(JSON.parse(body))
end

#google_local(keyword, proxy_country: nil, hl: nil, gl: nil) ⇒ Object

Search Google Local (Maps) and return the businesses as a Hash.

Returns up to ~20 businesses, each with name, rating, reviews, price, category, address, hours and a top review snippet. Local results are location-sensitive, so set proxy_country (and optionally gl).



99
100
101
102
 
# File 'lib/scrapeunblocker/client.rb', line 99

def google_local(keyword, proxy_country: nil, hl: nil, gl: nil)
  post_json("/maps/google-local",
            keyword: keyword, proxy_country: proxy_country, hl: hl, gl: gl)
end

#oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil) ⇒ Object

Search Oopbuy (1688, Taobao or official channel) and return the goods as a Hash.

Returns matched products, each with spu, channel, title, titleCn, price, originalPrice, priceCny, monthSold, image and url. channel is one of "1688" (default), "taobao" or "official"; sort is one of "default", "price_asc", "price_desc" or "best_selling". page_size max is 60. Brand keywords return HTTP 422.



111
112
113
114
115
 
# File 'lib/scrapeunblocker/client.rb', line 111

def oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil)
  post_json("/goods/oopbuy-search",
            keyword: keyword, channel: channel, page: page,
            page_size: page_size, sort: sort, proxy_country: proxy_country)
end

#post_json(path, params) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



180
181
182
 
# File 'lib/scrapeunblocker/client.rb', line 180

def post_json(path, params)
  JSON.parse(request(path, params)[:body])
end

#serp(keyword, proxy_country: nil, pages_to_check: 1, wait_after_load: 0, captcha_pause: 0) ⇒ Object

Run a Google search and return the parsed SERP as a Hash.



86
87
88
89
90
91
92
 
# File 'lib/scrapeunblocker/client.rb', line 86

def serp(keyword, proxy_country: nil, pages_to_check: 1, wait_after_load: 0, captcha_pause: 0)
  post_json("/serpApi",
            keyword: keyword, proxy_country: proxy_country,
            pages_to_check: pages_to_check,
            wait_after_load: (wait_after_load.zero? ? nil : wait_after_load),
            captcha_pause: (captcha_pause.zero? ? nil : captcha_pause))
end