Class: ScrapeUnblocker::Client

Inherits:
Object
  • Object
show all
Defined in:
lib/scrapeunblocker/client.rb

Overview

Client for the ScrapeUnblocker API.

su = ScrapeUnblocker::Client.new(api_key: "YOUR_API_KEY")
html = su.get_page_source("https://example.com")

Constant Summary collapse

DEFAULT_BASE_URL = 
"https://api.scrapeunblocker.com"
API_KEY_HEADER = 
"x-scrapeunblocker-key"
RETRYABLE = 
[429, 502, 503, 504].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 180, max_retries: 2, transport: nil) ⇒ Client

Returns a new instance of Client.



25
26
27
28
29
30
31
32
33
34
35
36
37
 
# File 'lib/scrapeunblocker/client.rb', line 25

def initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 180, max_retries: 2, transport: nil)
  @api_key = api_key || ENV["SCRAPEUNBLOCKER_KEY"]
  if @api_key.nil? || @api_key.empty?
    raise Error, "No API key provided. Pass api_key: or set the " \
                 "SCRAPEUNBLOCKER_KEY environment variable. Get your key " \
                 "at https://app.scrapeunblocker.com"
  end
  @base_url = base_url.sub(%r{/+\z}, "")
  @timeout = timeout
  @max_retries = max_retries
  @transport = transport || method(:net_http_transport)
  @skyscanner = Skyscanner.new(self)
end

Instance Attribute Details

#skyscanner ⇒ Skyscanner (readonly)

Returns the Skyscanner plugin endpoints.

Returns:

  • (Skyscanner) —

    the Skyscanner plugin endpoints



23
24
25
 
# File 'lib/scrapeunblocker/client.rb', line 23

def skyscanner
  @skyscanner
end

Instance Method Details

#amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil) ⇒ Object

Scrape one Amazon product by ASIN or URL and return it as a Hash.

Returns title, brand, numeric price and currency, list price and savings, availability, rating, review count, seller, feature bullets, categories and images. Prices come back in the marketplace's own currency: proxy_country defaults to the marketplace's home country (amazon.com -> US), pinning the exit over the ISP pool. Pass either asin (with marketplace) or a full product url.



166
167
168
169
170
 
# File 'lib/scrapeunblocker/client.rb', line 166

def amazon_product(asin: nil, url: nil, marketplace: "amazon.com", proxy_country: nil)
  post_json("/marketplace/amazon-product",
            asin: asin, url: url, marketplace: marketplace,
            proxy_country: proxy_country)
end

#amazon_search(keyword, marketplace: "amazon.com", page: 1, sort: "featured", min_price: nil, max_price: nil, proxy_country: nil) ⇒ Object

Search Amazon and return the result cards as an Array of Hashes.

Each card carries asin, title, numeric price and currency, list price, rating, review count, a clean product URL, image and the sponsored / prime flags. sort is "featured" (default), "price_asc", "price_desc", "avg_review" or "newest". Prices are in the marketplace's own currency; proxy_country defaults to the marketplace's home country.



179
180
181
182
183
184
185
 
# File 'lib/scrapeunblocker/client.rb', line 179

def amazon_search(keyword, marketplace: "amazon.com", page: 1, sort: "featured",
                  min_price: nil, max_price: nil, proxy_country: nil)
  post_json("/marketplace/amazon-search",
            keyword: keyword, marketplace: marketplace, page: page,
            sort: sort, min_price: min_price, max_price: max_price,
            proxy_country: proxy_country)
end

#ebay_search(keyword, marketplace: "ebay.com", page: 1, page_size: 60, condition: nil, sort: "best_match", listing_type: "all", min_price: nil, max_price: nil, free_shipping: false, seller: nil, category: nil, proxy_country: nil) ⇒ Object

Search eBay and return the listings as a Hash.

Each listing carries title, numeric price and currency, condition (with a normalised conditionCode), seller username and feedback, shipping cost, sold/watcher/bid counts, image and a clean item URL.

marketplace is a regional eBay host such as "ebay.com" (default) or "ebay.de"; condition is one of "new", "open_box", "refurbished", "used" or "for_parts"; sort is one of "best_match" (default), "newly_listed", "ending_soon", "price_asc" or "price_desc"; listing_type is "all" (default), "buy_it_now" or "auction"; page_size is 60, 120 or 240.

When eBay finds no exact match it still serves a page of loosely related suggestions, and the response then carries exactMatches: false.



144
145
146
147
148
149
150
151
152
153
154
155
156
 
# File 'lib/scrapeunblocker/client.rb', line 144

def ebay_search(keyword, marketplace: "ebay.com", page: 1, page_size: 60,
                condition: nil, sort: "best_match", listing_type: "all",
                min_price: nil, max_price: nil, free_shipping: false,
                seller: nil, category: nil, proxy_country: nil)
  post_json("/marketplace/ebay-search",
            keyword: keyword, marketplace: marketplace, page: page,
            page_size: page_size, condition: condition, sort: sort,
            listing_type: listing_type, min_price: min_price,
            max_price: max_price,
            free_shipping: free_shipping ? true : nil,
            seller: seller, category: category,
            proxy_country: proxy_country)
end

#get_image(url, proxy_country: nil) ⇒ Object

Fetch an image URL through the bypass chain and return its raw bytes.



188
189
190
 
# File 'lib/scrapeunblocker/client.rb', line 188

def get_image(url, proxy_country: nil)
  request("/getImage", url: url, proxy_country: proxy_country)[:body]
end

#get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil, steps: nil, list_elements: nil) ⇒ Object

Fetch a URL and return the fully rendered HTML.

steps is an ordered Array of browser-action Hashes the API runs in the real browser after the page loads (each Hash carries an action and its fields): wait_for selector_type?, timeout_ms?, wait_for_text timeout_ms?, wait value, click selector_type?, timeout_ms?, type selector_type?, value, clear?, timeout_ms?, select selector_type?, value, timeout_ms?, press_key value, scroll value. selector_type is one of "css" (default), "xPath", "className" or "tagName". The steps run once and are not idempotent; if a step fails the API answers HTTP 422 with a JSON body naming the failed step, which surfaces here as a ScrapeUnblocker::ValidationError.

list_elements, when true, makes the API return a JSON summary of the matched elements ("count", "elements") instead of HTML. This method then returns that parsed Hash rather than an HTML String.



55
56
57
58
59
60
61
62
63
64
65
 
# File 'lib/scrapeunblocker/client.rb', line 55

def get_page_source(url, proxy_country: nil, time_sleep: nil, method: nil, value: nil, method_timeout: nil,
                    steps: nil, list_elements: nil)
  body = request("/getPageSource",
                 url: url, proxy_country: proxy_country, time_sleep: time_sleep,
                 method: method, value: value, method_timeout: method_timeout,
                 steps: (steps ? JSON.generate(steps) : nil),
                 list_elements: (list_elements ? true : nil))[:body]
  return JSON.parse(body) if list_elements

  body
end

#get_page_with_cookies(url, proxy_country: nil, time_sleep: nil) ⇒ Object

Fetch a URL and also return the cookies and proxy that served it.



78
79
80
81
82
83
 
# File 'lib/scrapeunblocker/client.rb', line 78

def get_page_with_cookies(url, proxy_country: nil, time_sleep: nil)
  body = request("/getPageSource",
                 url: url, get_cookies: true, proxy_country: proxy_country,
                 time_sleep: time_sleep)[:body]
  PageResult.from_hash(JSON.parse(body))
end

#get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil) ⇒ Object

Fetch a URL and return structured JSON instead of HTML.



68
69
70
71
72
73
74
75
 
# File 'lib/scrapeunblocker/client.rb', line 68

def get_parsed(url, proxy_country: nil, time_sleep: nil, refresh_rules: false, rules_hint: nil)
  body = request("/getPageSource",
                 url: url, parsed_data: true, proxy_country: proxy_country,
                 time_sleep: time_sleep,
                 refresh_rules: (refresh_rules ? true : nil),
                 rules_hint: rules_hint)[:body]
  ParsedPage.from_hash(JSON.parse(body))
end

#google_local(keyword, proxy_country: nil, hl: nil, gl: nil) ⇒ Object

Search Google Local (Maps) and return the businesses as a Hash.

Returns up to ~20 businesses, each with name, rating, reviews, price, category, address, hours and a top review snippet. Local results are location-sensitive, so set proxy_country (and optionally gl).



99
100
101
102
 
# File 'lib/scrapeunblocker/client.rb', line 99

def google_local(keyword, proxy_country: nil, hl: nil, gl: nil)
  post_json("/maps/google-local",
            keyword: keyword, proxy_country: proxy_country, hl: hl, gl: gl)
end

#meta_ad_library(advertiser, country: nil, active_status: nil, media_type: nil, max_ads: nil) ⇒ Object

Fetch an advertiser's Meta (Facebook) Ad Library ads and return them as a Hash.

advertiser is the advertiser name or page to look up. Optional filters: country (the Ad Library region), active_status (active or inactive ads), media_type (image, video, etc.) and max_ads (a cap on how many ads to return). Unset filters are dropped from the request and the API applies its own defaults.



111
112
113
114
115
 
# File 'lib/scrapeunblocker/client.rb', line 111

def meta_ad_library(advertiser, country: nil, active_status: nil, media_type: nil, max_ads: nil)
  post_json("/ads/meta-ad-library",
            advertiser: advertiser, country: country, active_status: active_status,
            media_type: media_type, max_ads: max_ads)
end

#oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil) ⇒ Object

Search Oopbuy (1688, Taobao or official channel) and return the goods as a Hash.

Returns matched products, each with spu, channel, title, titleCn, price, originalPrice, priceCny, monthSold, image and url. channel is one of "1688" (default), "taobao" or "official"; sort is one of "default", "price_asc", "price_desc" or "best_selling". page_size max is 60. Brand keywords return HTTP 422.



124
125
126
127
128
 
# File 'lib/scrapeunblocker/client.rb', line 124

def oopbuy_search(keyword, channel: "1688", page: 1, page_size: 20, sort: "default", proxy_country: nil)
  post_json("/goods/oopbuy-search",
            keyword: keyword, channel: channel, page: page,
            page_size: page_size, sort: sort, proxy_country: proxy_country)
end

#post_json(path, params) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



193
194
195
 
# File 'lib/scrapeunblocker/client.rb', line 193

def post_json(path, params)
  JSON.parse(request(path, params)[:body])
end

#serp(keyword, proxy_country: nil, pages_to_check: 1, wait_after_load: 0, captcha_pause: 0) ⇒ Object

Run a Google search and return the parsed SERP as a Hash.



86
87
88
89
90
91
92
 
# File 'lib/scrapeunblocker/client.rb', line 86

def serp(keyword, proxy_country: nil, pages_to_check: 1, wait_after_load: 0, captcha_pause: 0)
  post_json("/serpApi",
            keyword: keyword, proxy_country: proxy_country,
            pages_to_check: pages_to_check,
            wait_after_load: (wait_after_load.zero? ? nil : wait_after_load),
            captcha_pause: (captcha_pause.zero? ? nil : captcha_pause))
end