Class: BandcampDiscover::Scrapers::Discover

Inherits:
Object
  • Object
show all
Defined in:
lib/bandcamp-discover/scrapers/discover.rb

Overview

Not Playwright: the /discover/ grid only exists after the Vue app hydrates, and Fastly serves that page a JS challenge to datacenter IPs that headless Chrome never clears. The XHR behind it is unchallenged.

It is also private and unversioned, hence assert_shape!.

Defined Under Namespace

Classes: ResponseError

Constant Summary collapse

API_URI = 
URI("https://bandcamp.com/api/discover/1/discover_web")
PAGE_SIZE = 
60
OPEN_TIMEOUT = 
10
READ_TIMEOUT = 
30
RESULT_TYPES =

"a" is albums, "s" is subscriptions.

%w[a s].freeze

Instance Method Summary collapse

Constructor Details

#initialize(genre:, browser: nil, max_tasks: Concurrent.processor_count, pages: 1) ⇒ Discover

browser: is unused, kept so existing callers do not break.



29
30
31
32
33
34
 
# File 'lib/bandcamp-discover/scrapers/discover.rb', line 29

def initialize(genre:, browser: nil, max_tasks: Concurrent.processor_count, pages: 1)
  @genre = genre
  @browser = browser
  @max_tasks = max_tasks
  @pages = pages
end

Instance Method Details

#scrape(force: false) ⇒ Object 



36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/bandcamp-discover/scrapers/discover.rb', line 36

def scrape(force: false)
  urls = band_urls

  barrier = Async::Barrier.new

  Sync do
    semaphore = Async::Semaphore.new(@max_tasks, parent: barrier)

    urls.map do |url|
      semaphore.async do
        if block_given?
          yield url
        else
          Scrapers::Label.new(url: url, browser: @browser).scrape
        end
      end
    end.map(&:wait).compact
  ensure
    barrier.stop
  end
end