Class: Ghostcrawl::CrawlRunsClient

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

Manage crawl runs — /v1/crawl-runs.

Instance Method Summary collapse

Constructor Details

#initialize(v1) ⇒ CrawlRunsClient

Returns a new instance of CrawlRunsClient.



466
467
468
 
# File 'lib/ghostcrawl/client.rb', line 466

def initialize(v1)
  @v1 = v1
end

Instance Method Details

#cancel(run_id) ⇒ Object

Cancel a running crawl run. Delegates to POST /v1/crawl-runs/run_id/cancel via the generated builder.



502
503
504
 
# File 'lib/ghostcrawl/client.rb', line 502

def cancel(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).cancel.post)
end

#get(run_id) ⇒ Object

Get a single crawl run by ID. Delegates to GET /v1/crawl-runs/run_id via the generated builder.



496
497
498
 
# File 'lib/ghostcrawl/client.rb', line 496

def get(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).get)
end

#list ⇒ Object

List crawl runs. Delegates to GET /v1/crawl-runs via the generated CrawlRunsRequestBuilder.



490
491
492
 
# File 'lib/ghostcrawl/client.rb', line 490

def list
  ResponseHelper.to_hash(@v1.crawl_runs.get)
end

#start(url:, max_depth: 2, max_pages: 100, raise_on_result_error: true, **opts) ⇒ Hash

Start a new crawl run from a seed URL. Delegates to POST /v1/crawl-runs via the generated CrawlRunsRequestBuilder. The endpoint is a tagged union: a start request requires action: "start" and a seed_urls array (not a bare url).

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    crawl run record with run_id and status



480
481
482
483
484
485
486
 
# File 'lib/ghostcrawl/client.rb', line 480

def start(url:, max_depth: 2, max_pages: 100, raise_on_result_error: true, **opts)
  body = AdditionalDataBody.new({ "action" => "start", "seed_urls" => [url],
                                  "max_depth" => max_depth, "max_pages" => max_pages }
                                  .merge(opts.transform_keys(&:to_s)))
  hash = ResponseHelper.to_hash(@v1.crawl_runs.post(body))
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end