Class: Ghostcrawl::CrawlRunsClient

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

Manage crawl runs — /v1/crawl-runs.

Constant Summary collapse

TERMINAL_STATUSES =

Terminal run states — a crawl run in any of these will never change again, so a wait can stop. Both spellings of "cancelled" are accepted.

%w[completed failed cancelled canceled].freeze
WAIT_WINDOW_S =

Per-request server-block window (seconds) for the long-poll wait. Each GET ...?wait=true&timeout_s=WAIT_WINDOW_S makes the SERVER block for up to this long, so the client never sleeps between checks. Kept well under the client read timeout (Ghostcrawl::Client::DEFAULT_TIMEOUT) so a single blocking request can never trip the transport timeout.

30

Instance Method Summary collapse

Constructor Details

#initialize(v1, adapter = nil) ⇒ CrawlRunsClient

Returns a new instance of CrawlRunsClient.



477
478
479
480
 
# File 'lib/ghostcrawl/client.rb', line 477

def initialize(v1, adapter = nil)
  @v1 = v1
  @adapter = adapter
end

Instance Method Details

#cancel(run_id) ⇒ Object

Cancel a running crawl run. Delegates to POST /v1/crawl-runs/run_id/cancel via the generated builder.



577
578
579
 
# File 'lib/ghostcrawl/client.rb', line 577

def cancel(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).cancel.post)
end

#get(run_id) ⇒ Object

Get a single crawl run by ID. Delegates to GET /v1/crawl-runs/run_id via the generated builder.



571
572
573
 
# File 'lib/ghostcrawl/client.rb', line 571

def get(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).get)
end

#list ⇒ Object

List crawl runs. Delegates to GET /v1/crawl-runs via the generated CrawlRunsRequestBuilder.



565
566
567
 
# File 'lib/ghostcrawl/client.rb', line 565

def list
  ResponseHelper.to_hash(@v1.crawl_runs.get)
end

#start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300, raise_on_result_error: true, **opts) ⇒ Hash

Start a new crawl run from a seed URL. Delegates to POST /v1/crawl-runs via the generated CrawlRunsRequestBuilder. The endpoint is a tagged union: a start request requires action: "start" and a seed_urls array (not a bare url).

Pass wait: true to block until the run reaches a terminal state (completed | failed | cancelled) or timeout seconds elapse. This sends wait_until: "completed" so the SERVER blocks — no client-side poll loop, no sleep. When the run is still running at timeout the current (non-terminal) record is returned; call #wait_for_completion again to keep waiting.

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • wait (Boolean) (defaults to: false) —

    block until the run is terminal (default false)

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait when wait: true (default 300)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true). Ignored on the wait: true path, which always returns the terminal run so the caller can inspect a failed status.

Returns:

  • (Hash) —

    crawl run record with run_id and status (results present when it completed while waiting)



505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
 
# File 'lib/ghostcrawl/client.rb', line 505

def start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300,
          raise_on_result_error: true, **opts)
  data = { "action" => "start", "seed_urls" => [url],
           "max_depth" => max_depth, "max_pages" => max_pages }
         .merge(opts.transform_keys(&:to_s))

  unless wait
    hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
    return raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
  end

  # Start-and-wait: ask the server to block until terminal. Bound the first
  # server block to a safe window, then long-poll the rest via GET so a low
  # per-request read timeout can never abort the whole wait.
  window = [timeout, WAIT_WINDOW_S].min
  data["wait_until"] = "completed"
  data["timeout_s"]  = window
  hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
  return hash if terminal?(hash)

  run_id = hash["run_id"]
  remaining = timeout - window
  return hash if run_id.nil? || run_id.to_s.empty? || remaining <= 0
  wait_for_completion(run_id, timeout: remaining)
end

#wait_for_completion(run_id, timeout: 300) ⇒ Hash

Block until an existing crawl run reaches a terminal state, or timeout seconds elapse. Event-driven: each iteration issues a SERVER-blocking GET /v1/crawl-runs/{run_id}?wait=true&timeout_s=N that returns the moment the run goes terminal (or after its own window). There is no client sleep — the wait cost lives on the server, and successive blocking windows are chained until the caller's timeout deadline.

On timeout the current non-terminal run is returned (never raises for a slow run); a terminal run is returned as soon as it is observed.

Parameters:

  • run_id (String) —

    the run to wait on

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait (default 300)

Returns:

  • (Hash) —

    the run record (terminal when it finished in time)



544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
 
# File 'lib/ghostcrawl/client.rb', line 544

def wait_for_completion(run_id, timeout: 300)
  if run_id.nil? || run_id.to_s.empty?
    raise ArgumentError, "run_id is required"
  end

  deadline = monotonic + timeout
  loop do
    remaining = deadline - monotonic
    # Deadline reached — one final non-blocking read of the current state.
    return get(run_id) if remaining <= 0

    window = [remaining, WAIT_WINDOW_S].min
    run = fetch_waiting(run_id, window)
    return run if terminal?(run)
    # The server just blocked for ~window seconds; loop straight into the
    # next blocking window. No client-side delay.
  end
end