Class: Ghostcrawl::CrawlRunsClient

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

Manage crawl runs — /v1/crawl-runs.

Constant Summary collapse

TERMINAL_STATUSES =

Terminal run states — a crawl run in any of these will never change again, so a wait can stop. Both spellings of "cancelled" are accepted.

%w[completed failed cancelled canceled].freeze
WAIT_WINDOW_S =

Per-request server-block window (seconds) for the long-poll wait. Each GET ...?wait=true&timeout_s=WAIT_WINDOW_S makes the SERVER block for up to this long, so the client never sleeps between checks. Kept well under the client read timeout (Ghostcrawl::Client::DEFAULT_TIMEOUT) so a single blocking request can never trip the transport timeout.

30

Instance Method Summary collapse

Constructor Details

#initialize(v1, adapter = nil) ⇒ CrawlRunsClient

Returns a new instance of CrawlRunsClient.



518
519
520
521
 
# File 'lib/ghostcrawl/client.rb', line 518

def initialize(v1, adapter = nil)
  @v1 = v1
  @adapter = adapter
end

Instance Method Details

#cancel(run_id) ⇒ Object

Cancel a running crawl run. Delegates to POST /v1/crawl-runs/run_id/cancel via the generated builder.



618
619
620
 
# File 'lib/ghostcrawl/client.rb', line 618

def cancel(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).cancel.post)
end

#get(run_id) ⇒ Object

Get a single crawl run by ID. Delegates to GET /v1/crawl-runs/run_id via the generated builder.



612
613
614
 
# File 'lib/ghostcrawl/client.rb', line 612

def get(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).get)
end

#list ⇒ Object

List crawl runs. Delegates to GET /v1/crawl-runs via the generated CrawlRunsRequestBuilder.



606
607
608
 
# File 'lib/ghostcrawl/client.rb', line 606

def list
  ResponseHelper.to_hash(@v1.crawl_runs.get)
end

#start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300, raise_on_result_error: true, **opts) ⇒ Hash

Start a new crawl run from a seed URL. Delegates to POST /v1/crawl-runs via the generated CrawlRunsRequestBuilder. The endpoint is a tagged union: a start request requires action: "start" and a seed_urls array (not a bare url).

Pass wait: true to block until the run reaches a terminal state (completed | failed | cancelled) or timeout seconds elapse. This sends wait_until: "completed" so the SERVER blocks — no client-side poll loop, no sleep. When the run is still running at timeout the current (non-terminal) record is returned; call #wait_for_completion again to keep waiting.

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • wait (Boolean) (defaults to: false) —

    block until the run is terminal (default false)

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait when wait: true (default 300)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true). Ignored on the wait: true path, which always returns the terminal run so the caller can inspect a failed status.

Returns:

  • (Hash) —

    crawl run record with run_id and status (results present when it completed while waiting)



546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
 
# File 'lib/ghostcrawl/client.rb', line 546

def start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300,
          raise_on_result_error: true, **opts)
  data = { "action" => "start", "seed_urls" => [url],
           "max_depth" => max_depth, "max_pages" => max_pages }
         .merge(opts.transform_keys(&:to_s))

  unless wait
    hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
    return raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
  end

  # Start-and-wait: ask the server to block until terminal. Bound the first
  # server block to a safe window, then long-poll the rest via GET so a low
  # per-request read timeout can never abort the whole wait.
  window = [timeout, WAIT_WINDOW_S].min
  data["wait_until"] = "completed"
  data["timeout_s"]  = window
  hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
  return hash if terminal?(hash)

  run_id = hash["run_id"]
  remaining = timeout - window
  return hash if run_id.nil? || run_id.to_s.empty? || remaining <= 0
  wait_for_completion(run_id, timeout: remaining)
end

#wait_for_completion(run_id, timeout: 300) ⇒ Hash

Block until an existing crawl run reaches a terminal state, or timeout seconds elapse. Event-driven: each iteration issues a SERVER-blocking GET /v1/crawl-runs/{run_id}?wait=true&timeout_s=N that returns the moment the run goes terminal (or after its own window). There is no client sleep — the wait cost lives on the server, and successive blocking windows are chained until the caller's timeout deadline.

On timeout the current non-terminal run is returned (never raises for a slow run); a terminal run is returned as soon as it is observed.

Parameters:

  • run_id (String) —

    the run to wait on

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait (default 300)

Returns:

  • (Hash) —

    the run record (terminal when it finished in time)



585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
 
# File 'lib/ghostcrawl/client.rb', line 585

def wait_for_completion(run_id, timeout: 300)
  if run_id.nil? || run_id.to_s.empty?
    raise ArgumentError, "run_id is required"
  end

  deadline = monotonic + timeout
  loop do
    remaining = deadline - monotonic
    # Deadline reached — one final non-blocking read of the current state.
    return get(run_id) if remaining <= 0

    window = [remaining, WAIT_WINDOW_S].min
    run = fetch_waiting(run_id, window)
    return run if terminal?(run)
    # The server just blocked for ~window seconds; loop straight into the
    # next blocking window. No client-side delay.
  end
end