Class: GhostCrawl::CrawlRunsClient

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

Manage crawl runs — /v1/crawl-runs.

Constant Summary collapse

TERMINAL_STATUSES =

Terminal run states — a crawl run in any of these will never change again, so a wait can stop. Both spellings of "cancelled" are accepted.

%w[completed failed cancelled canceled].freeze
WAIT_WINDOW_S =

Per-request server-block window (seconds) for the long-poll wait. Each GET ...?wait=true&timeout_s=WAIT_WINDOW_S makes the SERVER block for up to this long, so the client never sleeps between checks. Kept well under the client read timeout (GhostCrawl::Client::DEFAULT_TIMEOUT) so a single blocking request can never trip the transport timeout.

30

Instance Method Summary collapse

Constructor Details

#initialize(v1, adapter = nil) ⇒ CrawlRunsClient

Returns a new instance of CrawlRunsClient.



532
533
534
535
 
# File 'lib/ghostcrawl/client.rb', line 532

def initialize(v1, adapter = nil)
  @v1 = v1
  @adapter = adapter
end

Instance Method Details

#cancel(run_id) ⇒ Object

Cancel a running crawl run. Delegates to POST /v1/crawl-runs/run_id/cancel via the generated builder.



632
633
634
 
# File 'lib/ghostcrawl/client.rb', line 632

def cancel(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).cancel.post)
end

#get(run_id) ⇒ Object

Get a single crawl run by ID. Delegates to GET /v1/crawl-runs/run_id via the generated builder.



626
627
628
 
# File 'lib/ghostcrawl/client.rb', line 626

def get(run_id)
  ResponseHelper.to_hash(@v1.crawl_runs.by_run_id(run_id).get)
end

#list ⇒ Object

List crawl runs. Delegates to GET /v1/crawl-runs via the generated CrawlRunsRequestBuilder.



620
621
622
 
# File 'lib/ghostcrawl/client.rb', line 620

def list
  ResponseHelper.to_hash(@v1.crawl_runs.get)
end

#start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300, raise_on_result_error: true, **opts) ⇒ Hash

Start a new crawl run from a seed URL. Delegates to POST /v1/crawl-runs via the generated CrawlRunsRequestBuilder. The endpoint is a tagged union: a start request requires action: "start" and a seed_urls array (not a bare url).

Pass wait: true to block until the run reaches a terminal state (completed | failed | cancelled) or timeout seconds elapse. This sends wait_until: "completed" so the SERVER blocks — no client-side poll loop, no sleep. When the run is still running at timeout the current (non-terminal) record is returned; call #wait_for_completion again to keep waiting.

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • wait (Boolean) (defaults to: false) —

    block until the run is terminal (default false)

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait when wait: true (default 300)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true). Ignored on the wait: true path, which always returns the terminal run so the caller can inspect a failed status.

Returns:

  • (Hash) —

    crawl run record with run_id and status (results present when it completed while waiting)



560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
 
# File 'lib/ghostcrawl/client.rb', line 560

def start(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300,
          raise_on_result_error: true, **opts)
  data = { "action" => "start", "seed_urls" => [url],
           "max_depth" => max_depth, "max_pages" => max_pages }
         .merge(opts.transform_keys(&:to_s))

  unless wait
    hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
    return raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
  end

  # Start-and-wait: ask the server to block until terminal. Bound the first
  # server block to a safe window, then long-poll the rest via GET so a low
  # per-request read timeout can never abort the whole wait.
  window = [timeout, WAIT_WINDOW_S].min
  data["wait_until"] = "completed"
  data["timeout_s"]  = window
  hash = ResponseHelper.to_hash(@v1.crawl_runs.post(AdditionalDataBody.new(data)))
  return hash if terminal?(hash)

  run_id = hash["run_id"]
  remaining = timeout - window
  return hash if run_id.nil? || run_id.to_s.empty? || remaining <= 0
  wait_for_completion(run_id, timeout: remaining)
end

#wait_for_completion(run_id, timeout: 300) ⇒ Hash

Block until an existing crawl run reaches a terminal state, or timeout seconds elapse. Event-driven: each iteration issues a SERVER-blocking GET /v1/crawl-runs/{run_id}?wait=true&timeout_s=N that returns the moment the run goes terminal (or after its own window). There is no client sleep — the wait cost lives on the server, and successive blocking windows are chained until the caller's timeout deadline.

On timeout the current non-terminal run is returned (never raises for a slow run); a terminal run is returned as soon as it is observed.

Parameters:

  • run_id (String) —

    the run to wait on

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait (default 300)

Returns:

  • (Hash) —

    the run record (terminal when it finished in time)



599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
 
# File 'lib/ghostcrawl/client.rb', line 599

def wait_for_completion(run_id, timeout: 300)
  if run_id.nil? || run_id.to_s.empty?
    raise ArgumentError, "run_id is required"
  end

  deadline = monotonic + timeout
  loop do
    remaining = deadline - monotonic
    # Deadline reached — one final non-blocking read of the current state.
    return get(run_id) if remaining <= 0

    window = [remaining, WAIT_WINDOW_S].min
    run = fetch_waiting(run_id, window)
    return run if terminal?(run)
    # The server just blocked for ~window seconds; loop straight into the
    # next blocking window. No client-side delay.
  end
end