ghostcrawl — Ruby SDK

The official Ruby client for the GhostCrawl API. Collect web data at scale — scrape, crawl, search, and extract structured data.

Install

gem install ghostcrawl

Or in your Gemfile:

gem "ghostcrawl", "~> 2.1"

Requires Ruby 3.0+. No runtime dependencies — uses stdlib net/http only.

Quickstart

require "ghostcrawl"

# Token from constructor or GHOSTCRAWL_API_KEY env var
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# Scrape a URL
result = client.scrape(url: "https://example.com", format: "markdown")
puts result["content"]

# Start a crawl run and wait for it to finish — the server blocks until the run
# is terminal (no client-side poll-with-sleep loop).
run = client.crawl_runs.start(url: "https://example.com", max_depth: 2, max_pages: 50,
                              wait: true, timeout: 300)
puts "#{run["run_id"]} → #{run["status"]}"

# Web search
results = client.search(query: "latest AI research", engine: "google", limit: 10)
puts results["results"].length

Authentication

# Option 1: pass token directly
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# Option 2: set environment variable (recommended for production)
# export GHOSTCRAWL_API_KEY=gck_live_YOUR_KEY
client = Ghostcrawl::Client.new

Every request sends Authorization: Bearer <token>. This is the only auth scheme the API accepts.

Extract structured data

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

data = client.extract(
  url: "https://example.com/product",
  schema: {
    type: "object",
    properties: {
      name:        { type: "string" },
      price:       { type: "number" },
      description: { type: "string" }
    }
  }
)
puts "#{data["name"]} — $#{data["price"]}"

Browser utilities — content, screenshot, PDF

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# Rendered content as a JSON envelope: {url, status, format, status_code, content, bytes}
page = client.content(url: "https://example.com")
puts "#{page["status_code"]} — #{page["bytes"]} bytes"

# Screenshot — returns raw PNG bytes
png = client.screenshot(url: "https://example.com", full_page: true)
File.binwrite("page.png", png)

# PDF — returns raw application/pdf bytes (Chrome-only; a Firefox/WebKit identity
# is rejected with a 400 pdf_engine_unsupported)
pdf = client.pdf(url: "https://example.com", paper_format: "a4")
File.binwrite("page.pdf", pdf)

Agent (BYO model, account-gated)

The agent lane runs a natural-language browser task. It is bring-your-own-model (supply your own provider_config in the request body) and account-gated: the API returns 404 not_found unless the capability is enabled for your account. agent does not raise on that 404 — it returns the problem+json body as a Hash, so branch on result.key?("detail").

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# provider_config is BYO — reference your provider key by env-var name, never a literal.
result = client.agent(
  task: {
    "instruction" => "click the 'Books to Scrape' link",
    "start_url"   => "https://books.toscrape.com"
  },
  provider_config: {
    "provider" => "openai",
    "api_key"  => "OPENAI_API_KEY",
    "model"    => "gpt-4o"
  }
)
if result.key?("detail")
  puts "agent lane not enabled for this account (BYO/gated)"
else
  puts result
end

Crawl runs — wait for completion

Crawl runs are asynchronous. Instead of hand-writing a poll-with-sleep loop, use the event-driven wait API: the server blocks until the run reaches a terminal state (completed / failed / cancelled) and returns the finished run.

# Start and wait in one call — sends wait_until: "completed"
run = client.crawl_runs.start(
  url: "https://example.com",
  max_depth: 2,
  max_pages: 50,
  wait: true,      # block until terminal (no client sleep loop)
  timeout: 300     # give up after 300s, returning the still-running run
)
puts run["status"]        # "completed" | "failed" | "cancelled"

# Or wait on a run you already started (e.g. from a webhook or another process):
run = client.crawl_runs.wait_for_completion(run_id, timeout: 300)

# Deep crawl supports the same wait: keyword:
run = client.crawl(url: "https://example.com", wait: true, timeout: 300)

Under the hood wait_for_completion chains server-blocking GET /v1/crawl-runs/{run_id}?wait=true&timeout_s=N requests until the run is terminal or your timeout deadline — there is no sleep between checks. On timeout the current non-terminal run is returned, so you can simply call wait_for_completion again to keep waiting.

Prefer webhooks for fire-and-forget crawls. Register an endpoint with client.webhooks.create(url:, event_types: ["crawl.completed"]) and let GhostCrawl POST the result to you instead of holding a connection open.

Error handling

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

begin
  result = client.scrape(url: "https://example.com")
rescue Ghostcrawl::AuthenticationError
  puts "Invalid API key — check your token"
rescue Ghostcrawl::PaymentRequiredError
  puts "Usage limit reached — upgrade your plan"
rescue Ghostcrawl::RateLimitError
  puts "Rate limit reached — retry after a short delay"
rescue Ghostcrawl::APIError => e
  puts "Server error: #{e.status_code}"
rescue Ghostcrawl::GhostcrawlError => e
  puts "API error: #{e.message} (#{e.status_code})"
end

Self-hosted

client = Ghostcrawl::Client.new(
  token: "gck_live_YOUR_KEY",
  base_url: "http://localhost:8080"
)

All resources

Resource	Method / accessor	Key operations
Scraping	client.scrape(url:)	Render and return page content
Web search	client.search(query:)	Search Google, Bing, DuckDuckGo
Data extraction	client.extract(url:, schema:)	Structured JSON from any page
Deep crawl	client.crawl(url:)	Crawl a site depth-first
URL map	client.map(url:)	Discover all reachable URLs
Content	client.content(url:)	Rendered content JSON envelope
Screenshot	client.screenshot(url:)	Capture a URL to PNG bytes
PDF	client.pdf(url:)	Render a URL to PDF bytes (Chrome-only)
Agent (BYO)	client.agent(task:)	NL browser task — account-gated, BYO model
Account	client.me	Get account info and usage
Crawl runs	client.crawl_runs	start (with wait:), wait_for_completion, list, get, cancel
Sessions	client.sessions	create, list, extend, release
Profiles	client.profiles	list, get_profile, create, update, delete
Webhooks	client.webhooks	list, get_webhook, create, delete, rotate_secret
Schedules	client.schedules	list, get_schedule, create, delete
Datasets	client.datasets	list, get_dataset, create, delete, rows, append
Recordings	client.recordings	list, get_recording, delete
Key-Value Store	client.kv	get, set, delete

License

Proprietary — GhostCrawl Software License. See LICENSE.