ghostcrawl — Ruby SDK

The official Ruby client for the GhostCrawl API. Collect web data at scale — scrape, crawl, search, and extract structured data.

Install

gem install ghostcrawl

Or in your Gemfile:

gem "ghostcrawl", "~> 2.1"

Requires Ruby 3.0+. No runtime dependencies — uses stdlib net/http only.

Quickstart

require "ghostcrawl"

# Token from constructor or GHOSTCRAWL_API_KEY env var
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# Scrape a URL
result = client.scrape(url: "https://example.com", format: "markdown")
puts result["content"]

# Start a crawl run
run = client.crawl_runs.start(url: "https://example.com", max_depth: 2, max_pages: 50)
puts "run_id: #{run["run_id"]}"

# Web search
results = client.search(query: "latest AI research", engine: "google", limit: 10)
puts results["results"].length

Authentication

# Option 1: pass token directly
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

# Option 2: set environment variable (recommended for production)
# export GHOSTCRAWL_API_KEY=gck_live_YOUR_KEY
client = Ghostcrawl::Client.new

Every request sends Authorization: Bearer <token>. This is the only auth scheme the API accepts.

Extract structured data

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

data = client.extract(
  url: "https://example.com/product",
  schema: {
    type: "object",
    properties: {
      name:        { type: "string" },
      price:       { type: "number" },
      description: { type: "string" }
    }
  }
)
puts "#{data["name"]} — $#{data["price"]}"

Error handling

require "ghostcrawl"

client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")

begin
  result = client.scrape(url: "https://example.com")
rescue Ghostcrawl::AuthenticationError
  puts "Invalid API key — check your token"
rescue Ghostcrawl::PaymentRequiredError
  puts "Usage limit reached — upgrade your plan"
rescue Ghostcrawl::RateLimitError
  puts "Rate limit reached — retry after a short delay"
rescue Ghostcrawl::APIError => e
  puts "Server error: #{e.status_code}"
rescue Ghostcrawl::GhostcrawlError => e
  puts "API error: #{e.message} (#{e.status_code})"
end

Self-hosted

client = Ghostcrawl::Client.new(
  token: "gck_live_YOUR_KEY",
  base_url: "http://localhost:8080"
)

All resources

Resource	Method / accessor	Key operations
Scraping	client.scrape(url:)	Render and return page content
Web search	client.search(query:)	Search Google, Bing, DuckDuckGo
Data extraction	client.extract(url:, schema:)	Structured JSON from any page
Deep crawl	client.crawl(url:)	Crawl a site depth-first
URL map	client.map(url:)	Discover all reachable URLs
Account	client.me	Get account info and usage
Crawl runs	client.crawl_runs	start, list, get_run, cancel
Sessions	client.sessions	create, list, extend, release
Profiles	client.profiles	list, get_profile, create, update, delete
Webhooks	client.webhooks	list, get_webhook, create, delete, rotate_secret
Schedules	client.schedules	list, get_schedule, create, delete
Datasets	client.datasets	list, get_dataset, create, delete, rows, append
Recordings	client.recordings	list, get_recording, delete
Key-Value Store	client.kv	get, set, delete

License

Proprietary — GhostCrawl Software License. See LICENSE.