Class: Ghostcrawl::Client

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

GhostCrawl idiomatic API client.

Delegates all HTTP transport, URL routing, serialization, and auth to the Kiota-generated canonical core (_generated/). This facade is the shipped API.

Examples:

require "ghostcrawl"
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")
result = client.scrape(url: "https://example.com")

Constant Summary collapse

DEFAULT_TIMEOUT =

Default read timeout (seconds). Browser-rendered scrapes/crawls are slow, and the underlying net/http default (60s) is too short for them.

300

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(token: nil, base_url: nil, timeout: nil) ⇒ Client

Returns a new instance of Client.

Parameters:

  • token (String, nil) (defaults to: nil) —

    API key. Falls back to GHOSTCRAWL_API_KEY env var.

  • base_url (String, nil) (defaults to: nil) —

    Override API base URL. Falls back to GHOSTCRAWL_BASE_URL env var.

  • timeout (Integer, nil) (defaults to: nil) —

    Per-request read timeout in seconds. Falls back to GHOSTCRAWL_TIMEOUT env var, then DEFAULT_TIMEOUT.



916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
 
# File 'lib/ghostcrawl/client.rb', line 916

def initialize(token: nil, base_url: nil, timeout: nil)
  resolved_token = token || ENV.fetch("GHOSTCRAWL_API_KEY", nil)
  if resolved_token.nil? || resolved_token.empty?
    raise ArgumentError,
          "token is required — pass token: or set GHOSTCRAWL_API_KEY. " \
          "Get your key at https://ghostcrawl.io"
  end

  resolved_base = (base_url ||
                   ENV.fetch("GHOSTCRAWL_BASE_URL", nil) ||
                   DEFAULT_BASE_URL).gsub(%r{/+$}, "")

  resolved_timeout = (timeout ||
                      ENV.fetch("GHOSTCRAWL_TIMEOUT", nil) ||
                      DEFAULT_TIMEOUT).to_i

  # Build the Kiota core via BaseBearerTokenAuthenticationProvider + FaradayRequestAdapter.
  # All HTTP, auth, serialization, and URL routing delegate to the generated core.
  auth_provider = MicrosoftKiotaAbstractions::BaseBearerTokenAuthenticationProvider.new(
    StaticTokenProvider.new(resolved_token)
  )
  adapter = MicrosoftKiotaFaraday::FaradayRequestAdapter.new(auth_provider)
  adapter.set_base_url(resolved_base)

  # The default Faraday connection sets no timeout, so it inherits net/http's
  # 60s read timeout — too short for browser-rendered work. Raise it.
  if adapter.client.respond_to?(:options) && resolved_timeout.positive?
    adapter.client.options.timeout = resolved_timeout
    adapter.client.options.open_timeout = 30
  end

  @core    = Ghostcrawl::GhostcrawlClient.new(adapter)
  @v1      = @core.v1
  # Kept so #me can re-issue the GET /v1/me request with the raw-JSON
  # (Binary) response factory instead of the typed MeResponse model.
  @adapter = adapter

  # CrawlRunsClient needs the raw adapter for the event-driven long-poll wait
  # (the generated item builder can't attach ?wait/&timeout_s query params).
  @crawl_runs  = CrawlRunsClient.new(@v1, @adapter)
  @sessions    = SessionsClient.new(@v1)
  # Sub-clients with void-DELETE endpoints (204 No Content) also need the raw
  # adapter so ResponseHelper.void_request! can work around the broken generated
  # delete builders. See ResponseHelper.void_request! for the two Kiota defects.
  @profiles    = ProfilesClient.new(@v1, @adapter)
  @webhooks    = WebhooksClient.new(@v1, @adapter)
  @schedules   = SchedulesClient.new(@v1, @adapter)
  @datasets    = DatasetsClient.new(@v1)
  @recordings  = RecordingsClient.new(@v1, @adapter)
  @kv          = KVClient.new(@v1)
end

Instance Attribute Details

#crawl_runs ⇒ CrawlRunsClient (readonly)

Returns:



969
970
971
 
# File 'lib/ghostcrawl/client.rb', line 969

def crawl_runs
  @crawl_runs
end

#datasets ⇒ DatasetsClient (readonly)

Returns:



979
980
981
 
# File 'lib/ghostcrawl/client.rb', line 979

def datasets
  @datasets
end

#kv ⇒ KVClient (readonly)

Returns:



983
984
985
 
# File 'lib/ghostcrawl/client.rb', line 983

def kv
  @kv
end

#profiles ⇒ ProfilesClient (readonly)

Returns:



973
974
975
 
# File 'lib/ghostcrawl/client.rb', line 973

def profiles
  @profiles
end

#recordings ⇒ RecordingsClient (readonly)

Returns:



981
982
983
 
# File 'lib/ghostcrawl/client.rb', line 981

def recordings
  @recordings
end

#schedules ⇒ SchedulesClient (readonly)

Returns:



977
978
979
 
# File 'lib/ghostcrawl/client.rb', line 977

def schedules
  @schedules
end

#sessions ⇒ SessionsClient (readonly)

Returns:



971
972
973
 
# File 'lib/ghostcrawl/client.rb', line 971

def sessions
  @sessions
end

#webhooks ⇒ WebhooksClient (readonly)

Returns:



975
976
977
 
# File 'lib/ghostcrawl/client.rb', line 975

def webhooks
  @webhooks
end

Instance Method Details

#crawl(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300, raise_on_result_error: true, **opts) ⇒ Hash

Start a deep crawl from a seed URL. Delegates to POST /v1/crawl/deep via the generated CrawlDeepRequestBuilder.

Pass wait: true to block until the run is terminal. The deep-crawl start returns a run_id, then the wait is handled by the same event-driven server-blocking long-poll as Ghostcrawl::CrawlRunsClient#wait_for_completion — no client-side sleep loop.

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • wait (Boolean) (defaults to: false) —

    block until the run is terminal (default false)

  • timeout (Integer) (defaults to: 300) —

    total seconds to wait when wait: true (default 300)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true). Ignored on the wait: true path, which returns the terminal run.

Returns:

  • (Hash) —

    crawl run record (terminal when wait: true and it finished in time)



1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
 
# File 'lib/ghostcrawl/client.rb', line 1067

def crawl(url:, max_depth: 2, max_pages: 100, wait: false, timeout: 300,
          raise_on_result_error: true, **opts)
  data = { "seed_urls" => [url], "max_depth" => max_depth,
           "max_urls" => max_pages }.merge(opts.transform_keys(&:to_s))
  hash = ResponseHelper.to_hash(@v1.crawl.deep.post(AdditionalDataBody.new(data)))

  unless wait
    return raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
  end

  run_id = hash["run_id"]
  return hash if run_id.nil? || run_id.to_s.empty?
  @crawl_runs.wait_for_completion(run_id, timeout: timeout)
end

#extract(url:, schema:, raise_on_result_error: true, **opts) ⇒ Hash

Extract structured data from a URL using a JSON Schema. Delegates to POST /v1/extract via the generated ExtractRequestBuilder.

Parameters:

  • url (String) —

    target URL

  • schema (Hash) —

    JSON Schema describing the shape to extract

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    extracted data



1044
1045
1046
1047
1048
 
# File 'lib/ghostcrawl/client.rb', line 1044

def extract(url:, schema:, raise_on_result_error: true, **opts)
  data = { "url" => url, "schema" => schema }.merge(opts.transform_keys(&:to_s))
  hash = ResponseHelper.to_hash(@v1.extract.post(AdditionalDataBody.new(data)))
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end

#map(url:, **opts) ⇒ Hash

Map all URLs reachable from a seed URL. Delegates to POST /v1/map via the generated MapRequestBuilder.

Parameters:

  • url (String) —

    seed URL

Returns:

  • (Hash) —

    response with urls list



1086
1087
1088
1089
 
# File 'lib/ghostcrawl/client.rb', line 1086

def map(url:, **opts)
  data = { "url" => url }.merge(opts.transform_keys(&:to_s))
  ResponseHelper.to_hash(@v1.map.post(AdditionalDataBody.new(data)))
end

#me ⇒ Hash

Get the current account's profile. Delegates to GET /v1/me via the generated MeRequestBuilder.

The generated @v1.me.get deserializes into the typed MeResponse model, whose created_at composed-type member fails under the pinned Kiota JSON parser ("Error during deserialization"). To avoid that, we re-issue the SAME request (reusing the builder's URL/auth/header wiring via to_get_request_information) with the Binary response factory — the exact raw-JSON path the scrape builder already uses (see KiotaParseNodeFix).

Returns:

  • (Hash) —

    account info with user_id, email, created_at, primary_team



1101
1102
1103
1104
 
# File 'lib/ghostcrawl/client.rb', line 1101

def me
  request_info = @v1.me.to_get_request_information(nil)
  ResponseHelper.to_hash(@adapter.send_async(request_info, Ghostcrawl::V1::Binary, {}))
end

#scrape(url:, format: "markdown", engine: "auto", javascript: true, extract_schema: nil, raise_on_result_error: true, **opts) ⇒ Hash

Scrape a single URL and return the rendered content. Delegates to POST /v1/scrape via the generated ScrapeRequestBuilder.

Parameters:

  • url (String) —

    target URL

  • format (String) (defaults to: "markdown") —

    output format: "markdown" (default), "html", "text"

  • engine (String) (defaults to: "auto") —

    browser engine: "auto" (default), "chrome", "firefox", "webkit"

  • javascript (Boolean) (defaults to: true) —

    enable JavaScript rendering (default true)

  • extract_schema (Hash, nil) (defaults to: nil) —

    JSON Schema for structured extraction

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    response with content, markdown, status, and other fields



999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
 
# File 'lib/ghostcrawl/client.rb', line 999

def scrape(url:, format: "markdown", engine: "auto", javascript: true, extract_schema: nil,
           raise_on_result_error: true, **opts)
  # Use AdditionalDataBody to send only the fields we specify — the generated
  # ScrapeRequest model would serialize typed defaults (nulls + empty enums) that
  # cause 422 validation errors on the server.
  data = { "url" => url, "format" => format, "engine" => engine,
           "javascript_enabled" => javascript }.merge(opts.transform_keys(&:to_s))
  data["extract_schema"] = extract_schema unless extract_schema.nil?
  hash = ResponseHelper.to_hash(@v1.scrape.post(AdditionalDataBody.new(data)))
  normalize_scrape_content(hash)
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end

#search(query:, engine: "google", limit: 10, provider_key: nil, **opts) ⇒ Hash

Search the web and return results. Delegates to POST /v1/search via the generated SearchRequestBuilder.

/v1/search requires your own search-backend API key (BYO; GhostCrawl charges no markup). Pass it as provider_key — it is sent as the X-Provider-Authorization: Bearer <provider_key> header the backend requires. Without it the API replies 401 search_backend_key_missing.

Parameters:

  • query (String) —

    search query

  • engine (String) (defaults to: "google") —

    search engine: "google" (default), "bing", "duckduckgo"

  • limit (Integer) (defaults to: 10) —

    maximum results (default 10)

  • provider_key (String, nil) (defaults to: nil) —

    BYO search-backend key (sent as X-Provider-Authorization)

Returns:

  • (Hash) —

    response with results list



1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
 
# File 'lib/ghostcrawl/client.rb', line 1024

def search(query:, engine: "google", limit: 10, provider_key: nil, **opts)
  data = { "query" => query, "engine" => engine,
           "limit" => limit }.merge(opts.transform_keys(&:to_s))
  config = nil
  unless provider_key.nil?
    config = MicrosoftKiotaAbstractions::RequestConfiguration.new
    headers = MicrosoftKiotaAbstractions::RequestHeaders.new
    headers.add("X-Provider-Authorization", "Bearer #{provider_key}")
    config.headers = headers
  end
  ResponseHelper.to_hash(@v1.search.post(AdditionalDataBody.new(data), config))
end