Class: Ghostcrawl::Client

Inherits:
Object
  • Object
show all
Defined in:
lib/ghostcrawl/client.rb

Overview

GhostCrawl idiomatic API client.

Delegates all HTTP transport, URL routing, serialization, and auth to the Kiota-generated canonical core (_generated/). This facade is the shipped API.

Examples:

require "ghostcrawl"
client = Ghostcrawl::Client.new(token: "gck_live_YOUR_KEY")
result = client.scrape(url: "https://example.com")

Constant Summary collapse

DEFAULT_TIMEOUT =

Default read timeout (seconds). Browser-rendered scrapes/crawls are slow, and the underlying net/http default (60s) is too short for them.

300

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(token: nil, base_url: nil, timeout: nil) ⇒ Client

Returns a new instance of Client.

Parameters:

  • token (String, nil) (defaults to: nil) —

    API key. Falls back to GHOSTCRAWL_API_KEY env var.

  • base_url (String, nil) (defaults to: nil) —

    Override API base URL. Falls back to GHOSTCRAWL_BASE_URL env var.

  • timeout (Integer, nil) (defaults to: nil) —

    Per-request read timeout in seconds. Falls back to GHOSTCRAWL_TIMEOUT env var, then DEFAULT_TIMEOUT.



814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
 
# File 'lib/ghostcrawl/client.rb', line 814

def initialize(token: nil, base_url: nil, timeout: nil)
  resolved_token = token || ENV.fetch("GHOSTCRAWL_API_KEY", nil)
  if resolved_token.nil? || resolved_token.empty?
    raise ArgumentError,
          "token is required — pass token: or set GHOSTCRAWL_API_KEY. " \
          "Get your key at https://ghostcrawl.io"
  end

  resolved_base = (base_url ||
                   ENV.fetch("GHOSTCRAWL_BASE_URL", nil) ||
                   DEFAULT_BASE_URL).gsub(%r{/+$}, "")

  resolved_timeout = (timeout ||
                      ENV.fetch("GHOSTCRAWL_TIMEOUT", nil) ||
                      DEFAULT_TIMEOUT).to_i

  # Build the Kiota core via BaseBearerTokenAuthenticationProvider + FaradayRequestAdapter.
  # All HTTP, auth, serialization, and URL routing delegate to the generated core.
  auth_provider = MicrosoftKiotaAbstractions::BaseBearerTokenAuthenticationProvider.new(
    StaticTokenProvider.new(resolved_token)
  )
  adapter = MicrosoftKiotaFaraday::FaradayRequestAdapter.new(auth_provider)
  adapter.set_base_url(resolved_base)

  # The default Faraday connection sets no timeout, so it inherits net/http's
  # 60s read timeout — too short for browser-rendered work. Raise it.
  if adapter.client.respond_to?(:options) && resolved_timeout.positive?
    adapter.client.options.timeout = resolved_timeout
    adapter.client.options.open_timeout = 30
  end

  @core    = Ghostcrawl::GhostcrawlClient.new(adapter)
  @v1      = @core.v1
  # Kept so #me can re-issue the GET /v1/me request with the raw-JSON
  # (Binary) response factory instead of the typed MeResponse model.
  @adapter = adapter

  @crawl_runs  = CrawlRunsClient.new(@v1)
  @sessions    = SessionsClient.new(@v1)
  # Sub-clients with void-DELETE endpoints (204 No Content) also need the raw
  # adapter so ResponseHelper.void_request! can work around the broken generated
  # delete builders. See ResponseHelper.void_request! for the two Kiota defects.
  @profiles    = ProfilesClient.new(@v1, @adapter)
  @webhooks    = WebhooksClient.new(@v1, @adapter)
  @schedules   = SchedulesClient.new(@v1, @adapter)
  @datasets    = DatasetsClient.new(@v1)
  @recordings  = RecordingsClient.new(@v1, @adapter)
  @kv          = KVClient.new(@v1)
end

Instance Attribute Details

#crawl_runs ⇒ CrawlRunsClient (readonly)

Returns:



865
866
867
 
# File 'lib/ghostcrawl/client.rb', line 865

def crawl_runs
  @crawl_runs
end

#datasets ⇒ DatasetsClient (readonly)

Returns:



875
876
877
 
# File 'lib/ghostcrawl/client.rb', line 875

def datasets
  @datasets
end

#kv ⇒ KVClient (readonly)

Returns:



879
880
881
 
# File 'lib/ghostcrawl/client.rb', line 879

def kv
  @kv
end

#profiles ⇒ ProfilesClient (readonly)

Returns:



869
870
871
 
# File 'lib/ghostcrawl/client.rb', line 869

def profiles
  @profiles
end

#recordings ⇒ RecordingsClient (readonly)

Returns:



877
878
879
 
# File 'lib/ghostcrawl/client.rb', line 877

def recordings
  @recordings
end

#schedules ⇒ SchedulesClient (readonly)

Returns:



873
874
875
 
# File 'lib/ghostcrawl/client.rb', line 873

def schedules
  @schedules
end

#sessions ⇒ SessionsClient (readonly)

Returns:



867
868
869
 
# File 'lib/ghostcrawl/client.rb', line 867

def sessions
  @sessions
end

#webhooks ⇒ WebhooksClient (readonly)

Returns:



871
872
873
 
# File 'lib/ghostcrawl/client.rb', line 871

def webhooks
  @webhooks
end

Instance Method Details

#crawl(url:, max_depth: 2, max_pages: 100, raise_on_result_error: true, **opts) ⇒ Hash

Start a deep crawl from a seed URL. Delegates to POST /v1/crawl/deep via the generated CrawlDeepRequestBuilder.

Parameters:

  • url (String) —

    seed URL

  • max_depth (Integer) (defaults to: 2) —

    maximum crawl depth (default 2)

  • max_pages (Integer) (defaults to: 100) —

    maximum pages (default 100)

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    crawl run record



954
955
956
957
958
959
 
# File 'lib/ghostcrawl/client.rb', line 954

def crawl(url:, max_depth: 2, max_pages: 100, raise_on_result_error: true, **opts)
  data = { "seed_urls" => [url], "max_depth" => max_depth,
           "max_urls" => max_pages }.merge(opts.transform_keys(&:to_s))
  hash = ResponseHelper.to_hash(@v1.crawl.deep.post(AdditionalDataBody.new(data)))
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end

#extract(url:, schema:, raise_on_result_error: true, **opts) ⇒ Hash

Extract structured data from a URL using a JSON Schema. Delegates to POST /v1/extract via the generated ExtractRequestBuilder.

Parameters:

  • url (String) —

    target URL

  • schema (Hash) —

    JSON Schema describing the shape to extract

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    extracted data



940
941
942
943
944
 
# File 'lib/ghostcrawl/client.rb', line 940

def extract(url:, schema:, raise_on_result_error: true, **opts)
  data = { "url" => url, "schema" => schema }.merge(opts.transform_keys(&:to_s))
  hash = ResponseHelper.to_hash(@v1.extract.post(AdditionalDataBody.new(data)))
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end

#map(url:, **opts) ⇒ Hash

Map all URLs reachable from a seed URL. Delegates to POST /v1/map via the generated MapRequestBuilder.

Parameters:

  • url (String) —

    seed URL

Returns:

  • (Hash) —

    response with urls list



965
966
967
968
 
# File 'lib/ghostcrawl/client.rb', line 965

def map(url:, **opts)
  data = { "url" => url }.merge(opts.transform_keys(&:to_s))
  ResponseHelper.to_hash(@v1.map.post(AdditionalDataBody.new(data)))
end

#me ⇒ Hash

Get the current account's profile. Delegates to GET /v1/me via the generated MeRequestBuilder.

The generated @v1.me.get deserializes into the typed MeResponse model, whose created_at composed-type member fails under the pinned Kiota JSON parser ("Error during deserialization"). To avoid that, we re-issue the SAME request (reusing the builder's URL/auth/header wiring via to_get_request_information) with the Binary response factory — the exact raw-JSON path the scrape builder already uses (see KiotaParseNodeFix).

Returns:

  • (Hash) —

    account info with user_id, email, created_at, primary_team



980
981
982
983
 
# File 'lib/ghostcrawl/client.rb', line 980

def me
  request_info = @v1.me.to_get_request_information(nil)
  ResponseHelper.to_hash(@adapter.send_async(request_info, Ghostcrawl::V1::Binary, {}))
end

#scrape(url:, format: "markdown", engine: "auto", javascript: true, extract_schema: nil, raise_on_result_error: true, **opts) ⇒ Hash

Scrape a single URL and return the rendered content. Delegates to POST /v1/scrape via the generated ScrapeRequestBuilder.

Parameters:

  • url (String) —

    target URL

  • format (String) (defaults to: "markdown") —

    output format: "markdown" (default), "html", "text"

  • engine (String) (defaults to: "auto") —

    browser engine: "auto" (default), "chrome", "firefox", "webkit"

  • javascript (Boolean) (defaults to: true) —

    enable JavaScript rendering (default true)

  • extract_schema (Hash, nil) (defaults to: nil) —

    JSON Schema for structured extraction

  • raise_on_result_error (Boolean) (defaults to: true) —

    raise ScrapeError on a target-side (HTTP-200) failure instead of returning the raw hash (default true)

Returns:

  • (Hash) —

    response with content, markdown, status, and other fields



895
896
897
898
899
900
901
902
903
904
905
906
 
# File 'lib/ghostcrawl/client.rb', line 895

def scrape(url:, format: "markdown", engine: "auto", javascript: true, extract_schema: nil,
           raise_on_result_error: true, **opts)
  # Use AdditionalDataBody to send only the fields we specify — the generated
  # ScrapeRequest model would serialize typed defaults (nulls + empty enums) that
  # cause 422 validation errors on the server.
  data = { "url" => url, "format" => format, "engine" => engine,
           "javascript_enabled" => javascript }.merge(opts.transform_keys(&:to_s))
  data["extract_schema"] = extract_schema unless extract_schema.nil?
  hash = ResponseHelper.to_hash(@v1.scrape.post(AdditionalDataBody.new(data)))
  normalize_scrape_content(hash)
  raise_on_result_error ? ResponseHelper.raise_on_result_error!(hash) : hash
end

#search(query:, engine: "google", limit: 10, provider_key: nil, **opts) ⇒ Hash

Search the web and return results. Delegates to POST /v1/search via the generated SearchRequestBuilder.

/v1/search requires your own search-backend API key (BYO; GhostCrawl charges no markup). Pass it as provider_key — it is sent as the X-Provider-Authorization: Bearer <provider_key> header the backend requires. Without it the API replies 401 search_backend_key_missing.

Parameters:

  • query (String) —

    search query

  • engine (String) (defaults to: "google") —

    search engine: "google" (default), "bing", "duckduckgo"

  • limit (Integer) (defaults to: 10) —

    maximum results (default 10)

  • provider_key (String, nil) (defaults to: nil) —

    BYO search-backend key (sent as X-Provider-Authorization)

Returns:

  • (Hash) —

    response with results list



920
921
922
923
924
925
926
927
928
929
930
931
 
# File 'lib/ghostcrawl/client.rb', line 920

def search(query:, engine: "google", limit: 10, provider_key: nil, **opts)
  data = { "query" => query, "engine" => engine,
           "limit" => limit }.merge(opts.transform_keys(&:to_s))
  config = nil
  unless provider_key.nil?
    config = MicrosoftKiotaAbstractions::RequestConfiguration.new
    headers = MicrosoftKiotaAbstractions::RequestHeaders.new
    headers.add("X-Provider-Authorization", "Bearer #{provider_key}")
    config.headers = headers
  end
  ResponseHelper.to_hash(@v1.search.post(AdditionalDataBody.new(data), config))
end