Class: ContextDev::Resources::Batch

Inherits:
Object
  • Object
show all
Defined in:
lib/context_dev/resources/batch.rb,
sig/context_dev/resources/batch.rbs

Overview

Scrape many pages or crawl a site asynchronously.

Instance Method Summary collapse

Constructor Details

#initialize(client:) ⇒ Batch

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of Batch.

Parameters:



173
174
175
 
# File 'lib/context_dev/resources/batch.rb', line 173

def initialize(client:)
  @client = client
end

Instance Method Details

#cancel(batch_id, request_options: {}) ⇒ ContextDev::Models::BatchCancelResponse

Stop a batch from starting new pages. In-progress pages finish, and unused credits are refunded.

Parameters:

  • batch_id (String) —

    ID of the batch to retrieve or cancel.

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



97
98
99
100
101
102
103
104
 
# File 'lib/context_dev/resources/batch.rb', line 97

def cancel(batch_id, params = {})
  @client.request(
    method: :post,
    path: ["batch/%1$s/cancel", batch_id],
    model: ContextDev::Models::BatchCancelResponse,
    options: params[:request_options]
  )
end

#delete(batch_id, request_options: {}) ⇒ ContextDev::Models::BatchDeleteResponse

Permanently delete a finished batch and its stored results. Active batches must settle first.

Parameters:

  • batch_id (String) —

    ID of the batch to retrieve or cancel.

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



76
77
78
79
80
81
82
83
 
# File 'lib/context_dev/resources/batch.rb', line 76

def delete(batch_id, params = {})
  @client.request(
    method: :delete,
    path: ["batch/%1$s", batch_id],
    model: ContextDev::Models::BatchDeleteResponse,
    options: params[:request_options]
  )
end

#get_results(batch_id, cursor: nil, limit: nil, request_options: {}) ⇒ ContextDev::Models::BatchGetResultsResponse

Some parameter documentations has been truncated, see Models::BatchGetResultsParams for more details.

Page through a finished batch's results as JSON instead of downloading the NDJSON files.

Parameters:

  • batch_id (String) —

    ID of the batch to retrieve or cancel.

  • cursor (String) —

    next_cursor from the previous page.

  • limit (Integer) —

    Records per page. Defaults to 25. A page can close early so its payload stays un

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



125
126
127
128
129
130
131
132
133
134
135
 
# File 'lib/context_dev/resources/batch.rb', line 125

def get_results(batch_id, params = {})
  parsed, options = ContextDev::BatchGetResultsParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: ["batch/%1$s/results", batch_id],
    query: query,
    model: ContextDev::Models::BatchGetResultsResponse,
    options: options
  )
end

#list(cursor: nil, limit: nil, q: nil, search_type: nil, status: nil, tags: nil, request_options: {}) ⇒ ContextDev::Models::BatchListResponse

Some parameter documentations has been truncated, see Models::BatchListParams for more details.

List your batches from newest to oldest. Filter by status or continue with a cursor.

Parameters:

  • cursor (String) —

    Cursor from the previous page.

  • limit (Integer) —

    Batches per page. Defaults to 25.

  • q (String) —

    Free-text search term, matched against the batch id, crawl source (start URL or

  • search_type (Symbol, ContextDev::Models::BatchListParams::SearchType) —

    prefix for as-you-type prefix matching (default), exact for full-token match

  • status (Symbol, ContextDev::Models::BatchListParams::Status) —

    Filter by status.

  • tags (String) —

    Comma-separated list of tags to filter by (matches batches having any of them).

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



52
53
54
55
56
57
58
59
60
61
62
 
# File 'lib/context_dev/resources/batch.rb', line 52

def list(params = {})
  parsed, options = ContextDev::BatchListParams.dump_request(params)
  query = ContextDev::Internal::Util.encode_query_params(parsed)
  @client.request(
    method: :get,
    path: "batch/list",
    query: query,
    model: ContextDev::Models::BatchListResponse,
    options: options
  )
end

#retrieve(batch_id, request_options: {}) ⇒ ContextDev::Models::BatchRetrieveResponse

Check progress, and get download links once the batch finishes.

Parameters:

  • batch_id (String) —

    ID of the batch to retrieve or cancel.

  • request_options (ContextDev::RequestOptions, Hash{Symbol=>Object}, nil)

Returns:

See Also:



18
19
20
21
22
23
24
25
 
# File 'lib/context_dev/resources/batch.rb', line 18

def retrieve(batch_id, params = {})
  @client.request(
    method: :get,
    path: ["batch/%1$s", batch_id],
    model: ContextDev::Models::BatchRetrieveResponse,
    options: params[:request_options]
  )
end

#submit(input:, tags: nil, webhook_url: nil, idempotency_key: nil, request_options: {}) ⇒ ContextDev::Models::BatchSubmitResponse

Some parameter documentations has been truncated, see Models::BatchSubmitParams for more details.

Scrape 25K URLs or crawl large websites asynchronously.

Parameters:

Returns:

See Also:



157
158
159
160
161
162
163
164
165
166
167
168
 
# File 'lib/context_dev/resources/batch.rb', line 157

def submit(params)
  parsed, options = ContextDev::BatchSubmitParams.dump_request(params)
  header_params = {idempotency_key: "idempotency-key"}
  @client.request(
    method: :post,
    path: "batch/submit",
    headers: parsed.slice(*header_params.keys).transform_keys(header_params),
    body: parsed.except(*header_params.keys),
    model: ContextDev::Models::BatchSubmitResponse,
    options: options
  )
end