Class: Google::Apis::DatapipelinesV1::DatapipelinesService

Inherits:
Core::BaseService
  • Object
show all
Defined in:
lib/google/apis/datapipelines_v1/service.rb

Overview

Data pipelines API

Data Pipelines provides an interface for creating, updating, and managing recurring Data Analytics jobs.

Examples:

require 'google/apis/datapipelines_v1'

Datapipelines = Google::Apis::DatapipelinesV1 # Alias the module
service = Datapipelines::DatapipelinesService.new

See Also:

Constant Summary collapse

DEFAULT_ENDPOINT_TEMPLATE = 
"https://datapipelines.$UNIVERSE_DOMAIN$/"

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize ⇒ DatapipelinesService

Returns a new instance of DatapipelinesService.



48
49
50
51
52
53
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 48

def initialize
  super(DEFAULT_ENDPOINT_TEMPLATE, '',
        client_name: 'google-apis-datapipelines_v1',
        client_version: Google::Apis::DatapipelinesV1::GEM_VERSION)
  @batch_path = 'batch'
end

Instance Attribute Details

#key ⇒ String

Returns API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.

Returns:

  • (String) —

    API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.



41
42
43
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 41

def key
  @key
end

#quota_user ⇒ String

Returns Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

Returns:

  • (String) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.



46
47
48
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 46

def quota_user
  @quota_user
end

Instance Method Details

#create_project_location_pipeline(parent, google_cloud_datapipelines_v1_pipeline_object = nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline

Creates a pipeline. For a batch pipeline, you can pass scheduler information. Data Pipelines uses the scheduler information to create an internal scheduler that runs jobs periodically. If the internal scheduler is not configured, you can use RunPipeline to run jobs.

Parameters:

  • parent (String) —

    Required. The location name. For example: projects/PROJECT_ID/locations/ LOCATION_ID.

  • google_cloud_datapipelines_v1_pipeline_object (Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline) (defaults to: nil)
  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 80

def create_project_location_pipeline(parent, google_cloud_datapipelines_v1_pipeline_object = nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:post, 'v1/{+parent}/pipelines', options)
  command.request_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.request_object = google_cloud_datapipelines_v1_pipeline_object
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline
  command.params['parent'] = parent unless parent.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#delete_project_location_pipeline(name, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleProtobufEmpty

Deletes a pipeline. If a scheduler job is attached to the pipeline, it will be deleted.

Parameters:

  • name (String) —

    Required. The pipeline name. For example: projects/PROJECT_ID/locations/ LOCATION_ID/pipelines/PIPELINE_ID.

  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



114
115
116
117
118
119
120
121
122
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 114

def delete_project_location_pipeline(name, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:delete, 'v1/{+name}', options)
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleProtobufEmpty::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleProtobufEmpty
  command.params['name'] = name unless name.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#get_project_location_pipeline(name, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline

Looks up a single pipeline. Returns a "NOT_FOUND" error if no such pipeline exists. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.

Parameters:

  • name (String) —

    Required. The pipeline name. For example: projects/PROJECT_ID/locations/ LOCATION_ID/pipelines/PIPELINE_ID.

  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



147
148
149
150
151
152
153
154
155
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 147

def get_project_location_pipeline(name, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:get, 'v1/{+name}', options)
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline
  command.params['name'] = name unless name.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#list_project_location_pipeline_jobs(parent, page_size: nil, page_token: nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListJobsResponse

Lists jobs for a given pipeline. Throws a "FORBIDDEN" error if the caller doesn't have permission to access it.

Parameters:

  • parent (String) —

    Required. The pipeline name. For example: projects/PROJECT_ID/locations/ LOCATION_ID/pipelines/PIPELINE_ID.

  • page_size (Fixnum) (defaults to: nil) —

    The maximum number of entities to return. The service may return fewer than this value, even if there are additional pages. If unspecified, the max limit will be determined by the backend implementation.

  • page_token (String) (defaults to: nil) —

    A page token, received from a previous ListJobs call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListJobs must match the call that provided the page token.

  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



361
362
363
364
365
366
367
368
369
370
371
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 361

def list_project_location_pipeline_jobs(parent, page_size: nil, page_token: nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:get, 'v1/{+parent}/jobs', options)
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListJobsResponse::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListJobsResponse
  command.params['parent'] = parent unless parent.nil?
  command.query['pageSize'] = page_size unless page_size.nil?
  command.query['pageToken'] = page_token unless page_token.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#list_project_location_pipelines(parent, filter: nil, page_size: nil, page_token: nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListPipelinesResponse

Lists pipelines. Returns a "FORBIDDEN" error if the caller doesn't have permission to access it.

Parameters:

  • parent (String) —

    Required. The location name. For example: projects/PROJECT_ID/locations/ LOCATION_ID.

  • filter (String) (defaults to: nil) —

    An expression for filtering the results of the request. If unspecified, all pipelines will be returned. Multiple filters can be applied and must be comma separated. Fields eligible for filtering are: + type: The type of the pipeline (streaming or batch). Allowed values are ALL, BATCH, and STREAMING. + status: The activity status of the pipeline. Allowed values are ALL, ACTIVE, ARCHIVED, and PAUSED. For example, to limit results to active batch processing pipelines: type:BATCH,status:ACTIVE

  • page_size (Fixnum) (defaults to: nil) —

    The maximum number of entities to return. The service may return fewer than this value, even if there are additional pages. If unspecified, the max limit is yet to be determined by the backend implementation.

  • page_token (String) (defaults to: nil) —

    A page token, received from a previous ListPipelines call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListPipelines must match the call that provided the page token.

  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



195
196
197
198
199
200
201
202
203
204
205
206
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 195

def list_project_location_pipelines(parent, filter: nil, page_size: nil, page_token: nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:get, 'v1/{+parent}/pipelines', options)
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListPipelinesResponse::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1ListPipelinesResponse
  command.params['parent'] = parent unless parent.nil?
  command.query['filter'] = filter unless filter.nil?
  command.query['pageSize'] = page_size unless page_size.nil?
  command.query['pageToken'] = page_token unless page_token.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#patch_project_location_pipeline(name, google_cloud_datapipelines_v1_pipeline_object = nil, update_mask: nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline

Updates a pipeline. If successful, the updated Pipeline is returned. Returns NOT_FOUND if the pipeline doesn't exist. If UpdatePipeline does not return successfully, you can retry the UpdatePipeline request until you receive a successful response.

Parameters:

  • name (String) —

    The pipeline name. For example: projects/PROJECT_ID/locations/LOCATION_ID/ pipelines/PIPELINE_ID. * PROJECT_ID can contain letters ([A-Za-z]), numbers ([0-9]), hyphens (-), colons (:), and periods (.). For more information, see Identifying projects. * LOCATION_ID is the canonical ID for the pipeline's location. The list of available locations can be obtained by calling google.cloud.location.Locations.ListLocations. Note that the Data Pipelines service is not available in all regions. It depends on Cloud Scheduler, an App Engine application, so it's only available in App Engine regions. * PIPELINE_ID is the ID of the pipeline. Must be unique for the selected project and location.

  • google_cloud_datapipelines_v1_pipeline_object (Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline) (defaults to: nil)
  • update_mask (String) (defaults to: nil) —

    The list of fields to be updated.

  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



244
245
246
247
248
249
250
251
252
253
254
255
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 244

def patch_project_location_pipeline(name, google_cloud_datapipelines_v1_pipeline_object = nil, update_mask: nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:patch, 'v1/{+name}', options)
  command.request_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.request_object = google_cloud_datapipelines_v1_pipeline_object
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline
  command.params['name'] = name unless name.nil?
  command.query['updateMask'] = update_mask unless update_mask.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#run_project_location_pipeline(name, google_cloud_datapipelines_v1_run_pipeline_request_object = nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1RunPipelineResponse

Creates a job for the specified pipeline directly. You can use this method when the internal scheduler is not configured and you want to trigger the job directly or through an external system. Returns a "NOT_FOUND" error if the pipeline doesn't exist. Returns a "FORBIDDEN" error if the user doesn't have permission to access the pipeline or run jobs for the pipeline.

Parameters:

  • name (String) —

    Required. The pipeline name. For example: projects/PROJECT_ID/locations/ LOCATION_ID/pipelines/PIPELINE_ID.

  • google_cloud_datapipelines_v1_run_pipeline_request_object (Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1RunPipelineRequest) (defaults to: nil)
  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



283
284
285
286
287
288
289
290
291
292
293
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 283

def run_project_location_pipeline(name, google_cloud_datapipelines_v1_run_pipeline_request_object = nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:post, 'v1/{+name}:run', options)
  command.request_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1RunPipelineRequest::Representation
  command.request_object = google_cloud_datapipelines_v1_run_pipeline_request_object
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1RunPipelineResponse::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1RunPipelineResponse
  command.params['name'] = name unless name.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end

#stop_project_location_pipeline(name, google_cloud_datapipelines_v1_stop_pipeline_request_object = nil, fields: nil, quota_user: nil, options: nil) {|result, err| ... } ⇒ Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline

Freezes pipeline execution permanently. If there's a corresponding scheduler entry, it's deleted, and the pipeline state is changed to "ARCHIVED". However, pipeline metadata is retained.

Parameters:

  • name (String) —

    Required. The pipeline name. For example: projects/PROJECT_ID/locations/ LOCATION_ID/pipelines/PIPELINE_ID.

  • google_cloud_datapipelines_v1_stop_pipeline_request_object (Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1StopPipelineRequest) (defaults to: nil)
  • fields (String) (defaults to: nil) —

    Selector specifying which fields to include in a partial response.

  • quota_user (String) (defaults to: nil) —

    Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters.

  • options (Google::Apis::RequestOptions) (defaults to: nil) —

    Request-specific options

Yields:

  • (result, err) —

    Result & error if block supplied

Yield Parameters:

Returns:

Raises:

  • (Google::Apis::ServerError) —

    An error occurred on the server and the request can be retried

  • (Google::Apis::ClientError) —

    The request is invalid and should not be retried without modification

  • (Google::Apis::AuthorizationError) —

    Authorization is required



319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/google/apis/datapipelines_v1/service.rb', line 319

def stop_project_location_pipeline(name, google_cloud_datapipelines_v1_stop_pipeline_request_object = nil, fields: nil, quota_user: nil, options: nil, &block)
  command = make_simple_command(:post, 'v1/{+name}:stop', options)
  command.request_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1StopPipelineRequest::Representation
  command.request_object = google_cloud_datapipelines_v1_stop_pipeline_request_object
  command.response_representation = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline::Representation
  command.response_class = Google::Apis::DatapipelinesV1::GoogleCloudDatapipelinesV1Pipeline
  command.params['name'] = name unless name.nil?
  command.query['fields'] = fields unless fields.nil?
  command.query['quotaUser'] = quota_user unless quota_user.nil?
  execute_or_queue_command(command, &block)
end