Class: Aws::Glue::Types::KinesisStreamingSourceOptions

Inherits:

Struct

Object
Struct
Aws::Glue::Types::KinesisStreamingSourceOptions

show all

Includes:: Structure

Defined in:: lib/aws-sdk-glue/types.rb

Overview

Note:

When making an API call, you may pass KinesisStreamingSourceOptions data as a hash:

{
  endpoint_url: "EnclosedInStringProperty",
  stream_name: "EnclosedInStringProperty",
  classification: "EnclosedInStringProperty",
  delimiter: "EnclosedInStringProperty",
  starting_position: "latest", # accepts latest, trim_horizon, earliest
  max_fetch_time_in_ms: 1,
  max_fetch_records_per_shard: 1,
  max_record_per_read: 1,
  add_idle_time_between_reads: false,
  idle_time_between_reads_in_ms: 1,
  describe_shard_interval: 1,
  num_retries: 1,
  retry_interval_ms: 1,
  max_retry_interval_ms: 1,
  avoid_empty_batches: false,
  stream_arn: "EnclosedInStringProperty",
  role_arn: "EnclosedInStringProperty",
  role_session_name: "EnclosedInStringProperty",
}

Additional options for the Amazon Kinesis streaming data source.

Constant Summary collapse

SENSITIVE =

[]

Instance Attribute Summary collapse

#add_idle_time_between_reads ⇒ Boolean

Adds a time delay between two consecutive getRecords operations.
#avoid_empty_batches ⇒ Boolean

Avoids creating an empty microbatch job by checking for unread data in the Kinesis data stream before the batch is started.
#classification ⇒ String

An optional classification.
#delimiter ⇒ String

Specifies the delimiter character.
#describe_shard_interval ⇒ Integer

The minimum time interval between two ListShards API calls for your script to consider resharding.
#endpoint_url ⇒ String

The URL of the Kinesis endpoint.
#idle_time_between_reads_in_ms ⇒ Integer

The minimum time delay between two consecutive getRecords operations, specified in ms.
#max_fetch_records_per_shard ⇒ Integer

The maximum number of records to fetch per shard in the Kinesis data stream.
#max_fetch_time_in_ms ⇒ Integer

The maximum time spent in the job executor to fetch a record from the Kinesis data stream per shard, specified in milliseconds (ms).
#max_record_per_read ⇒ Integer

The maximum number of records to fetch from the Kinesis data stream in each getRecords operation.
#max_retry_interval_ms ⇒ Integer

The maximum cool-off time period (specified in ms) between two retries of a Kinesis Data Streams API call.
#num_retries ⇒ Integer

The maximum number of retries for Kinesis Data Streams API requests.
#retry_interval_ms ⇒ Integer

The cool-off time period (specified in ms) before retrying the Kinesis Data Streams API call.
#role_arn ⇒ String

The Amazon Resource Name (ARN) of the role to assume using AWS Security Token Service (AWS STS).
#role_session_name ⇒ String

An identifier for the session assuming the role using AWS STS.
#starting_position ⇒ String

The starting position in the Kinesis data stream to read data from.
#stream_arn ⇒ String

The Amazon Resource Name (ARN) of the Kinesis data stream.
#stream_name ⇒ String

The name of the Kinesis data stream.

Instance Attribute Details

#add_idle_time_between_reads ⇒ `Boolean`

Adds a time delay between two consecutive getRecords operations. The default value is `“False”`. This option is only configurable for Glue version 2.0 and above.

Returns:

(Boolean)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#avoid_empty_batches ⇒ `Boolean`

Avoids creating an empty microbatch job by checking for unread data in the Kinesis data stream before the batch is started. The default value is `“False”`.

Returns:

(Boolean)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#classification ⇒ `String`

An optional classification.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#delimiter ⇒ `String`

Specifies the delimiter character.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#describe_shard_interval ⇒ `Integer`

The minimum time interval between two ListShards API calls for your script to consider resharding. The default value is `1s`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#endpoint_url ⇒ `String`

The URL of the Kinesis endpoint.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#idle_time_between_reads_in_ms ⇒ `Integer`

The minimum time delay between two consecutive getRecords operations, specified in ms. The default value is `1000`. This option is only configurable for Glue version 2.0 and above.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#max_fetch_records_per_shard ⇒ `Integer`

The maximum number of records to fetch per shard in the Kinesis data stream. The default value is `100000`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#max_fetch_time_in_ms ⇒ `Integer`

The maximum time spent in the job executor to fetch a record from the Kinesis data stream per shard, specified in milliseconds (ms). The default value is `1000`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#max_record_per_read ⇒ `Integer`

The maximum number of records to fetch from the Kinesis data stream in each getRecords operation. The default value is `10000`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#max_retry_interval_ms ⇒ `Integer`

The maximum cool-off time period (specified in ms) between two retries of a Kinesis Data Streams API call. The default value is `10000`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#num_retries ⇒ `Integer`

The maximum number of retries for Kinesis Data Streams API requests. The default value is `3`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#retry_interval_ms ⇒ `Integer`

The cool-off time period (specified in ms) before retrying the Kinesis Data Streams API call. The default value is `1000`.

Returns:

(Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#role_arn ⇒ `String`

The Amazon Resource Name (ARN) of the role to assume using AWS Security Token Service (AWS STS). This role must have permissions for describe or read record operations for the Kinesis data stream. You must use this parameter when accessing a data stream in a different account. Used in conjunction with `“awsSTSSessionName”`.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#role_session_name ⇒ `String`

An identifier for the session assuming the role using AWS STS. You must use this parameter when accessing a data stream in a different account. Used in conjunction with `“awsSTSRoleARN”`.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#starting_position ⇒ `String`

The starting position in the Kinesis data stream to read data from. The possible values are `“latest”`, `“trim_horizon”`, or `“earliest”`. The default value is `“latest”`.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#stream_arn ⇒ `String`

The Amazon Resource Name (ARN) of the Kinesis data stream.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

#stream_name ⇒ `String`

The name of the Kinesis data stream.

Returns:

(String)

# File 'lib/aws-sdk-glue/types.rb', line 16376

class KinesisStreamingSourceOptions < Struct.new(
  :endpoint_url,
  :stream_name,
  :classification,
  :delimiter,
  :starting_position,
  :max_fetch_time_in_ms,
  :max_fetch_records_per_shard,
  :max_record_per_read,
  :add_idle_time_between_reads,
  :idle_time_between_reads_in_ms,
  :describe_shard_interval,
  :num_retries,
  :retry_interval_ms,
  :max_retry_interval_ms,
  :avoid_empty_batches,
  :stream_arn,
  :role_arn,
  :role_session_name)
  SENSITIVE = []
  include Aws::Structure
end

Class: Aws::Glue::Types::KinesisStreamingSourceOptions

Overview

Constant Summary collapse

Instance Attribute Summary collapse

Instance Attribute Details

#add_idle_time_between_reads ⇒ Boolean

#avoid_empty_batches ⇒ Boolean

#classification ⇒ String

#delimiter ⇒ String

#describe_shard_interval ⇒ Integer

#endpoint_url ⇒ String

#idle_time_between_reads_in_ms ⇒ Integer

#max_fetch_records_per_shard ⇒ Integer

#max_fetch_time_in_ms ⇒ Integer

#max_record_per_read ⇒ Integer

#max_retry_interval_ms ⇒ Integer

#num_retries ⇒ Integer

#retry_interval_ms ⇒ Integer

#role_arn ⇒ String

#role_session_name ⇒ String

#starting_position ⇒ String

#stream_arn ⇒ String

#stream_name ⇒ String

#add_idle_time_between_reads ⇒ `Boolean`

#avoid_empty_batches ⇒ `Boolean`

#classification ⇒ `String`

#delimiter ⇒ `String`

#describe_shard_interval ⇒ `Integer`

#endpoint_url ⇒ `String`

#idle_time_between_reads_in_ms ⇒ `Integer`

#max_fetch_records_per_shard ⇒ `Integer`

#max_fetch_time_in_ms ⇒ `Integer`

#max_record_per_read ⇒ `Integer`

#max_retry_interval_ms ⇒ `Integer`

#num_retries ⇒ `Integer`

#retry_interval_ms ⇒ `Integer`

#role_arn ⇒ `String`

#role_session_name ⇒ `String`

#starting_position ⇒ `String`

#stream_arn ⇒ `String`

#stream_name ⇒ `String`