Class: Aws::Glue::Types::KafkaStreamingSourceOptions

Inherits:

Struct

• Object
• Struct
• Aws::Glue::Types::KafkaStreamingSourceOptions

Includes:

Structure

Defined in:

lib/aws-sdk-glue/types.rb

Overview

Note:

When making an API call, you may pass KafkaStreamingSourceOptions data as a hash:

{
  bootstrap_servers: "EnclosedInStringProperty",
  security_protocol: "EnclosedInStringProperty",
  connection_name: "EnclosedInStringProperty",
  topic_name: "EnclosedInStringProperty",
  assign: "EnclosedInStringProperty",
  subscribe_pattern: "EnclosedInStringProperty",
  classification: "EnclosedInStringProperty",
  delimiter: "EnclosedInStringProperty",
  starting_offsets: "EnclosedInStringProperty",
  ending_offsets: "EnclosedInStringProperty",
  poll_timeout_ms: 1,
  num_retries: 1,
  retry_interval_ms: 1,
  max_offsets_per_trigger: 1,
  min_partitions: 1,
}

Additional options for streaming.

See Also:

• AWS API Documentation

Constant Summary

SENSITIVE =

[]

Instance Attribute Summary

• #assign ⇒ String

  The specific `TopicPartitions` to consume.

• #bootstrap_servers ⇒ String

  A list of bootstrap server URLs, for example, as `b-1.vpc-test-2.o4q88o.c6.kafka.us-east-1.amazonaws.com:9094`.

• #classification ⇒ String

  An optional classification.

• #connection_name ⇒ String

  The name of the connection.

• #delimiter ⇒ String

  Specifies the delimiter character.

• #ending_offsets ⇒ String

  The end point when a batch query is ended.

• #max_offsets_per_trigger ⇒ Integer

  The rate limit on the maximum number of offsets that are processed per trigger interval.

• #min_partitions ⇒ Integer

  The desired minimum number of partitions to read from Kafka.

• #num_retries ⇒ Integer

  The number of times to retry before failing to fetch Kafka offsets.

• #poll_timeout_ms ⇒ Integer

  The timeout in milliseconds to poll data from Kafka in Spark job executors.

• #retry_interval_ms ⇒ Integer

  The time in milliseconds to wait before retrying to fetch Kafka offsets.

• #security_protocol ⇒ String

  The protocol used to communicate with brokers.

• #starting_offsets ⇒ String

  The starting position in the Kafka topic to read data from.

• #subscribe_pattern ⇒ String

  A Java regex string that identifies the topic list to subscribe to.

• #topic_name ⇒ String

  The topic name as specified in Apache Kafka.

Instance Attribute Details

#assign ⇒ String

The specific `TopicPartitions` to consume. You must specify at least one of `“topicName”`, `“assign”` or `“subscribePattern”`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#bootstrap_servers ⇒ String

A list of bootstrap server URLs, for example, as `b-1.vpc-test-2.o4q88o.c6.kafka.us-east-1.amazonaws.com:9094`. This option must be specified in the API call or defined in the table metadata in the Data Catalog.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#classification ⇒ String

An optional classification.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#connection_name ⇒ String

The name of the connection.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#delimiter ⇒ String

Specifies the delimiter character.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#ending_offsets ⇒ String

The end point when a batch query is ended. Possible values are either `“latest”` or a JSON string that specifies an ending offset for each `TopicPartition`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#max_offsets_per_trigger ⇒ Integer

The rate limit on the maximum number of offsets that are processed per trigger interval. The specified total number of offsets is proportionally split across `topicPartitions` of different volumes. The default value is null, which means that the consumer reads all offsets until the known latest offset.

Returns:

• (Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#min_partitions ⇒ Integer

The desired minimum number of partitions to read from Kafka. The default value is null, which means that the number of spark partitions is equal to the number of Kafka partitions.

Returns:

• (Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#num_retries ⇒ Integer

The number of times to retry before failing to fetch Kafka offsets. The default value is `3`.

Returns:

• (Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#poll_timeout_ms ⇒ Integer

The timeout in milliseconds to poll data from Kafka in Spark job executors. The default value is `512`.

Returns:

• (Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#retry_interval_ms ⇒ Integer

The time in milliseconds to wait before retrying to fetch Kafka offsets. The default value is `10`.

Returns:

• (Integer)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#security_protocol ⇒ String

The protocol used to communicate with brokers. The possible values are `“SSL”` or `“PLAINTEXT”`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#starting_offsets ⇒ String

The starting position in the Kafka topic to read data from. The possible values are `“earliest”` or `“latest”`. The default value is `“latest”`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#subscribe_pattern ⇒ String

A Java regex string that identifies the topic list to subscribe to. You must specify at least one of `“topicName”`, `“assign”` or `“subscribePattern”`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end

#topic_name ⇒ String

The topic name as specified in Apache Kafka. You must specify at least one of `“topicName”`, `“assign”` or `“subscribePattern”`.

Returns:

• (String)

# File 'lib/aws-sdk-glue/types.rb', line 16048

class KafkaStreamingSourceOptions < Struct.new(
  :bootstrap_servers,
  :security_protocol,
  :connection_name,
  :topic_name,
  :assign,
  :subscribe_pattern,
  :classification,
  :delimiter,
  :starting_offsets,
  :ending_offsets,
  :poll_timeout_ms,
  :num_retries,
  :retry_interval_ms,
  :max_offsets_per_trigger,
  :min_partitions)
  SENSITIVE = []
  include Aws::Structure
end