Class: Aws::EMR::Client
- Inherits:
-
Seahorse::Client::Base
- Object
- Seahorse::Client::Base
- Aws::EMR::Client
- Includes:
- ClientStubs
- Defined in:
- lib/aws-sdk-emr/client.rb
Overview
An API client for EMR. To construct a client, you need to configure a ‘:region` and `:credentials`.
client = Aws::EMR::Client.new(
region: region_name,
credentials: credentials,
# ...
)
For details on configuring region and credentials see the [developer guide](/sdk-for-ruby/v3/developer-guide/setup-config.html).
See #initialize for a full list of supported configuration options.
Class Attribute Summary collapse
- .identifier ⇒ Object readonly private
API Operations collapse
-
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
-
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
-
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster.
-
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource, such as a cluster or an Amazon EMR Studio.
-
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster.
-
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
-
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
-
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by ‘StudioId`, and applies a session policy to refine Studio permissions for that user or group.
-
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
-
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
-
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
-
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
-
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed.
-
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
-
#describe_release_label(params = {}) ⇒ Types::DescribeReleaseLabelOutput
Provides Amazon EMR release label details, such as the releases available the Region where the API request is run, and the available applications for a specific Amazon EMR release label.
-
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
-
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
-
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
-
#get_auto_termination_policy(params = {}) ⇒ Types::GetAutoTerminationPolicyOutput
Returns the auto-termination policy for an Amazon EMR cluster.
-
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your Amazon Web Services account in the current Region.
-
#get_cluster_session_credentials(params = {}) ⇒ Types::GetClusterSessionCredentialsOutput
Provides temporary, HTTP basic credentials that are associated with a given runtime IAM role and used by a cluster with fine-grained access control activated.
-
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
-
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
-
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
-
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this Amazon Web Services account.
-
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
-
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
-
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active Amazon EC2 instances and Amazon EC2 instances terminated in the last 30 days, up to a maximum of 2,000.
-
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions.
-
#list_release_labels(params = {}) ⇒ Types::ListReleaseLabelsOutput
Retrieves release labels of Amazon EMR services in the Region where the API is called.
-
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names.
-
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you specify ‘stepIds` with the request or filter by `StepStates`.
-
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon EMR Studio specified by ‘StudioId`.
-
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the Amazon Web Services account.
-
#list_supported_instance_types(params = {}) ⇒ Types::ListSupportedInstanceTypesOutput
A list of the instance types that Amazon EMR supports.
-
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
-
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID.
-
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group.
-
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster.
-
#put_auto_termination_policy(params = {}) ⇒ Struct
<note markdown=“1”> Auto-termination is supported in Amazon EMR releases 5.30.0 and 6.1.0 and later.
-
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your Amazon Web Services account in the current Region.
-
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster.
-
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an Amazon EMR cluster.
-
#remove_auto_termination_policy(params = {}) ⇒ Struct
Removes an auto-termination policy from an Amazon EMR cluster.
-
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified Amazon EMR cluster.
-
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource, such as a cluster or Amazon EMR Studio.
-
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow).
-
#set_keep_job_flow_alive_when_no_steps(params = {}) ⇒ Struct
You can use the ‘SetKeepJobFlowAliveWhenNoSteps` to configure a cluster (job flow) to terminate after the step execution, i.e., all your steps are executed.
-
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the Amazon EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error.
-
#set_unhealthy_node_replacement(params = {}) ⇒ Struct
Specify whether to enable unhealthy node replacement, which lets Amazon EMR gracefully replace core nodes on a cluster if any nodes become unhealthy.
-
#set_visible_to_all_users(params = {}) ⇒ Struct
The SetVisibleToAllUsers parameter is no longer supported.
-
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
-
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
-
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down.
-
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
-
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
Class Method Summary collapse
- .errors_module ⇒ Object private
Instance Method Summary collapse
- #build_request(operation_name, params = {}) ⇒ Object private
-
#initialize(options) ⇒ Client
constructor
A new instance of Client.
-
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
- #waiter_names ⇒ Object deprecated private Deprecated.
Constructor Details
#initialize(options) ⇒ Client
Returns a new instance of Client.
429 430 431 |
# File 'lib/aws-sdk-emr/client.rb', line 429 def initialize(*args) super end |
Class Attribute Details
.identifier ⇒ Object (readonly)
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
4317 4318 4319 |
# File 'lib/aws-sdk-emr/client.rb', line 4317 def identifier @identifier end |
Class Method Details
.errors_module ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
4320 4321 4322 |
# File 'lib/aws-sdk-emr/client.rb', line 4320 def errors_module Errors end |
Instance Method Details
#add_instance_fleet(params = {}) ⇒ Types::AddInstanceFleetOutput
Adds an instance fleet to a running cluster.
<note markdown=“1”> The instance fleet configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x.
</note>
535 536 537 538 |
# File 'lib/aws-sdk-emr/client.rb', line 535 def add_instance_fleet(params = {}, = {}) req = build_request(:add_instance_fleet, params) req.send_request() end |
#add_instance_groups(params = {}) ⇒ Types::AddInstanceGroupsOutput
Adds one or more instance groups to a running cluster.
645 646 647 648 |
# File 'lib/aws-sdk-emr/client.rb', line 645 def add_instance_groups(params = {}, = {}) req = build_request(:add_instance_groups, params) req.send_request() end |
#add_job_flow_steps(params = {}) ⇒ Types::AddJobFlowStepsOutput
AddJobFlowSteps adds new steps to a running cluster. A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using SSH to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop.
A step specifies the location of a JAR file stored either on the master node of the cluster or in Amazon S3. Each step is performed by the main function of the main class of the JAR file. The main class can be specified either in the manifest of the JAR or by using the MainFunction parameter of the step.
Amazon EMR executes each step in the order listed. For a step to be considered complete, the main function must exit with a zero exit code and all Hadoop jobs started while the step was running must have completed and run successfully.
You can only add steps to a cluster that is in one of the following states: STARTING, BOOTSTRAPPING, RUNNING, or WAITING.
<note markdown=“1”> The string values passed into ‘HadoopJarStep` object cannot exceed a total of 10240 characters.
</note>
732 733 734 735 |
# File 'lib/aws-sdk-emr/client.rb', line 732 def add_job_flow_steps(params = {}, = {}) req = build_request(:add_job_flow_steps, params) req.send_request() end |
#add_tags(params = {}) ⇒ Struct
Adds tags to an Amazon EMR resource, such as a cluster or an Amazon EMR Studio. Tags make it easier to associate resources in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see [Tag Clusters].
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-tags.html
774 775 776 777 |
# File 'lib/aws-sdk-emr/client.rb', line 774 def (params = {}, = {}) req = build_request(:add_tags, params) req.send_request() end |
#build_request(operation_name, params = {}) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
4178 4179 4180 4181 4182 4183 4184 4185 4186 4187 4188 4189 |
# File 'lib/aws-sdk-emr/client.rb', line 4178 def build_request(operation_name, params = {}) handlers = @handlers.for(operation_name) context = Seahorse::Client::RequestContext.new( operation_name: operation_name, operation: config.api.operation(operation_name), client: self, params: params, config: config) context[:gem_name] = 'aws-sdk-emr' context[:gem_version] = '1.92.0' Seahorse::Client::Request.new(handlers, context) end |
#cancel_steps(params = {}) ⇒ Types::CancelStepsOutput
Cancels a pending step or steps in a running cluster. Available only in Amazon EMR versions 4.8.0 and later, excluding version 5.0.0. A maximum of 256 steps are allowed in each CancelSteps request. CancelSteps is idempotent but asynchronous; it does not guarantee that a step will be canceled, even if the request is successfully submitted. When you use Amazon EMR releases 5.28.0 and later, you can cancel steps that are in a ‘PENDING` or `RUNNING` state. In earlier versions of Amazon EMR, you can only cancel steps that are in a `PENDING` state.
824 825 826 827 |
# File 'lib/aws-sdk-emr/client.rb', line 824 def cancel_steps(params = {}, = {}) req = build_request(:cancel_steps, params) req.send_request() end |
#create_security_configuration(params = {}) ⇒ Types::CreateSecurityConfigurationOutput
Creates a security configuration, which is stored in the service and can be specified when a cluster is created.
865 866 867 868 |
# File 'lib/aws-sdk-emr/client.rb', line 865 def create_security_configuration(params = {}, = {}) req = build_request(:create_security_configuration, params) req.send_request() end |
#create_studio(params = {}) ⇒ Types::CreateStudioOutput
Creates a new Amazon EMR Studio.
996 997 998 999 |
# File 'lib/aws-sdk-emr/client.rb', line 996 def create_studio(params = {}, = {}) req = build_request(:create_studio, params) req.send_request() end |
#create_studio_session_mapping(params = {}) ⇒ Struct
Maps a user or group to the Amazon EMR Studio specified by ‘StudioId`, and applies a session policy to refine Studio permissions for that user or group. Use `CreateStudioSessionMapping` to assign users to a Studio when you use IAM Identity Center authentication. For instructions on how to assign users to a Studio when you use IAM authentication, see [Assign a user or group to your EMR Studio].
1070 1071 1072 1073 |
# File 'lib/aws-sdk-emr/client.rb', line 1070 def create_studio_session_mapping(params = {}, = {}) req = build_request(:create_studio_session_mapping, params) req.send_request() end |
#delete_security_configuration(params = {}) ⇒ Struct
Deletes a security configuration.
1092 1093 1094 1095 |
# File 'lib/aws-sdk-emr/client.rb', line 1092 def delete_security_configuration(params = {}, = {}) req = build_request(:delete_security_configuration, params) req.send_request() end |
#delete_studio(params = {}) ⇒ Struct
Removes an Amazon EMR Studio from the Studio metadata store.
1114 1115 1116 1117 |
# File 'lib/aws-sdk-emr/client.rb', line 1114 def delete_studio(params = {}, = {}) req = build_request(:delete_studio, params) req.send_request() end |
#delete_studio_session_mapping(params = {}) ⇒ Struct
Removes a user or group from an Amazon EMR Studio.
1165 1166 1167 1168 |
# File 'lib/aws-sdk-emr/client.rb', line 1165 def delete_studio_session_mapping(params = {}, = {}) req = build_request(:delete_studio_session_mapping, params) req.send_request() end |
#describe_cluster(params = {}) ⇒ Types::DescribeClusterOutput
Provides cluster-level details including status, hardware and software configuration, VPC settings, and so on.
The following waiters are defined for this operation (see #wait_until for detailed usage):
* cluster_running
* cluster_terminated
1276 1277 1278 1279 |
# File 'lib/aws-sdk-emr/client.rb', line 1276 def describe_cluster(params = {}, = {}) req = build_request(:describe_cluster, params) req.send_request() end |
#describe_job_flows(params = {}) ⇒ Types::DescribeJobFlowsOutput
This API is no longer supported and will eventually be removed. We recommend you use ListClusters, DescribeCluster, ListSteps, ListInstanceGroups and ListBootstrapActions instead.
DescribeJobFlows returns a list of job flows that match all of the supplied parameters. The parameters can include a list of job flow IDs, job flow states, and restrictions on job flow creation date and time.
Regardless of supplied parameters, only job flows created within the last two months are returned.
If no parameters are supplied, then job flows matching either of the following criteria are returned:
-
Job flows created and completed in the last two weeks
-
Job flows created within the last two months that are in one of the following states: ‘RUNNING`, `WAITING`, `SHUTTING_DOWN`, `STARTING`
Amazon EMR can return a maximum of 512 job flow descriptions.
1405 1406 1407 1408 |
# File 'lib/aws-sdk-emr/client.rb', line 1405 def describe_job_flows(params = {}, = {}) req = build_request(:describe_job_flows, params) req.send_request() end |
#describe_notebook_execution(params = {}) ⇒ Types::DescribeNotebookExecutionOutput
Provides details of a notebook execution.
1457 1458 1459 1460 |
# File 'lib/aws-sdk-emr/client.rb', line 1457 def describe_notebook_execution(params = {}, = {}) req = build_request(:describe_notebook_execution, params) req.send_request() end |
#describe_release_label(params = {}) ⇒ Types::DescribeReleaseLabelOutput
Provides Amazon EMR release label details, such as the releases available the Region where the API request is run, and the available applications for a specific Amazon EMR release label. Can also list Amazon EMR releases that support a specified version of Spark.
1505 1506 1507 1508 |
# File 'lib/aws-sdk-emr/client.rb', line 1505 def describe_release_label(params = {}, = {}) req = build_request(:describe_release_label, params) req.send_request() end |
#describe_security_configuration(params = {}) ⇒ Types::DescribeSecurityConfigurationOutput
Provides the details of a security configuration by returning the configuration JSON.
1538 1539 1540 1541 |
# File 'lib/aws-sdk-emr/client.rb', line 1538 def describe_security_configuration(params = {}, = {}) req = build_request(:describe_security_configuration, params) req.send_request() end |
#describe_step(params = {}) ⇒ Types::DescribeStepOutput
Provides more detail about the cluster step.
The following waiters are defined for this operation (see #wait_until for detailed usage):
* step_complete
1593 1594 1595 1596 |
# File 'lib/aws-sdk-emr/client.rb', line 1593 def describe_step(params = {}, = {}) req = build_request(:describe_step, params) req.send_request() end |
#describe_studio(params = {}) ⇒ Types::DescribeStudioOutput
Returns details for the specified Amazon EMR Studio including ID, Name, VPC, Studio access URL, and so on.
1645 1646 1647 1648 |
# File 'lib/aws-sdk-emr/client.rb', line 1645 def describe_studio(params = {}, = {}) req = build_request(:describe_studio, params) req.send_request() end |
#get_auto_termination_policy(params = {}) ⇒ Types::GetAutoTerminationPolicyOutput
Returns the auto-termination policy for an Amazon EMR cluster.
1674 1675 1676 1677 |
# File 'lib/aws-sdk-emr/client.rb', line 1674 def get_auto_termination_policy(params = {}, = {}) req = build_request(:get_auto_termination_policy, params) req.send_request() end |
#get_block_public_access_configuration(params = {}) ⇒ Types::GetBlockPublicAccessConfigurationOutput
Returns the Amazon EMR block public access configuration for your Amazon Web Services account in the current Region. For more information see [Configure Block Public Access for Amazon EMR] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/configure-block-public-access.html
1706 1707 1708 1709 |
# File 'lib/aws-sdk-emr/client.rb', line 1706 def get_block_public_access_configuration(params = {}, = {}) req = build_request(:get_block_public_access_configuration, params) req.send_request() end |
#get_cluster_session_credentials(params = {}) ⇒ Types::GetClusterSessionCredentialsOutput
Provides temporary, HTTP basic credentials that are associated with a given runtime IAM role and used by a cluster with fine-grained access control activated. You can use these credentials to connect to cluster endpoints that support username and password authentication.
1748 1749 1750 1751 |
# File 'lib/aws-sdk-emr/client.rb', line 1748 def get_cluster_session_credentials(params = {}, = {}) req = build_request(:get_cluster_session_credentials, params) req.send_request() end |
#get_managed_scaling_policy(params = {}) ⇒ Types::GetManagedScalingPolicyOutput
Fetches the attached managed scaling policy for an Amazon EMR cluster.
1781 1782 1783 1784 |
# File 'lib/aws-sdk-emr/client.rb', line 1781 def get_managed_scaling_policy(params = {}, = {}) req = build_request(:get_managed_scaling_policy, params) req.send_request() end |
#get_studio_session_mapping(params = {}) ⇒ Types::GetStudioSessionMappingOutput
Fetches mapping details for the specified Amazon EMR Studio and identity (user or group).
1844 1845 1846 1847 |
# File 'lib/aws-sdk-emr/client.rb', line 1844 def get_studio_session_mapping(params = {}, = {}) req = build_request(:get_studio_session_mapping, params) req.send_request() end |
#list_bootstrap_actions(params = {}) ⇒ Types::ListBootstrapActionsOutput
Provides information about the bootstrap actions associated with a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1886 1887 1888 1889 |
# File 'lib/aws-sdk-emr/client.rb', line 1886 def list_bootstrap_actions(params = {}, = {}) req = build_request(:list_bootstrap_actions, params) req.send_request() end |
#list_clusters(params = {}) ⇒ Types::ListClustersOutput
Provides the status of all clusters visible to this Amazon Web Services account. Allows you to filter the list of clusters based on certain criteria; for example, filtering by cluster creation date and time or by status. This call returns a maximum of 50 clusters in unsorted order per call, but returns a marker to track the paging of the cluster list across multiple ListClusters calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
1956 1957 1958 1959 |
# File 'lib/aws-sdk-emr/client.rb', line 1956 def list_clusters(params = {}, = {}) req = build_request(:list_clusters, params) req.send_request() end |
#list_instance_fleets(params = {}) ⇒ Types::ListInstanceFleetsOutput
Lists all available details about the instance fleets in a cluster.
<note markdown=“1”> The instance fleet configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x versions.
</note>
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2040 2041 2042 2043 |
# File 'lib/aws-sdk-emr/client.rb', line 2040 def list_instance_fleets(params = {}, = {}) req = build_request(:list_instance_fleets, params) req.send_request() end |
#list_instance_groups(params = {}) ⇒ Types::ListInstanceGroupsOutput
Provides all available details about the instance groups in a cluster.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2140 2141 2142 2143 |
# File 'lib/aws-sdk-emr/client.rb', line 2140 def list_instance_groups(params = {}, = {}) req = build_request(:list_instance_groups, params) req.send_request() end |
#list_instances(params = {}) ⇒ Types::ListInstancesOutput
Provides information for all active Amazon EC2 instances and Amazon EC2 instances terminated in the last 30 days, up to a maximum of 2,000. Amazon EC2 instances in any of the following states are considered active: AWAITING_FULFILLMENT, PROVISIONING, BOOTSTRAPPING, RUNNING.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2222 2223 2224 2225 |
# File 'lib/aws-sdk-emr/client.rb', line 2222 def list_instances(params = {}, = {}) req = build_request(:list_instances, params) req.send_request() end |
#list_notebook_executions(params = {}) ⇒ Types::ListNotebookExecutionsOutput
Provides summaries of all notebook executions. You can filter the list based on multiple criteria such as status, time range, and editor id. Returns a maximum of 50 notebook executions and a marker to track the paging of a longer notebook execution list across multiple ‘ListNotebookExecutions` calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2318 2319 2320 2321 |
# File 'lib/aws-sdk-emr/client.rb', line 2318 def list_notebook_executions(params = {}, = {}) req = build_request(:list_notebook_executions, params) req.send_request() end |
#list_release_labels(params = {}) ⇒ Types::ListReleaseLabelsOutput
Retrieves release labels of Amazon EMR services in the Region where the API is called.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2372 2373 2374 2375 |
# File 'lib/aws-sdk-emr/client.rb', line 2372 def list_release_labels(params = {}, = {}) req = build_request(:list_release_labels, params) req.send_request() end |
#list_security_configurations(params = {}) ⇒ Types::ListSecurityConfigurationsOutput
Lists all the security configurations visible to this account, providing their creation dates and times, and their names. This call returns a maximum of 50 clusters per call, but returns a marker to track the paging of the cluster list across multiple ListSecurityConfigurations calls.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2410 2411 2412 2413 |
# File 'lib/aws-sdk-emr/client.rb', line 2410 def list_security_configurations(params = {}, = {}) req = build_request(:list_security_configurations, params) req.send_request() end |
#list_steps(params = {}) ⇒ Types::ListStepsOutput
Provides a list of steps for the cluster in reverse order unless you specify ‘stepIds` with the request or filter by `StepStates`. You can specify a maximum of 10 `stepIDs`. The CLI automatically paginates results to return a list greater than 50 steps. To return more than 50 steps using the CLI, specify a `Marker`, which is a pagination token that indicates the next set of steps to retrieve.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2482 2483 2484 2485 |
# File 'lib/aws-sdk-emr/client.rb', line 2482 def list_steps(params = {}, = {}) req = build_request(:list_steps, params) req.send_request() end |
#list_studio_session_mappings(params = {}) ⇒ Types::ListStudioSessionMappingsOutput
Returns a list of all user or group session mappings for the Amazon EMR Studio specified by ‘StudioId`.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2531 2532 2533 2534 |
# File 'lib/aws-sdk-emr/client.rb', line 2531 def list_studio_session_mappings(params = {}, = {}) req = build_request(:list_studio_session_mappings, params) req.send_request() end |
#list_studios(params = {}) ⇒ Types::ListStudiosOutput
Returns a list of all Amazon EMR Studios associated with the Amazon Web Services account. The list includes details such as ID, Studio Access URL, and creation time for each Studio.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2572 2573 2574 2575 |
# File 'lib/aws-sdk-emr/client.rb', line 2572 def list_studios(params = {}, = {}) req = build_request(:list_studios, params) req.send_request() end |
#list_supported_instance_types(params = {}) ⇒ Types::ListSupportedInstanceTypesOutput
A list of the instance types that Amazon EMR supports. You can filter the list by Amazon Web Services Region and Amazon EMR release.
The returned response is a pageable response and is Enumerable. For details on usage see PageableResponse.
2631 2632 2633 2634 |
# File 'lib/aws-sdk-emr/client.rb', line 2631 def list_supported_instance_types(params = {}, = {}) req = build_request(:list_supported_instance_types, params) req.send_request() end |
#modify_cluster(params = {}) ⇒ Types::ModifyClusterOutput
Modifies the number of steps that can be executed concurrently for the cluster specified using ClusterID.
2668 2669 2670 2671 |
# File 'lib/aws-sdk-emr/client.rb', line 2668 def modify_cluster(params = {}, = {}) req = build_request(:modify_cluster, params) req.send_request() end |
#modify_instance_fleet(params = {}) ⇒ Struct
Modifies the target On-Demand and target Spot capacities for the instance fleet with the specified InstanceFleetID within the cluster specified using ClusterID. The call either succeeds or fails atomically.
<note markdown=“1”> The instance fleet configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x versions.
</note>
2714 2715 2716 2717 |
# File 'lib/aws-sdk-emr/client.rb', line 2714 def modify_instance_fleet(params = {}, = {}) req = build_request(:modify_instance_fleet, params) req.send_request() end |
#modify_instance_groups(params = {}) ⇒ Struct
ModifyInstanceGroups modifies the number of nodes and configuration settings of an instance group. The input parameters include the new target instance count for the group and the instance group ID. The call will either succeed or fail atomically.
2769 2770 2771 2772 |
# File 'lib/aws-sdk-emr/client.rb', line 2769 def modify_instance_groups(params = {}, = {}) req = build_request(:modify_instance_groups, params) req.send_request() end |
#put_auto_scaling_policy(params = {}) ⇒ Types::PutAutoScalingPolicyOutput
Creates or updates an automatic scaling policy for a core instance group or task instance group in an Amazon EMR cluster. The automatic scaling policy defines how an instance group dynamically adds and terminates Amazon EC2 instances in response to the value of a CloudWatch metric.
2876 2877 2878 2879 |
# File 'lib/aws-sdk-emr/client.rb', line 2876 def put_auto_scaling_policy(params = {}, = {}) req = build_request(:put_auto_scaling_policy, params) req.send_request() end |
#put_auto_termination_policy(params = {}) ⇒ Struct
<note markdown=“1”> Auto-termination is supported in Amazon EMR releases 5.30.0 and 6.1.0 and later. For more information, see [Using an auto-termination policy].
</note>
Creates or updates an auto-termination policy for an Amazon EMR cluster. An auto-termination policy defines the amount of idle time in seconds after which a cluster automatically terminates. For alternative cluster termination options, see [Control cluster termination].
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-auto-termination-policy.html [2]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-termination.html
2920 2921 2922 2923 |
# File 'lib/aws-sdk-emr/client.rb', line 2920 def put_auto_termination_policy(params = {}, = {}) req = build_request(:put_auto_termination_policy, params) req.send_request() end |
#put_block_public_access_configuration(params = {}) ⇒ Struct
Creates or updates an Amazon EMR block public access configuration for your Amazon Web Services account in the current Region. For more information see [Configure Block Public Access for Amazon EMR] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/configure-block-public-access.html
2975 2976 2977 2978 |
# File 'lib/aws-sdk-emr/client.rb', line 2975 def put_block_public_access_configuration(params = {}, = {}) req = build_request(:put_block_public_access_configuration, params) req.send_request() end |
#put_managed_scaling_policy(params = {}) ⇒ Struct
Creates or updates a managed scaling policy for an Amazon EMR cluster. The managed scaling policy defines the limits for resources, such as Amazon EC2 instances that can be added or terminated from a cluster. The policy only applies to the core and task nodes. The master node cannot be scaled after initial configuration.
3014 3015 3016 3017 |
# File 'lib/aws-sdk-emr/client.rb', line 3014 def put_managed_scaling_policy(params = {}, = {}) req = build_request(:put_managed_scaling_policy, params) req.send_request() end |
#remove_auto_scaling_policy(params = {}) ⇒ Struct
Removes an automatic scaling policy from a specified instance group within an Amazon EMR cluster.
3043 3044 3045 3046 |
# File 'lib/aws-sdk-emr/client.rb', line 3043 def remove_auto_scaling_policy(params = {}, = {}) req = build_request(:remove_auto_scaling_policy, params) req.send_request() end |
#remove_auto_termination_policy(params = {}) ⇒ Struct
Removes an auto-termination policy from an Amazon EMR cluster.
3066 3067 3068 3069 |
# File 'lib/aws-sdk-emr/client.rb', line 3066 def remove_auto_termination_policy(params = {}, = {}) req = build_request(:remove_auto_termination_policy, params) req.send_request() end |
#remove_managed_scaling_policy(params = {}) ⇒ Struct
Removes a managed scaling policy from a specified Amazon EMR cluster.
3089 3090 3091 3092 |
# File 'lib/aws-sdk-emr/client.rb', line 3089 def remove_managed_scaling_policy(params = {}, = {}) req = build_request(:remove_managed_scaling_policy, params) req.send_request() end |
#remove_tags(params = {}) ⇒ Struct
Removes tags from an Amazon EMR resource, such as a cluster or Amazon EMR Studio. Tags make it easier to associate resources in various ways, such as grouping clusters to track your Amazon EMR resource allocation costs. For more information, see [Tag Clusters].
The following example removes the stack tag with value Prod from a cluster:
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-tags.html
3126 3127 3128 3129 |
# File 'lib/aws-sdk-emr/client.rb', line 3126 def (params = {}, = {}) req = build_request(:remove_tags, params) req.send_request() end |
#run_job_flow(params = {}) ⇒ Types::RunJobFlowOutput
RunJobFlow creates and starts running a new cluster (job flow). The cluster runs the steps specified. After the steps complete, the cluster stops and the HDFS partition is lost. To prevent loss of data, configure the last step of the job flow to store results in Amazon S3. If the JobFlowInstancesConfig ‘KeepJobFlowAliveWhenNoSteps` parameter is set to `TRUE`, the cluster transitions to the WAITING state rather than shutting down after the steps have completed.
For additional protection, you can set the JobFlowInstancesConfig ‘TerminationProtected` parameter to `TRUE` to lock the cluster and prevent it from being terminated by API call, user intervention, or in the event of a job flow error.
A maximum of 256 steps are allowed in each job flow.
If your cluster is long-running (such as a Hive data warehouse) or complex, you may require more than 256 steps to process your data. You can bypass the 256-step limitation in various ways, including using the SSH shell to connect to the master node and submitting queries directly to the software running on the master node, such as Hive and Hadoop.
For long-running clusters, we recommend that you periodically store your results.
<note markdown=“1”> The instance fleets configuration is available only in Amazon EMR releases 4.8.0 and later, excluding 5.0.x versions. The RunJobFlow request can contain InstanceFleets parameters or InstanceGroups parameters, but not both.
</note>
3703 3704 3705 3706 |
# File 'lib/aws-sdk-emr/client.rb', line 3703 def run_job_flow(params = {}, = {}) req = build_request(:run_job_flow, params) req.send_request() end |
#set_keep_job_flow_alive_when_no_steps(params = {}) ⇒ Struct
You can use the ‘SetKeepJobFlowAliveWhenNoSteps` to configure a cluster (job flow) to terminate after the step execution, i.e., all your steps are executed. If you want a transient cluster that shuts down after the last of the current executing steps are completed, you can configure `SetKeepJobFlowAliveWhenNoSteps` to false. If you want a long running cluster, configure `SetKeepJobFlowAliveWhenNoSteps` to true.
For more information, see [Managing Cluster Termination] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_TerminationProtection.html
3750 3751 3752 3753 |
# File 'lib/aws-sdk-emr/client.rb', line 3750 def set_keep_job_flow_alive_when_no_steps(params = {}, = {}) req = build_request(:set_keep_job_flow_alive_when_no_steps, params) req.send_request() end |
#set_termination_protection(params = {}) ⇒ Struct
SetTerminationProtection locks a cluster (job flow) so the Amazon EC2 instances in the cluster cannot be terminated by user intervention, an API call, or in the event of a job-flow error. The cluster still terminates upon successful completion of the job flow. Calling ‘SetTerminationProtection` on a cluster is similar to calling the Amazon EC2 `DisableAPITermination` API on all Amazon EC2 instances in a cluster.
‘SetTerminationProtection` is used to prevent accidental termination of a cluster and to ensure that in the event of an error, the instances persist so that you can recover any data stored in their ephemeral instance storage.
To terminate a cluster that has been locked by setting ‘SetTerminationProtection` to `true`, you must first unlock the job flow by a subsequent call to `SetTerminationProtection` in which you set the value to `false`.
For more information, see [Managing Cluster Termination] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/UsingEMR_TerminationProtection.html
3803 3804 3805 3806 |
# File 'lib/aws-sdk-emr/client.rb', line 3803 def set_termination_protection(params = {}, = {}) req = build_request(:set_termination_protection, params) req.send_request() end |
#set_unhealthy_node_replacement(params = {}) ⇒ Struct
Specify whether to enable unhealthy node replacement, which lets Amazon EMR gracefully replace core nodes on a cluster if any nodes become unhealthy. For example, a node becomes unhealthy if disk usage is above 90%. If unhealthy node replacement is on and ‘TerminationProtected` are off, Amazon EMR immediately terminates the unhealthy core nodes. To use unhealthy node replacement and retain unhealthy core nodes, use to turn on termination protection. In such cases, Amazon EMR adds the unhealthy nodes to a denylist, reducing job interruptions and failures.
If unhealthy node replacement is on, Amazon EMR notifies YARN and other applications on the cluster to stop scheduling tasks with these nodes, moves the data, and then terminates the nodes.
For more information, see [graceful node replacement] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-node-replacement.html
3851 3852 3853 3854 |
# File 'lib/aws-sdk-emr/client.rb', line 3851 def set_unhealthy_node_replacement(params = {}, = {}) req = build_request(:set_unhealthy_node_replacement, params) req.send_request() end |
#set_visible_to_all_users(params = {}) ⇒ Struct
The SetVisibleToAllUsers parameter is no longer supported. Your cluster may be visible to all users in your account. To restrict cluster access using an IAM policy, see [Identity and Access Management for Amazon EMR].
Sets the Cluster$VisibleToAllUsers value for an Amazon EMR cluster. When ‘true`, IAM principals in the Amazon Web Services account can perform Amazon EMR cluster actions that their IAM policies allow. When `false`, only the IAM principal that created the cluster and the Amazon Web Services account root user can perform Amazon EMR actions on the cluster, regardless of IAM permissions policies attached to other IAM principals.
This action works on running clusters. When you create a cluster, use the RunJobFlowInput$VisibleToAllUsers parameter.
For more information, see [Understanding the Amazon EMR Cluster VisibleToAllUsers Setting] in the *Amazon EMR Management Guide*.
[1]: docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-access-IAM.html [2]: docs.aws.amazon.com/emr/latest/ManagementGuide/security_IAM_emr-with-IAM.html#security_set_visible_to_all_users
3904 3905 3906 3907 |
# File 'lib/aws-sdk-emr/client.rb', line 3904 def set_visible_to_all_users(params = {}, = {}) req = build_request(:set_visible_to_all_users, params) req.send_request() end |
#start_notebook_execution(params = {}) ⇒ Types::StartNotebookExecutionOutput
Starts a notebook execution.
4010 4011 4012 4013 |
# File 'lib/aws-sdk-emr/client.rb', line 4010 def start_notebook_execution(params = {}, = {}) req = build_request(:start_notebook_execution, params) req.send_request() end |
#stop_notebook_execution(params = {}) ⇒ Struct
Stops a notebook execution.
4032 4033 4034 4035 |
# File 'lib/aws-sdk-emr/client.rb', line 4032 def stop_notebook_execution(params = {}, = {}) req = build_request(:stop_notebook_execution, params) req.send_request() end |
#terminate_job_flows(params = {}) ⇒ Struct
TerminateJobFlows shuts a list of clusters (job flows) down. When a job flow is shut down, any step not yet completed is canceled and the Amazon EC2 instances on which the cluster is running are stopped. Any log files not already saved are uploaded to Amazon S3 if a LogUri was specified when the cluster was created.
The maximum number of clusters allowed is 10. The call to ‘TerminateJobFlows` is asynchronous. Depending on the configuration of the cluster, it may take up to 1-5 minutes for the cluster to completely terminate and release allocated resources, such as Amazon EC2 instances.
4064 4065 4066 4067 |
# File 'lib/aws-sdk-emr/client.rb', line 4064 def terminate_job_flows(params = {}, = {}) req = build_request(:terminate_job_flows, params) req.send_request() end |
#update_studio(params = {}) ⇒ Struct
Updates an Amazon EMR Studio configuration, including attributes such as name, description, and subnets.
4113 4114 4115 4116 |
# File 'lib/aws-sdk-emr/client.rb', line 4113 def update_studio(params = {}, = {}) req = build_request(:update_studio, params) req.send_request() end |
#update_studio_session_mapping(params = {}) ⇒ Struct
Updates the session policy attached to the user or group for the specified Amazon EMR Studio.
4169 4170 4171 4172 |
# File 'lib/aws-sdk-emr/client.rb', line 4169 def update_studio_session_mapping(params = {}, = {}) req = build_request(:update_studio_session_mapping, params) req.send_request() end |
#wait_until(waiter_name, params = {}, options = {}) {|w.waiter| ... } ⇒ Boolean
Polls an API operation until a resource enters a desired state.
## Basic Usage
A waiter will call an API operation until:
-
It is successful
-
It enters a terminal state
-
It makes the maximum number of attempts
In between attempts, the waiter will sleep.
# polls in a loop, sleeping between attempts
client.wait_until(waiter_name, params)
## Configuration
You can configure the maximum number of polling attempts, and the delay (in seconds) between each polling attempt. You can pass configuration as the final arguments hash.
# poll for ~25 seconds
client.wait_until(waiter_name, params, {
max_attempts: 5,
delay: 5,
})
## Callbacks
You can be notified before each polling attempt and before each delay. If you throw ‘:success` or `:failure` from these callbacks, it will terminate the waiter.
started_at = Time.now
client.wait_until(waiter_name, params, {
# disable max attempts
max_attempts: nil,
# poll for 1 hour, instead of a number of attempts
before_wait: -> (attempts, response) do
throw :failure if Time.now - started_at > 3600
end
})
## Handling Errors
When a waiter is unsuccessful, it will raise an error. All of the failure errors extend from Waiters::Errors::WaiterFailed.
begin
client.wait_until(...)
rescue Aws::Waiters::Errors::WaiterFailed
# resource did not enter the desired state in time
end
## Valid Waiters
The following table lists the valid waiter names, the operations they call, and the default ‘:delay` and `:max_attempts` values.
| waiter_name | params | :delay | :max_attempts | | —————— | ————————- | ——– | ————- | | cluster_running | #describe_cluster | 30 | 60 | | cluster_terminated | #describe_cluster | 30 | 60 | | step_complete | #describe_step | 30 | 60 |
4281 4282 4283 4284 4285 |
# File 'lib/aws-sdk-emr/client.rb', line 4281 def wait_until(waiter_name, params = {}, = {}) w = waiter(waiter_name, ) yield(w.waiter) if block_given? # deprecated w.wait(params) end |
#waiter_names ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
4289 4290 4291 |
# File 'lib/aws-sdk-emr/client.rb', line 4289 def waiter_names waiters.keys end |