Class: DataDrain::Storage::S3

Inherits:
Base
  • Object
show all
Defined in:
lib/data_drain/storage/s3.rb

Overview

Adaptador de almacenamiento para Amazon S3. Configura credenciales en DuckDB y provee destrucción de particiones vía AWS SDK.

Instance Attribute Summary

Attributes inherited from Base

#config

Instance Method Summary collapse

Methods inherited from Base

#initialize, #prepare_export_path

Constructor Details

This class inherits a constructor from DataDrain::Storage::Base

Instance Method Details

#build_path(bucket, folder_name, partition_path) ⇒ String

Parameters:

  • bucket (String)
  • folder_name (String)
  • partition_path (String, nil)

Returns:

  • (String) 


25
26
27
 
# File 'lib/data_drain/storage/s3.rb', line 25

def build_path(bucket, folder_name, partition_path)
  "s3://#{build_path_base(bucket, folder_name, partition_path)}/**/*.parquet"
end

#destroy_partitions(bucket, folder_name, partition_keys, partitions) ⇒ Integer

Parameters:

  • bucket (String)
  • folder_name (String)
  • partition_keys (Array<Symbol>)
  • partitions (Hash)

Returns:

  • (Integer) 


34
35
36
37
38
39
 
# File 'lib/data_drain/storage/s3.rb', line 34

def destroy_partitions(bucket, folder_name, partition_keys, partitions)
  client = s3_client
  prefix, pattern_regex = build_destroy_pattern(folder_name, partition_keys, partitions)
  objects = collect_matching_objects(client, bucket, prefix, pattern_regex)
  delete_in_batches(client, bucket, objects)
end

#setup_duckdb(connection) ⇒ Object

Carga la extensión httpfs en DuckDB e inyecta las credenciales de AWS. Si aws_access_key_id y aws_secret_access_key están seteados, usa credenciales explícitas. Si no, usa credential_chain (IAM role, env vars, ~/.aws/credentials).

Parameters:

  • connection (DuckDB::Connection)

Raises:



16
17
18
19
 
# File 'lib/data_drain/storage/s3.rb', line 16

def setup_duckdb(connection)
  connection.query("INSTALL httpfs; LOAD httpfs;")
  create_s3_secret(connection)
end