Class: Aws::S3::DirectoryUploader Private

Inherits:
Object
  • Object
show all
Defined in:
lib/aws-sdk-s3/directory_uploader.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

This is a one-shot class that uploads files from a local directory to a bucket. This works as follows:

  • FileProducer runs in a background thread, scanning the directory and pushing entries into a SizedQueue (max: 100).

  • An internal executor pulls from that queue and posts work. Each task uses FileUploader to upload files then signals completion via ‘completion_queue`.

We track how many tasks we posted, then pop that many times from ‘completion_queue` to wait for everything to finish.

Errors are collected in a mutex-protected array. On failure (unless ignore_failure is set), we call abort which closes the queue - the producer catches ClosedQueueError and exits cleanly.

Defined Under Namespace

Classes: FileProducer

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ DirectoryUploader

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of DirectoryUploader.



21
22
23
24
25
26
27
 
# File 'lib/aws-sdk-s3/directory_uploader.rb', line 21

def initialize(options = {})
  @client = options[:client] || Client.new
  @executor = options[:executor] || DefaultExecutor.new
  @logger = options[:logger]
  @producer = nil
  @mutex = Mutex.new
end

Instance Attribute Details

#client ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



29
30
31
 
# File 'lib/aws-sdk-s3/directory_uploader.rb', line 29

def client
  @client
end

#executor ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



29
30
31
 
# File 'lib/aws-sdk-s3/directory_uploader.rb', line 29

def executor
  @executor
end

Instance Method Details

#abort ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



31
32
33
 
# File 'lib/aws-sdk-s3/directory_uploader.rb', line 31

def abort
  @producer&.close
end

#upload(source_directory, bucket, **opts) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Raises:

  • (ArgumentError) 


35
36
37
38
39
40
41
42
43
44
45
46
47
48
 
# File 'lib/aws-sdk-s3/directory_uploader.rb', line 35

def upload(source_directory, bucket, **opts)
  raise ArgumentError, 'Invalid directory' unless Dir.exist?(source_directory)

  uploader = FileUploader.new(
    multipart_threshold: opts.delete(:multipart_threshold),
    http_chunk_size: opts.delete(:http_chunk_size),
    client: @client,
    executor: @executor
  )
  upload_opts = build_upload_opts(opts)
  @producer = FileProducer.new(build_producer_opts(source_directory, bucket, opts))
  uploads, errors = process_upload_queue(uploader, upload_opts)
  build_result(uploads, errors)
end