Class: Aws::S3::DirectoryDownloader Private

Inherits:
Object
  • Object
show all
Defined in:
lib/aws-sdk-s3/directory_downloader.rb

Overview

This class is part of a private API. You should avoid using this class if possible, as it may be removed or be changed in the future.

This is a one-shot class that downloads objects from a bucket to a local directory. This works as follows:

  • ObjectProducer runs in a background thread, calling ‘list_objects_v2` and pushing entries into a SizedQueue (max: 100).

  • An internal executor pulls from that queue and posts work. Each task uses FileDownloader to download objects then signals completion via ‘completion_queue`.

We track how many tasks we posted, then pop that many times from ‘completion_queue` to wait for everything to finish.

Errors are collected in a mutex-protected array. On failure (unless ignore_failure is set), we call abort which closes the queue - the producer catches ClosedQueueError and exits cleanly.

Defined Under Namespace

Classes: ObjectProducer

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(options = {}) ⇒ DirectoryDownloader

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns a new instance of DirectoryDownloader.



19
20
21
22
23
24
25
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 19

def initialize(options = {})
  @client = options[:client] || Client.new
  @executor = options[:executor] || DefaultExecutor.new
  @logger = options[:logger]
  @producer = nil
  @mutex = Mutex.new
end

Instance Attribute Details

#client ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



27
28
29
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 27

def client
  @client
end

#executor ⇒ Object (readonly)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



27
28
29
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 27

def executor
  @executor
end

Instance Method Details

#abort ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



29
30
31
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 29

def abort
  @producer&.close
end

#download(destination, bucket:, **options) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



33
34
35
36
37
38
39
40
41
42
43
44
45
 
# File 'lib/aws-sdk-s3/directory_downloader.rb', line 33

def download(destination, bucket:, **options)
  if File.exist?(destination)
    raise ArgumentError, 'invalid destination, expected a directory' unless File.directory?(destination)
  else
    FileUtils.mkdir_p(destination)
  end

  download_opts = build_download_opts(destination, options)
  @producer = ObjectProducer.new(build_producer_opts(destination, bucket, options))
  downloader = FileDownloader.new(client: @client, executor: @executor)
  downloads, errors = process_download_queue(downloader, download_opts)
  build_result(downloads, errors)
end