Class: Shrine::Storage::S3

Inherits:
Object
  • Object
show all
Includes:
ClientSideEncryption
Defined in:
lib/shrine/storage/s3.rb

Defined Under Namespace

Modules: ClientSideEncryption

Constant Summary collapse

MAX_MULTIPART_PARTS = 
10_000
MIN_PART_SIZE = 
5*1024*1024
MULTIPART_THRESHOLD = 
{ upload: 15*1024*1024, copy: 100*1024*1024 }
COPY_OPTIONS = 
{ tagging_directive: "REPLACE" }

Instance Attribute Summary collapse

Attributes included from ClientSideEncryption

#encryption_client

Instance Method Summary collapse

Constructor Details

#initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, max_multipart_parts: nil, signer: nil, public: nil, copy_options: COPY_OPTIONS) ⇒ S3

Initializes a storage for uploading to S3. All options are forwarded to [‘Aws::S3::Client#initialize`], except the following:

:bucket : (Required). Name of the S3 bucket.

:client : By default an ‘Aws::S3::Client` instance is created internally from

additional options, but you can use this option to provide your own
client. This can be an `Aws::S3::Client` or an
`Aws::S3::Encryption::Client` object.

:prefix : “Directory” inside the bucket to store files into.

:upload_options : Additional options that will be used for uploading files, they will

be passed to [`Aws::S3::Object#put`], [`Aws::S3::Object#copy_from`]
and [`Aws::S3::Bucket#presigned_post`].

:copy_options : Additional options that will be used for copying files, they will

be passed to [`Aws::S3::Object#copy_from`].

:multipart_threshold : If the input file is larger than the specified size, a parallelized

multipart will be used for the upload/copy. Defaults to
`{upload: 15*1024*1024, copy: 100*1024*1024}` (15MB for upload
requests, 100MB for copy requests).

:max_multipart_parts : Limits the number of parts if parellized multipart upload/copy is used. Defaults to 10_000.

In addition to specifying the ‘:bucket`, you’ll also need to provide AWS credentials. The most common way is to provide them directly via ‘:access_key_id`, `:secret_access_key`, and `:region` options. But you can also use any other way of authentication specified in the [AWS SDK documentation][configuring AWS SDK].

[‘Aws::S3::Object#put`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method [`Aws::S3::Object#copy_from`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_from-instance_method [`Aws::S3::Bucket#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Client#initialize`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method [configuring AWS SDK]: docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html

Raises:

  • (ArgumentError) 


71
72
73
74
75
76
77
78
79
80
81
82
83
84
 
# File 'lib/shrine/storage/s3.rb', line 71

def initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, max_multipart_parts: nil, signer: nil, public: nil, copy_options: COPY_OPTIONS, **)
  raise ArgumentError, "the :bucket option is nil" unless bucket

  @client = client || Aws::S3::Client.new(**)
  @transfer_manager = Aws::S3::TransferManager.new(client: @client) if defined?(Aws::S3::TransferManager)
  @bucket = Aws::S3::Bucket.new(name: bucket, client: @client)
  @prefix = prefix
  @upload_options = upload_options
  @copy_options = copy_options
  @multipart_threshold = MULTIPART_THRESHOLD.merge(multipart_threshold)
  @max_multipart_parts = max_multipart_parts || MAX_MULTIPART_PARTS
  @signer = signer
  @public = public
end

Instance Attribute Details

#bucket ⇒ Object (readonly)

Returns the value of attribute bucket.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def bucket
  @bucket
end

#client ⇒ Object (readonly)

Returns the value of attribute client.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def client
  @client
end

#copy_options ⇒ Object (readonly)

Returns the value of attribute copy_options.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def copy_options
  @copy_options
end

#prefix ⇒ Object (readonly)

Returns the value of attribute prefix.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def prefix
  @prefix
end

#public ⇒ Object (readonly)

Returns the value of attribute public.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def public
  @public
end

#signer ⇒ Object (readonly)

Returns the value of attribute signer.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def signer
  @signer
end

#upload_options ⇒ Object (readonly)

Returns the value of attribute upload_options.



17
18
19
 
# File 'lib/shrine/storage/s3.rb', line 17

def upload_options
  @upload_options
end

Instance Method Details

#clear!(&block) ⇒ Object

If block is given, deletes all objects from the storage for which the block evaluates to true. Otherwise deletes all objects from the storage.

s3.clear!
# or
s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 }



219
220
221
222
223
224
 
# File 'lib/shrine/storage/s3.rb', line 219

def clear!(&block)
  objects_to_delete = bucket.objects(prefix: prefix)
  objects_to_delete = objects_to_delete.lazy.select(&block) if block

  delete_objects(objects_to_delete)
end

#delete(id) ⇒ Object

Deletes the file from the storage.



199
200
201
 
# File 'lib/shrine/storage/s3.rb', line 199

def delete(id)
  object(id).delete
end

#delete_prefixed(delete_prefix) ⇒ Object

Deletes objects at keys starting with the specified prefix.

s3.delete_prefixed("somekey/derivatives/")



206
207
208
209
210
211
 
# File 'lib/shrine/storage/s3.rb', line 206

def delete_prefixed(delete_prefix)
  # We need to make sure to combine with storage prefix, and
  # that it ends in '/' cause S3 can be squirrely about matching interior.
  delete_prefix = delete_prefix.chomp("/") + "/"
  bucket.objects(prefix: [*prefix, delete_prefix].join("/")).batch_delete!
end

#exists?(id) ⇒ Boolean

Returns true file exists on S3.

Returns:

  • (Boolean) 


128
129
130
 
# File 'lib/shrine/storage/s3.rb', line 128

def exists?(id)
  object(id).exists?
end

#object(id) ⇒ Object

Returns an ‘Aws::S3::Object` for the given id.



227
228
229
 
# File 'lib/shrine/storage/s3.rb', line 227

def object(id)
  bucket.object(object_key(id))
end

#open(id, rewindable: true, encoding: nil) ⇒ Object

Returns a ‘Down::ChunkedIO` object that downloads S3 object content on-demand. By default, read content will be cached onto disk so that it can be rewinded, but if you don’t need that you can pass ‘rewindable: false`. A required character encoding can be passed in `encoding`; the default is `Encoding::BINARY` via `Down::ChunkedIO`.

Any additional options are forwarded to [‘Aws::S3::Object#get`].

[‘Aws::S3::Object#get`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method



119
120
121
122
123
124
125
 
# File 'lib/shrine/storage/s3.rb', line 119

def open(id, rewindable: true, encoding: nil, **)
  chunks, length = get(id, **)

  Down::ChunkedIO.new(chunks: chunks, rewindable: rewindable, size: length, encoding: encoding)
rescue Aws::S3::Errors::NoSuchKey
  raise Shrine::FileNotFound, "file #{id.inspect} not found on storage"
end

#presign(id, method: :post, **presign_options) ⇒ Object

Returns URL, params, headers, and verb for direct uploads.

s3.presign("key") #=>
# {
#   url: "https://my-bucket.s3.amazonaws.com/...",
#   fields: { ... },  # blank for PUT presigns
#   headers: { ... }, # blank for POST presigns
#   method: "post",
# }

By default it calls [‘Aws::S3::Object#presigned_post`] which generates data for a POST request, but you can also specify `method: :put` for PUT uploads which calls [`Aws::S3::Object#presigned_url`].

s3.presign("key", method: :post) # for POST upload (default)
s3.presign("key", method: :put)  # for PUT upload

Any additional options are forwarded to the underlying AWS SDK method.

[‘Aws::S3::Object#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method



189
190
191
192
193
194
195
196
 
# File 'lib/shrine/storage/s3.rb', line 189

def presign(id, method: :post, **presign_options)
  options = {}
  options[:acl] = "public-read" if public

  options.merge!(@upload_options, presign_options)

  send(:"presign_#{method}", id, options)
end

#upload(io, id, shrine_metadata: {}, **upload_options) ⇒ Object

If the file is an UploadedFile from S3, issues a COPY command, otherwise uploads the file. For files larger than ‘:multipart_threshold` a multipart upload/copy will be used for better performance and more resilient uploads.

It assigns the correct “Content-Type” taken from the MIME type, because by default S3 sets everything to “application/octet-stream”.



93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/shrine/storage/s3.rb', line 93

def upload(io, id, shrine_metadata: {}, **upload_options)
  content_type, filename = shrine_metadata.values_at("mime_type", "filename")

  options = {}
  options[:content_type] = content_type if content_type
  options[:content_disposition] = ContentDisposition.inline(filename) if filename
  options[:acl] = "public-read" if public

  options.merge!(@upload_options, upload_options)

  if copyable?(io)
    copy(io, id, **options)
  else
    put(io, id, **options)
  end
end

#url(id, public: self.public, host: nil) ⇒ Object

Returns the presigned URL to the file.

:host : This option replaces the host part of the returned URL, and is

typically useful for setting CDN hosts (e.g.
`http://abc123.cloudfront.net`)

:public : Returns the unsigned URL to the S3 object. This requires the S3

object to be public.

All other options are forwarded to [‘Aws::S3::Object#presigned_url`] or [`Aws::S3::Object#public_url`].

[‘Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method [`Aws::S3::Object#public_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#public_url-instance_method



148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
 
# File 'lib/shrine/storage/s3.rb', line 148

def url(id, public: self.public, host: nil, **)
  if public || signer
    url = object(id).public_url(**)
  else
    url = object(id).presigned_url(:get, **)
  end

  if host
    uri = URI.parse(url)
    uri.path = uri.path.match(/^\/#{bucket.name}/).post_match unless uri.host.include?(bucket.name)
    url = URI.join(host, uri.request_uri[1..-1]).to_s
  end

  if signer
    url = signer.call(url, **)
  end

  url
end