Class: HLS::Uploader

Inherits:
Object
  • Object
show all
Defined in:
lib/hls/uploader.rb

Overview

Walks an encoded HLS bundle and pushes each file to the configured bucket. Idempotent and resumable: a state sidecar tracks per-file MD5 digests, and files whose remote upload matches the local digest are skipped.

Constant Summary collapse

CONTENT_TYPES = 
{
  ".m3u8" => "application/vnd.apple.mpegurl",
  ".ts"   => "video/MP2T",
  ".jpg"  => "image/jpeg",
  ".jpeg" => "image/jpeg",
  ".png"  => "image/png",
  ".vtt"  => "text/vtt"
}.freeze
CACHE_CONTROL_IMMUTABLE = 
"public, max-age=31536000, immutable"
CACHE_CONTROL_PLAYLIST =

VOD playlists are also immutable once written — segments don’t get rewritten, the playlist itself doesn’t change. Use a shorter max-age than the segments themselves so deploys can publish a superseding bundle, but allow CDN caching at the playlist edge.

"public, max-age=300"
DEFAULT_MAX_RETRIES =

Default retries on top of the AWS SDK’s own retry behavior. Bumped to absorb transient network errors during long, multi-object uploads where the SDK’s retry budget per call has been exhausted.

3
DEFAULT_INITIAL_BACKOFF = 
0.5
DEFAULT_CONCURRENCY =

Parallel upload workers. The SDK is thread-safe, S3 is throughput- bound on typical connections, and a 3-rendition bundle is dozens of small segments — overlapping their PUTs cuts wall-clock time significantly. 4 workers is a good default for home/office links; bump higher on a beefy server with a fat pipe.

4
TRANSIENT_ERRORS =

Errors classed as transient and worth retrying. We deliberately don’t include the broad Aws::Errors::ServiceError parent — a 403 or NoSuchBucket should fail fast, not retry.

[
  Seahorse::Client::NetworkingError,
  Aws::S3::Errors::RequestTimeout,
  Aws::S3::Errors::ServiceUnavailable,
  Aws::S3::Errors::SlowDown,
  Aws::S3::Errors::InternalError
].freeze

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(storage:, output:, key_prefix:, state:, max_retries: DEFAULT_MAX_RETRIES, initial_backoff: DEFAULT_INITIAL_BACKOFF, concurrency: DEFAULT_CONCURRENCY) ⇒ Uploader

Returns a new instance of Uploader.



60
61
62
63
64
65
66
67
68
69
70
71
72
 
# File 'lib/hls/uploader.rb', line 60

def initialize(storage:, output:, key_prefix:, state:,
               max_retries: DEFAULT_MAX_RETRIES,
               initial_backoff: DEFAULT_INITIAL_BACKOFF,
               concurrency: DEFAULT_CONCURRENCY)
  @storage = storage
  @output = Pathname.new(output)
  @key_prefix = key_prefix.to_s.sub(%r{\A/}, "").sub(%r{/\z}, "")
  @state = state
  @max_retries = max_retries
  @initial_backoff = initial_backoff
  @concurrency = [concurrency.to_i, 1].max
  @state_mutex = Mutex.new
end

Instance Attribute Details

#concurrency ⇒ Object (readonly)

Returns the value of attribute concurrency.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def concurrency
  @concurrency
end

#initial_backoff ⇒ Object (readonly)

Returns the value of attribute initial_backoff.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def initial_backoff
  @initial_backoff
end

#key_prefix ⇒ Object (readonly)

Returns the value of attribute key_prefix.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def key_prefix
  @key_prefix
end

#max_retries ⇒ Object (readonly)

Returns the value of attribute max_retries.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def max_retries
  @max_retries
end

#output ⇒ Object (readonly)

Returns the value of attribute output.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def output
  @output
end

#state ⇒ Object (readonly)

Returns the value of attribute state.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def state
  @state
end

#storage ⇒ Object (readonly)

Returns the value of attribute storage.



57
58
59
 
# File 'lib/hls/uploader.rb', line 57

def storage
  @storage
end

Instance Method Details

#perform ⇒ Object

Upload everything under the output directory that hasn’t been uploaded yet. Returns a hash with :uploaded and :skipped counts.



76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
 
# File 'lib/hls/uploader.rb', line 76

def perform
  pending = uploadable_files.filter_map do |file|
    relative_key = relative_key_for(file)
    digest = md5_of(file)
    next nil if @state_mutex.synchronize {
      state.uploaded?(relative_key: relative_key, digest: digest)
    }
    [file, relative_key, digest]
  end

  skipped = uploadable_files.size - pending.size
  uploaded = upload_in_parallel(pending)

  { uploaded: uploaded, skipped: skipped }
end