Class: ActiveEncode::EngineAdapters::MediaConvertAdapter

Inherits:
Object
  • Object
show all
Defined in:
lib/active_encode/engine_adapters/media_convert_adapter.rb

Overview

An adapter for using [AWS Elemental MediaConvert](aws.amazon.com/mediaconvert/) to encode.

Note: this adapter does not perform input characterization, does not provide technical metadata on inputs.

## Configuration

ActiveEncode::Base.engine_adapter = :media_convert

ActiveEncode::Base.engine_adapter.role = 'arn:aws:iam::123456789012:role/service-role/MediaConvert_Default_Role'
ActiveEncode::Base.engine_adapter.output_bucket = 'output-bucket'

# optionally and probably not needed

ActiveEncode::Base.engine_adapter.queue = my_mediaconvert_queue_name
ActiveEncode::Base.engine_adapter.log_group = my_log_group_name

## Capturing output information

[AWS Elemental MediaConvert](aws.amazon.com/mediaconvert/) doesn’t provide detailed output information in the job description that can be pulled directly from the service. Instead, it provides that information along with the job status notification when the job status changes to ‘COMPLETE`. The only way to capture that notification is through an [Amazon Eventbridge](aws.amazon.com/eventbridge/) rule that forwards the a MediaWatch job status change on `COMPLETE` to another service, such as [CloudWatch Logs] (aws.amazon.com/cloudwatch/) log group

This adapter is written to get output information from a CloudWatch log group that has had MediaWatch complete events forwarded to it by an EventBridge group. The ‘setup!` method can be used to create these for you, at conventional names the adapter will be default use.

ActiveEncode::Base.engine_adapter.setup!

OR There is functionality to get what we can directly from the job without requiring a CloudWatch log – this only works for HLS and file outputs at present. To opt-in, and not require CloudWatch logs:

ActiveEncode::Base.engine_adapter.direct_output_lookup = true

MediaConvert also provides a probe endpoint to extract technical metadata about a file stored in s3. This can be used to gather output metadata when using direct_output_lookup instead of having to look in the CloudWatch logs.

## Example

ActiveEncode::Base.engine_adapter = :media_convert
ActiveEncode::Base.engine_adapter.role = 'arn:aws:iam::123456789012:role/service-role/MediaConvert_Default_Role'
ActiveEncode::Base.engine_adapter.output_bucket = 'output-bucket'

ActiveEncode::Base.engine_adapter.setup!

encode = ActiveEncode::Base.create(
  "file://path/to/file.mp4",
  {
    masterfile_bucket: "name-of-my-masterfile_bucket"
    output_prefix: "path/to/output/base_name_of_outputs",
    use_original_url: true,
    outputs: [
      { preset: "my-hls-preset-high", modifier: "_high" },
      { preset: "my-hls-preset-medium", modifier: "_medium" },
      { preset: "my-hls-preset-low", modifier: "_low" },
    ]
  }
)

## More info

A more detailed guide is available in the repo at [guides/media_convert_adapter.md](../../../guides/media_convert_adapter.md)

Defined Under Namespace

Classes: ResultsNotAvailable, RetriableClient

Constant Summary collapse

JOB_STATES = 
{
  "SUBMITTED" => :running, "PROGRESSING" => :running, "CANCELED" => :cancelled,
  "ERROR" => :failed, "COMPLETE" => :completed
}.freeze
OUTPUT_GROUP_TEMPLATES = 
{
  hls: { min_segment_length: 0, segment_control: "SEGMENTED_FILES", segment_length: 10 },
  dash_iso: { fragment_length: 2, segment_control: "SEGMENTED_FILES", segment_length: 30 },
  file: {},
  ms_smooth: { fragment_length: 2 },
  cmaf: { fragment_length: 2, segment_control: "SEGMENTED_FILES", segment_length: 10 }
}.freeze
SETUP_LOG_GROUP_RETENTION_DAYS = 
3

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#direct_output_lookup ⇒ Object

Returns the value of attribute direct_output_lookup.



136
137
138
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

def direct_output_lookup
  @direct_output_lookup
end

#direct_output_lookup if true, do NOT get output information from cloudwatch, ⇒ Object

instead retrieve and construct it only from job itself. Currently working only for HLS output. (default: false)



136
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

attr_accessor :role, :output_bucket, :direct_output_lookup, :use_probe

#log_group ⇒ Object 



283
284
285
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 283

def log_group
  @log_group ||= "/aws/events/active-encode/mediaconvert/#{queue}"
end

#log_group log_group_name that is being used to capture output=(value) ⇒ Object (writeonly) 



145
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

attr_writer :log_group, :queue, :output_id_format, :output_label_format

#output_bucket ⇒ Object

Returns the value of attribute output_bucket.



136
137
138
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

def output_bucket
  @output_bucket
end

#output_bucket simple bucket name to write output to(simplebucketnametowriteoutputto) ⇒ Object 



136
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

attr_accessor :role, :output_bucket, :direct_output_lookup, :use_probe

#output_id_format sprintf format for output ids (default: "%{job_id}-output%{suffix}")=(value) ⇒ Object (writeonly)

Available string variables are job_id and the keys in the MediaConvertOutput tech meatdata hash



145
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

attr_writer :log_group, :queue, :output_id_format, :output_label_format

#output_id_format=(value) ⇒ Object

Sets the attribute output_id_format

Parameters:

  • value —

    the value to set the attribute output_id_format to.



145
146
147
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

def output_id_format=(value)
  @output_id_format = value
end

#output_label_format sprintf format for output ids (default: "%{basename}" if output_url is preset=(value) ⇒ Object (writeonly)

otherwise “%ActiveEncode::EngineAdapters::MediaConvertAdapter.suffix”) Available string variables are job_id, basename, and the keys in the MediaConvertOutput tech metadata hash



145
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

attr_writer :log_group, :queue, :output_id_format, :output_label_format

#output_label_format=(value) ⇒ Object

Sets the attribute output_label_format

Parameters:

  • value —

    the value to set the attribute output_label_format to.



145
146
147
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

def output_label_format=(value)
  @output_label_format = value
end

#queue ⇒ Object 



287
288
289
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 287

def queue
  @queue ||= "Default"
end

#queue name of MediaConvert queue to use.=(value) ⇒ Object (writeonly) 



145
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 145

attr_writer :log_group, :queue, :output_id_format, :output_label_format

#role ⇒ Object

Returns the value of attribute role.



136
137
138
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

def role
  @role
end

#role simple name of AWS role to pass to MediaConvert, eg `my-role-name`(simplenameofAWSroletopasstoMediaConvert, eg`my-role-name`) ⇒ Object 



136
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

attr_accessor :role, :output_bucket, :direct_output_lookup, :use_probe

#use_probe ⇒ Object

Returns the value of attribute use_probe.



136
137
138
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

def use_probe
  @use_probe
end

#use_probe if true use probe endpoint to get technical metadata about input and outputs (default: false) ⇒ Object 



136
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 136

attr_accessor :role, :output_bucket, :direct_output_lookup, :use_probe

Instance Method Details

#cancel(id) ⇒ Object 



278
279
280
281
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 278

def cancel(id)
  mediaconvert.cancel_job(id: id)
  find(id)
end

#create(input_url, options = {}) ⇒ Object

Required options:

  • ‘output_prefix`: The S3 key prefix to use as the base for all outputs. Will be

    combined with configured `output_bucket` to be passed to MediaConvert
`destination`. Alternately see `destination` arg; one or the other
is required.

  • ‘destination`: The full s3:// URL to be passed to MediaConvert `destination` as output

    location an filename base.  `output_bucket` config is ignored if you
pass `destination`. Alternately see `output_prefix` arg; one or the
other is required.

  • ‘outputs`: An array of `modifier` options defining how to transcode and

    name the outputs. The "modifier" option will be passed as `name_modifier`
to AWS, to be added as a suffix on to `output_prefix` to create the
filenames for each output.

Optional options:

  • ‘masterfile_bucket`: All input will first be copied to this bucket, before being passed

    to MediaConvert. You can skip this copy by passing `use_original_url`
option, and an S3-based input. `masterfile_bucket` **is** required
unless use_original_url is true and an S3 input source.

  • ‘use_original_url`: If `true`, any S3 URL passed in as input will be passed directly to

    MediaConvert as the file input instead of copying the source to
the `masterfile_bucket`.

  • ‘media_type`: `audio` or `video`. Default `video`. Triggers use of a correspoinding

    template for arguments sent to AWS create_job API.

  • ‘output_type`: One of: `hls`, `dash_iso`, `file`, `ms_smooth`, `cmaf`. Default `hls`.

    Triggers use of a corresponding template for arguments sent to AWS
create_job API.

  • ‘output_group_destination_settings`: A hash of additional `destination_settings` to be

    sent to MediaConvert with the output_group.  Can include `s3_settings` key
with  `access_control` and `encryption` settings. See examples at:
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/MediaConvert/Client.html#create_job-instance_method

Example: 

output_prefix: "path/to/output/files",
outputs: [
    {preset: "System-Avc_16x9_1080p_29_97fps_8500kbps", modifier: "-1080",
    "System-Avc_16x9_720p_29_97fps_5000kbps", modifier: "-720",
    "System-Avc_16x9_540p_29_97fps_3500kbps", modifier: "-540"
  ]
}

}



251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 251

def create(input_url, options = {})
  input_url = s3_uri(input_url, options)

  input = options[:media_type] == :audio ? make_audio_input(input_url) : make_video_input(input_url)

  create_job_params = {
    queue: queue,
    role: role,
    settings: {
      inputs: [input],
      output_groups: make_output_groups(options)
    }
  }

  response = mediaconvert.create_job(create_job_params)
  job = response.job
  build_encode(job)
end

#find(id, _opts = {}) ⇒ Object 



270
271
272
273
274
275
276
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 270

def find(id, _opts = {})
  response = mediaconvert.get_job(id: id)
  job = response.job
  build_encode(job)
rescue Aws::MediaConvert::Errors::NotFound
  raise ActiveEncode::NotFound, "Job #{id} not found"
end

#setup! ⇒ Object

Creates a [CloudWatch Logs] (aws.amazon.com/cloudwatch/) log group and an EventBridge rule to forward status change notifications to the log group, to catch result information from MediaConvert jobs.

Will use the configured ‘queue` and `log_group` values.

The active AWS user/role when calling the ‘#setup!` method will require permissions to create the necessary CloudWatch and EventBridge resources

This method chooses a conventional name for the EventBridge rule, if a rule by that name already exists, it will silently exit. So this method can be called in a boot process, to check if this infrastructure already exists, and create it only if it does not.



159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
 
# File 'lib/active_encode/engine_adapters/media_convert_adapter.rb', line 159

def setup!
  rule_name = "active-encode-mediaconvert-#{queue}"
  return true if event_rule_exists?(rule_name)

  queue_arn = mediaconvert.get_queue(name: queue).queue.arn

  event_pattern = {
    source: ["aws.mediaconvert"],
    "detail-type": ["MediaConvert Job State Change"],
    detail: {
      queue: [queue_arn],
      status: ["COMPLETE"]
    }
  }

  # AWS is inconsistent about whether a cloudwatch ARN has :* appended
  # to the end, and we need to make sure it doesn't in the rule target.
  log_group_arn = create_log_group(log_group).arn.chomp(":*")

  cloudwatch_events.put_rule(
    name: rule_name,
    event_pattern: event_pattern.to_json,
    state: "ENABLED",
    description: "Forward MediaConvert job state changes on COMPLETE from queue #{queue} to #{log_group}"
  )

  cloudwatch_events.put_targets(
    rule: rule_name,
    targets: [
      {
        id: "Id#{SecureRandom.uuid}",
        arn: log_group_arn
      }
    ]
  )
  true
end