Orchestrating video transcoding in Ruby: A story

Michał Matyas

@nerdblogpl

https://nerdblog.pl

You can watch the slides live on your phone/laptop

https://tiny.cc/transcoding

I will avoid talking about the business side of the project.

Please don't ask, thank you!

Introduction

There was once a project...

Lesson #1

Hindsight 20/20

It's easy to look back and think "this should've been obvious" but you usually lack all the knowledge and experience at the time.

class Video < ActiveRecord::Base

mount_uploader VideoUploader

process_in_background UploadWorker

before_save

before_processing

Video.status = processing

carrierwave-video (ffmpeg)

after_processing_success

after_processing_failure

Video.status = ready

Video.status = failed

CALLBACK ALL THINGS!!!

(please don't)

Lesson #2

Avoid callbacks

Callbacks make your code harder to reason and harder to isolate, slowly turning everything into a tightly coupled co-dependent mess.

Carrierwave versions

include CarrierWave::Video
include ::CarrierWave::Backgrounder::Delay

storage :file

version :mp4 do
  process :encode_video => [:mp4]
end

version :webm do
  process :encode_video => [:webm]
end

Carrierwave versions

support_format :mp4_1080p, {
  resolution: '1920x1080',
  progress: :on_progress_1080p,
  if: :allow_1080p?
}

support_format :mp4_720p, {
  resolution: '1920x1080',
  progress: :on_progress_720p
}

support_format :mp4_480p, {
  resolution: '852x480',
  custom: '-preset ultrafast -g 5', 
  progress: :on_progress_480p
}

Lesson #3

Don't use Carrierwave versions

Or at least don't use them for anything more complex. They are good for other things (probably).

Consider using Shrine instead.

Video

kind: "720p"

Version

kind: "480p"

Version

kind: nil

Version

kind: "720p"
special: true

Version

kind: "480p"
special: true

Version

kind: nil
special: true

Version

class Video < ActiveRecord::Base

mount_uploader VideoUploader

process_in_background UploadWorker

before_save

before_processing

Video.status = processing

carrierwave-video (ffmpeg)

after_processing_success

after_processing_failure

Video.status = ready

Video.status = failed

class Version < ActiveRecord::Base

mount_uploader VersionUploader

process_in_background VersionUploadWorker

before_save

before_processing

Version.status = processing

carrierwave-video (ffmpeg)

after_processing_success

after_processing_failure

Version.status = ready

Version.status = failed

class Version < ActiveRecord::Base

mount_uploader VersionUploader

process_in_background VersionUploadWorker

before_save

before_processing

Version.status = processing

carrierwave-video (ffmpeg)

after_processing_success

after_processing_failure

Version.status = ready

Version.status = failed

class Version < ActiveRecord::Base

mount_uploader VersionUploader

process_in_background VersionUploadWorker

before_save

before_processing

Version.status = processing

carrierwave-video (ffmpeg)

after_processing_success

after_processing_failure

Version.status = ready

Version.status = failed

class Video < ActiveRecord::Base

Video.status = processing

Video.status = ready

Video

original version gets uploaded

ProcessingWorker

checks for existing versions

store in local cache

creates missing ones

starts processing based on the priority
(low quality video - higher priority because it's gonna be ready fastest)

uploads to S3
(in a separate worker once the version is ready)

processing uses streamio-ffmpeg directly to pass custom arguments to ffmpeg based on the video format and allows transcoding from other versions rather than original file (much much faster)

Lesson #4

Simplest solutions are often best solutions

We would have avoided a lot of pain if we didn't try to do things The Rails Way™ but went with the simplest solution instead.

You live and learn!

But why didn't you use...

AWS Lambda - 15 minute of max execution time
Zencoder, Amazon Elastic Transcoder etc. - expensive
Docker - not that popular (and well supported in production) at the time

Let's talk about MP4

FTYP - File identification
MDAT - media data (streams)
MOOV - metadata

Tip #1

Copy streams!

When transcoding things, the best approach is to copy streams and match codecs as much as possible because it's way way faster.

MOOV IS MADE OF ATOMS

Fixed parameters of the file
Specific pointers for each frame of data and audio

MOOV IS MADE OF ATOMS

TRKH - file times, duration, speed etc.
MDIA - Media Data
STCO - start of each video/audio chunk
STSC - Sample to chunk atom
STSZ - Sample size atom

Tip #2

Optimize for streaming

ffmpeg has arguments that make video better at streaming.
The most important:
-movflags +faststart

Tip #3

Bring Your Own Arguments

When using streamio-ffmpeg just ignore all the DSL they have and go for direct passing of as many arguments as you can.

Tip #4

There are no universal solutions

When transcoding videos from network, sometimes it's faster to download-then-process, sometimes to process on the fly. It heavily depends on the codec though.

Tip #5
Use presets

ffmpeg already has presets for most common configuration options but it's a balance between speed and quality. Use fast presets for low resolution videos.

Tip #6
Use profiles

H.264 has profiles (sets of codec features) but not every device will support all of them.

Go with the highest one you can afford.

Tip #7
Always convert to YUV420

There are different pixel formats that define how color information is stored. Use -pix_fmt yuv420 for max compatibility in browsers.

Tip #8
Collect metadata

It is always a good idea to collect metadata from ffprobe for both the incoming video and processed versions to look for clues when things break

Tip #9
Trust but verify

Sometimes file can be cut during transcoding and still technically be "valid", even if incomplete.

BUT...

Tip #10
Trust but verify manually

Be super-careful about invalidating the video or audio based on the duration given from ffprobe. It's sometimes approximated without a warning.

That's it, folks!

@nerdblogpl

https://nerdblog.pl

sources:

https://trac.ffmpeg.org/wiki/Encode/H.264
https://www.dfrws.org/sites/default/files/session-files/pres-video_authentication_using_file_structure_and_metadata.pdf
http://xhelmboyx.tripod.com/formats/mp4-layout.txt
https://www.cnwrecovery.com/manual/MP4FileStructure.html

Orchestrating video transcoding in Ruby: A story

You can watch the slides live on your phone/laptop

I will avoid talking about the business side of the project.

Please don't ask, thank you!

Introduction

There was once a project...

Lesson #1

Hindsight 20/20

It's easy to look back and think "this should've been obvious" but you usually lack all the knowledge and experience at the time.

Lesson #2

Avoid callbacks

Callbacks make your code harder to reason and harder to isolate, slowly turning everything into a tightly coupled co-dependent mess.

Carrierwave versions

Carrierwave versions

Lesson #3

Don't use Carrierwave versions

Or at least don't use them for anything more complex. They are good for other things (probably).

Lesson #4

Simplest solutions are often best solutions

We would have avoided a lot of pain if we didn't try to do things The Rails Way™ but went with the simplest solution instead.

You live and learn!

But why didn't you use...

Let's talk about MP4

Tip #1

Copy streams!

When transcoding things, the best approach is to copy streams and match codecs as much as possible because it's way way faster.

Tip #2

Optimize for streaming

ffmpeg has arguments that make video better at streaming. ​The most important: -movflags +faststart

Tip #3

Bring Your Own Arguments

When using streamio-ffmpeg just ignore all the DSL they have and go for direct passing of as many arguments as you can.

Tip #4

There are no universal solutions

When transcoding videos from network, sometimes it's faster to download-then-process, sometimes to process on the fly. It heavily depends on the codec though.

Tip #5 Use presets

ffmpeg already has presets for most common configuration options but it's a balance between speed and quality. Use fast presets for low resolution videos.

Tip #6 Use profiles

H.264 has profiles (sets of codec features) but not every device will support all of them.

Go with the highest one you can afford.

Tip #7 Always convert to YUV420

There are different pixel formats that define how color information is stored. Use -pix_fmt yuv420 for max compatibility in browsers.

Tip #8 Collect metadata

It is always a good idea to collect metadata from ffprobe for both the incoming video and processed versions to look for clues when things break

Tip #9 Trust but verify

Sometimes file can be cut during transcoding and still technically be "valid", even if incomplete.

Tip #10 Trust but verify manually

Be super-careful about invalidating the video or audio based on the duration given from ffprobe. It's sometimes approximated without a warning.

That's it, folks!

ffmpeg has arguments that make video better at streaming.
The most important:
-movflags +faststart

Tip #5
Use presets

Tip #6
Use profiles

Tip #7
Always convert to YUV420

Tip #8
Collect metadata

Tip #9
Trust but verify

Tip #10
Trust but verify manually