Approach for Archiving (I)

OpenTok

Platform

Amazon

S3

Approach for Archiving (II)

• Was the archive created?
• Did it have the expected contents?
  • Audio / Video
• Did it have a good quality?
  • Bitrate
  • Framerate
  • Artifacts

Next iteration for Archiving

Quality Issues

Current iteration for Archiving

Approach for Broadcast (I)

Publisher stream

HQ stream

HLS streams

CDN

CloudFront

Clients

200kbps

500kbps

800kbps

Approach for Broadcast (II)

Mostly similar to the archiving approach

Next iteration for Broadcast (I)

• Key frames...

• ... at the beginning of fragments

Next iteration for Broadcast (II)

• Binary encoded frame number...

0x159

• Multiple streams delivery

0x16F

• ... for fragment alignment check

Current iteration for Broadcast

• Key frames
  • Distance between any two key frames
  • Average intra-frames distance

Layout Control (I)

• What are layouts?

OpenTok

Amazon S3

Layout Control (II)

• Checking video...

255

254

255

224

252

250

252

255

255

... using RGB values in pixels...

Layout Control (III)

• ... and audio

changing from the time domain

to the frequency domain

using the Fast Fourier Transform

Audio Check in Layout Control

1000Hz

2000Hz

3000Hz

Live video sessions (I)

• Functional tests
• Performance tests that stress the platform
  • Without WebRTC (signalling, messaging)
  • With WebRTC (media), collecting stats

Live video sessions (II)

• We don't have a file we can process offline...

OpenTok cloud

Live video sessions (III)

• ... but we do have WebRTC stats

• A/V Bitrate
• A/V Packet Loss
• Jitter

• PLIs
• NACKs
• RTT

Live video sessions (IV)

• We combine all of these data to determine the session quality for each Publisher or Subscriber

Soak tests

• Tests that put a sustained load on the platform for a long time (in the order of days)

Soak tests

In progress and Future Work (I)

• Resource monitoring

• but directly from the tests code, both in client and server, using dstat

In progress and Future Work (II)

• Use Ansible to leverage large scale testing and benchmarking
• Subscriber Quality in non-perfect networks (PL, delay, jitter, disordered packages...)
• Synthetic vs real video
• Capture and analyze media in live sessions
• Reference & No-reference algorithms
• Subjective Quality indicator (MOS)
• Correlate current and future metrics
• ...