Video Testing at Tokbox

Approach for Archiving (II)

• Was the archive created?
• Did it have the expected contents?
  • Audio / Video
• Did it have a good quality?
  • Bitrate
  • Framerate
  • Artifacts

Next iteration for Archiving

Quality Issues

Current iteration for Archiving

Approach for Broadcast (II)

Mostly similar to the archiving approach

Next iteration for Broadcast (I)

• Key frames...

• ... at the beginning of fragments

Next iteration for Broadcast (II)

• Binary encoded frame number...

0x159

• Multiple streams delivery

0x16F

• ... for fragment alignment check

Current iteration for Broadcast

• Key frames
  • Distance between any two key frames
  • Average intra-frames distance

Layout Control (II)

255

254

255

224

252

250

252

255

255

Layout Control (III)

• ... and audio

changing from the time domain

to the frequency domain

using the Fast Fourier Transform

Audio Check in Layout Control

1000Hz

2000Hz

3000Hz

Live video sessions (II)

• We don't have a file we can process offline...

OpenTok cloud

Live video sessions (III)

• ... but we do have WebRTC stats

• A/V Bitrate
• A/V Packet Loss
• Jitter

• PLIs
• NACKs
• RTT

Live video sessions (IV)

• We combine all of these data to determine the session quality for each Publisher or Subscriber

Soak tests

In progress and Future Work (II)

• Use Ansible to leverage large scale testing and benchmarking
• Subscriber Quality in non-perfect networks (PL, delay, jitter, disordered packages...)
• Synthetic vs real video
• Capture and analyze media in live sessions
• Reference & No-reference algorithms
• Subjective Quality indicator (MOS)
• Correlate current and future metrics
• ...