Video Testing in Tokbox
Francisco Javier Cano Sandoval
Dimitrios Christodoulou
QA Engineers @ Tokbox
A Telefónica Company
What is Tokbox about?
- Live video-conferencing with WebRTC
- Archiving video-conferences
- Broadcast using a CDN
- Layout control in live broadcasts and archives
- Interactive broadcast (300+ users)
- Not a Request-Response protocol
- Non-deterministic:
- Dependant on the network conditions
- Codecs
- Heuristics used
- Objective vs Subjective Quality
Differences with other systems
Approach for Archiving (I)
OpenTok
Platform
Amazon
S3
Approach for Archiving (II)
- Was the archive created?
- Did it have the expected contents?
- Audio / Video
- Did it have a good quality?
- Bitrate
- Framerate
- Artifacts
Next iteration for Archiving
Quality Issues
Current iteration for Archiving
Approach for Broadcast (I)
Publisher stream
HQ stream
HLS streams
CDN
CloudFront
Clients
200kbps
500kbps
800kbps
Approach for Broadcast (II)
Mostly similar to the archiving approach
Next iteration for Broadcast (I)
- Key frames...
- ... at the beginning of fragments
Next iteration for Broadcast (II)
- Binary encoded frame number...
0x159
- Multiple streams delivery
0x16F
- ... for fragment alignment check
Current iteration for Broadcast
- Key frames
- Distance between any two key frames
- Average intra-frames distance
Layout Control (I)
- What are layouts?
OpenTok
Amazon S3
Layout Control (II)
- Checking video...
255
254
255
224
252
250
252
255
255
... using RGB values in pixels...
Layout Control (III)
- ... and audio
changing from the time domain
to the frequency domain
using the Fast Fourier Transform
Audio Check in Layout Control
1000Hz
2000Hz
3000Hz
Live video sessions (I)
- Functional tests
- Performance tests that stress the platform
- Without WebRTC (signalling, messaging)
- With WebRTC (media), collecting stats
Live video sessions (II)
- We don't have a file we can process offline...
OpenTok cloud
Live video sessions (III)
- ... but we do have WebRTC stats
- A/V Bitrate
- A/V Packet Loss
- Jitter
- PLIs
- NACKs
- RTT
Live video sessions (IV)
- We combine all of these data to determine the session quality for each Publisher or Subscriber
Soak tests
- Tests that put a sustained load on the platform for a long time (in the order of days)
Soak tests
In progress and Future Work (I)
- Resource monitoring
- but directly from the tests code, both in client and server, using dstat
In progress and Future Work (II)
- Use Ansible to leverage large scale testing and benchmarking
- Subscriber Quality in non-perfect networks (PL, delay, jitter, disordered packages...)
- Synthetic vs real video
- Capture and analyze media in live sessions
- Reference & No-reference algorithms
- Subjective Quality indicator (MOS)
- Correlate current and future metrics
- ...
MOS Measurement
network impairment
Tools and technologies used
- NodeJS 4.5+ / C++
- FFMpeg / ffprobe
- Extract a frame from a video file
- Obtain streams information from a video file
- Generate synthetic audio and video
- Extract frames information (resolution, size, index, duration...)
- Chrome WebRTC stats
- Live sessions tests
- dstat, htop
- Monitoring resources usage
- Proprietary:
- Neutrino.js, for performance tests
- TBMedia, for functional browser-like tests
Links
- https://wiki.linuxfoundation.org/networking/netem
-
https://docs.google.com/document/d/1D5rBFrKw4GocrEv10OApSV9zvOQd3-DwDXs1arrz1kU/edit#
Q&A
fran@tokbox.com
dimitri@tokbox.com
Thanks for your attention
Video Testing in Tokbox Francisco Javier Cano Sandoval Dimitrios Christodoulou QA Engineers @ Tokbox A Telefónica Company