WebRTCBench
Performance Assessment of WebRTC Implementations
Sajjad Taheri, Laleh Aghababaie Beni, Rosario Cammarota, Alexander Veidenbaum, Alexandru Nicolau, Jianlin Qiu, Qiang Lu and Mohammad Haghighat
Presented by: Tiago Mück
The Center for Embedded Systems and Cyber-Physical Systems, UC Irvine
Outline
- WebRTC Introduction
- WebRTCBench
- Design and implementation
- Measurements
- Evaluation
- Summary
WebRTC
- Real-time communication for the Web
- HTML5 Tags and JavaScript API
- Access media devices
- Establish peer-to-peer connections
- Transfer data and media contents
- Set of underlying protocols
- HTML5 Tags and JavaScript API
- Industry effort
- Supported by Chrome, Firefox, Opera
- Indivisual and enterprise users
Why a new Benchmark?
- There are plenty of JavaScript benchmarks
- WebRTCBench aims toward WebRTC performance
-
It focuses on performance of WebRTC functionalities
- particularly important at the current stage of standardization and development
- Provides insight to guide performance optimization
WebRTCBench In a Nutshell
- Portable benchmark
- Using JavaScript APIs without ever changing the browsers source code
- Quantitative comparison between current and future WebRTC implementations
- Supports most WebRTC functionalities
- MediaStream, RTCPeerConnection, RTCDataChannel
- Automated benchmarking
- Database integration
How it works
- A fully functional WebRTC application using HTML5 API
- Audio/Video calling and data transfer
- App is annotated to measure/collect and store the performance numbers
- A Web/Signaling server
- Serves the WebRTC app and does the signalling
WebRTCBench is built on top of a representative real world WebRTC System
How To Use: Web based user interface
Users navigate to the server address using browsers of choice to download the HTML5 app
- Provide device/peer information
- Set experiment types/parameters
- Connect to the other peers
- Perform the experiments
How to use: Automation scripts
- Automated Performance evaluations
- More convenient and reproducible evaluations
- Based on Selenium testing framework
- Launch browsers and execute different benchmarking scenarios
- Access the DOM elements to navigate to the server, schedule actions and collects performance numbers
- Desktop and Android support (Through Android Debug Bridge)
Implementation
- Server application is implemented using Node.js
- Socket.io for signalling
- WebRTC app is annotated
- events completion time are measured using
-
JavaScript accurate timers: "performance.now()"
-
-
Statistics are collected using
-
WebRTC statistics API
-
HTML5 video element API
-
- events completion time are measured using
-
Database adapter for MySQL is provided
Inputs and Constratins
- Different codecs can be considered
- VP8/VP9 and H264
- Input from different sources
- Live webcam stream
- Uncompressed videos
- WebRTC constraints are supported. e.g.
- Reliable/Unreliable data channel
- device selection or resolution
Measurements
- Performance of each RTC peer connection is evaluated
- Initialization and Connection establishment
- Media Streaming
- Data channel transfer
Measurements: Initialization and Connection Establishment
Based on RTC peer connection establishment flow
Measurements include:
- Initialize RTC data structures
- Getting user media
- Making offers/answers
- ICE hole punching
Measurements: Media Engine, Data channels
- Performed after successfull RTC peer connection establishment
-
Media engine measurements include:
- Encodeer/decoder framerate, bitrate, resolution
- Jitter, RTT
- Datachannel measurements
- Throughput and latency of user messages
Media engine pipeline: capturing/encoding/decoding at the same time
EVALUATION
- Evaluation is done on variety of devices, operating systems and browsers
- Demonstrating the effectiveness of the benchmark
Specification of evaluation devices
Initialization and Peer connection performance
Making offer/answer
instantiating an PeerConnection Object
Getting User Media
ICE hole punching
Encoder Rampup Over Time
- Encoders ramp up bitrate over time to avoid congestion
- Experiments are performed after
Encoder bitrate ramp-up for live video
MEDIA Engine Encoder/decoder performance
Decoder bitrate/framerate/resolution
Encoder bitrate/framerate/resolution
Video inputs
SUMMARY
- WebRTCBench is a portable benchmark
- Useful tool for industry and academia
- Necessary for standardization stage of WebRTC
- To detect performance related issues
- To guide performance optimization
Thank YOU!!
Questions:
email: sajjadt@uci.edu
Initialization performance
instantiating an PeerConnection Object
Getting/Playing User Media
connection establishment performance
Making offer/answer
ICE hole punching
DAtaChannel performance
sending user level packets
sending documents with different sizes
Encoder/DECODER Performance
- Uncompressed videos are used in this experiment
- Decoder test uses Chromium running on a powerful Desktop computer as encoder
Soccer
640*480
60FPS
Sintel
1280*720
24FPS
Encoder Performance
VP8 encoder resolution versus encoding/rendering framerate
DECODER Performance
VP8 decoder resolution versus encoding/rendering framerate
WebRTCBench Performance Assessment of WebRTC Implementations Sajjad Taheri, Laleh Aghababaie Beni, Rosario Cammarota, Alexander Veidenbaum, Alexandru Nicolau, Jianlin Qiu, Qiang Lu and Mohammad Haghighat Presented by: Tiago Mück The Center for Embedded Systems and Cyber-Physical Systems, UC Irvine
WebRTCBench: Performance Assesment of WebRTC Implementations
By Sajjad Taheri
WebRTCBench: Performance Assesment of WebRTC Implementations
- 591
