WebRTCBench

Performance Assessment of WebRTC Implementations

Sajjad Taheri, Laleh Aghababaie Beni, Rosario Cammarota, Alexander Veidenbaum, Alexandru Nicolau, Jianlin Qiu, Qiang Lu and Mohammad Haghighat

Presented by: Tiago Mück

The Center for Embedded Systems and Cyber-Physical Systems, UC Irvine

Outline

• WebRTC Introduction
• WebRTCBench
  • Design and implementation
  • Measurements
• Evaluation
• Summary

WebRTC

• Real-time communication for the Web
  • HTML5 Tags and JavaScript API
    • Access media devices
    • Establish peer-to-peer connections
    • Transfer data and media contents
  • Set of underlying protocols
• Industry effort
  • Supported by Chrome, Firefox, Opera
• Indivisual and enterprise users

Why a new Benchmark?

• There are plenty of JavaScript benchmarks
• WebRTCBench aims toward WebRTC performance
• It focuses on performance of WebRTC functionalities
  • particularly important at the current stage of standardization and development
  • Provides insight to guide performance optimization

WebRTCBench In a Nutshell

1. Portable benchmark
   • Using JavaScript APIs without ever changing the browsers source code
   • Quantitative comparison between current and future WebRTC implementations
2.  Supports most WebRTC functionalities
   • MediaStream, RTCPeerConnection, RTCDataChannel
3. Automated benchmarking
4. Database integration

How it works

• A fully functional WebRTC application using HTML5 API
  • Audio/Video calling and data transfer
  • App is annotated to measure/collect and store the performance numbers
• A Web/Signaling server
  • Serves the WebRTC app and does the signalling

WebRTCBench is built on top of a representative real world WebRTC System

How To Use: Web based user interface

Users navigate to the server address using browsers of choice to download the HTML5 app

• Provide device/peer information
• Set experiment types/parameters
• Connect to the other peers
• Perform the experiments

How to use: Automation scripts

• Automated Performance evaluations
• More convenient and reproducible evaluations
• Based on Selenium testing framework
  • Launch browsers and execute different benchmarking scenarios
  • Access the DOM elements to navigate to the server, schedule actions and collects performance numbers
• Desktop and Android support (Through Android Debug Bridge)

Implementation

• Server application is implemented using Node.js
  • Socket.io for signalling
• WebRTC app is annotated
  • events completion time are measured using

    • JavaScript accurate timers: "performance.now()"

  • Statistics are collected using

    • WebRTC statistics API

    • HTML5 video element API

• Database adapter for MySQL is provided

Inputs and Constratins

• Different codecs can be considered
  • VP8/VP9 and H264
• Input from different sources
  • Live webcam stream
  • Uncompressed videos

• WebRTC constraints are supported. e.g.
  • Reliable/Unreliable data channel
  • device selection or resolution

Measurements

• Performance of each RTC peer connection is evaluated
  • Initialization and Connection establishment
  • Media Streaming
  • Data channel transfer

Measurements: Initialization and Connection Establishment

Based on RTC peer connection establishment flow

Measurements include:

• Initialize RTC data structures
• Getting user media
• Making offers/answers
• ICE hole punching

Measurements: Media Engine, Data channels

• Performed after successfull RTC peer connection establishment
• Media engine measurements include:
  • Encodeer/decoder framerate, bitrate, resolution
  • Jitter, RTT    
• Datachannel measurements
  • Throughput and latency of user messages

Media engine pipeline: capturing/encoding/decoding at the same time

EVALUATION

• Evaluation is done on variety of devices, operating systems and browsers
• Demonstrating the effectiveness of the benchmark

Specification of evaluation devices

Initialization and Peer connection performance

Making offer/answer

instantiating an PeerConnection Object

Getting User Media

ICE hole punching

Encoder Rampup Over Time

• Encoders ramp up bitrate over time to avoid congestion
• Experiments are performed after

Encoder bitrate ramp-up for live video

MEDIA Engine Encoder/decoder performance

Decoder bitrate/framerate/resolution

Encoder bitrate/framerate/resolution

Video inputs

SUMMARY

• WebRTCBench is a portable benchmark
• Useful tool for industry and academia
• Necessary for standardization stage of WebRTC
  • To detect performance related issues
  • To guide performance optimization

Thank YOU!!

Questions:

email: sajjadt@uci.edu

Initialization performance

instantiating an PeerConnection Object

Getting/Playing User Media

connection establishment performance

Making offer/answer

ICE hole punching

DAtaChannel performance

sending user level packets

sending documents with different sizes

Encoder/DECODER Performance

• Uncompressed videos are used in this experiment
• Decoder test uses Chromium running on a powerful Desktop computer as encoder

Soccer

640*480

60FPS

Sintel

1280*720

24FPS

Encoder Performance

VP8 encoder resolution versus encoding/rendering framerate

DECODER Performance

VP8 decoder resolution versus encoding/rendering framerate