Brief intro to WebRTC

$ whoami

Mischa @z0r1k Lieibenson

So what is WebRTC?

WebRTC is a free, open project that provides browsers and mobile applications with Real-Time Communications (RTC) capabilities.

WebRTC is used to deliver audio, video, screen- and file- sharing in browser between peers.

Is WebRTC ready yet?

iswebrtcreadyyet.com

IE and Safari support is possible with Temasys plugin

• Media device handling and engines
• Codec implementations
• Media transport layer built on proven protocols
• Reliable, realtime, bidirectional data channels

What's inside

Slide from Michael Froehlich

WebRTC media transport

RTP, RTCP and STUN multiplexed on one connection

• RTP streams used to transport media

• RTCP streams used to provide transport feedback (e.g. PLI)

• STUN used to authenticate
  (Used to exchange credentials negotiated in offer/answer)

• STUN pings
  (Transmitted every 500ms)

• RTP, RTCP are symmetric to keep NAT traversal pinholes open

Slide from Michael Froehlich

Some SDP and a lot of asynchronous events

Typical WebRTC flow

• Offer created by one party

• Answered by the other

• Async IP candidate gathering

• Async connection establishment

Slide from Michael Froehlich

WebRTC Offers & Answers

v=0
o=- 3709720962883168841 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio
a=msid-semantic: WMS OwFf4ndJLqJTV8yAucuMC11MIsnKgmz5HrSH
m=audio 49997 RTP/SAVPF 111 103 104 0 8 106 105 13 126
c=IN IP4 54.72.232.91
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:wG4d9cZxWJe+1/J3
...
        
        

v=0
o=- 120247245346114869 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio
a=msid-semantic: WMS 074F5EBE-9D32-C14B-84B95E6802A10EB0
m=audio 45001 RTP/SAVPF 103
c=IN IP4 10.39.32.26
a=rtcp:1 IN IP4 0.0.0.0
a=sendrecv
a=ssrc:3540656010 cname:kbjGX1Qis4+ZyDHn
...

Slide from Michael Froehlich

Let's talk about JS API

In many slides I provide Chrome only example with "webkit" vendor prefix. 

For firefox please use "moz" prefix instead.

getUserMedia

navigator.webkitGetUserMedia({

    audio: true,
    video: true

}, function(stream){     

    // handle stream
    var url = URL.createObjectURL(stream); 
    // url is like blob:https%3A//localhost%3A3443/b4d040dd-633c-4bb1-9fb9-6fd98f0dca43
    // get video element and set src-attribute using url from above
    // add some filters or something...

}, function(err){        

   // handle errors
   console.error("Oops", err);

});

Promise based API is coming.

getUserMedia

Triggers Mic/Cam access dialog to ask user about permissions.

It happens per constraint. 
If you request first audio and then video it will be two dialogs.

If at the same time then only one combined dialog.

If your web app is served via HTTP getUserMedia will trigger access dialog on every call.

If your web app is served via HTTPS then user response will be persisted. 

Device management

MediaStreamTrack.getSources(function(devices){
    // filter your devices if you want only audio or video
    // because device enumeration gives you both
    console.log(devices);
});

// device example
// SourceInfo
// {
//     facing: "",
//     id: "b6ab4c86bb2cb76b14535fa61fd2414e3f7b54325824a3da8915c7c508b05642",
//     kind: "video",
//     label: "HD Pro Webcam C920 (046d:082d)"
// }

Chrome only!

If user didn't granted permissions before "label" field will be empty.
In case of HTTP label will be empty in any case.

id is unique and generated new on every session.

Seting input device

navigator.webkitGetUserMedia({
    
    audio: {
        optional: [
            {
                // device id
                sourceId: "b6ab4c86bb2cb76b14535fa61fd2414e3"
            }
        ]
    },
    video: true

}, function(stream){
    // handle stream
}, function(err){
    // handle errors
});

Seting video dimentions

navigator.webkitGetUserMedia({

    audio: true,
    video: {
        mandatory: {
            minWidth:  320,
            minHeight: 240,
            maxWidth:  1280,
            maxHeight: 720
        }
    }

}, function(stream){
    // handle stream
}, function(err){
    // handle errors
});

Please note that "mandatory" is an object and "optional" is an array of objects

Screen Sharing

navigator.webkitGetUserMedia({
    video: {
        mandatory: {
            maxWidth: 1280,
            maxHeight: 720,
            maxFrameRate: 3,
            chromeMediaSource: 'desktop',
            chromeMediaSourceId: id
        }
    }
}, callback, errCallback);

Captures desktops, windows, tabs.

chrome.desktopCapture.chooseDesktopMedia(['screen', 'window'], null, function(id){
    // returns a stream id
});

Install extension via JS Chrome API or inline HTML.

RTCPeerConnection

Peer Connection is your transport between peers.

new webkitRTCPeerConnection({
    iceServers: [{
        url: "stun:stun.l.google.com:19302"
    },{
        url: "turn:turn.citrixonline.com:5060",
        username: "Allice",
        credential: "LetMeIn"
    },{
        url: "turn:turn.citrixonline.com:443?transport=tcp",
        username: "Bob"
    }]
},{
    mandatory: {
        googIPv6: false
    },
    optional: [{
        DtlsSrtpKeyAgreement: true
    }]
});

RTCPeerConnection

Peer Connection is your transport between peers.

PeerConnection contains own signalling channel for STUN pings but not for handshake!

So offer/answer should be exchanged separately.

For example using HTTP or WebSocket.

STUN/TURN is used to poke holes in NAT.

Offer

var pc = new webkitRTCPeerConnection({iceServers: []});

pc.createOffer(function(offer){
    // send offer to remote peer
    // also you can set local description here
}, function(err){
    console.error(err);
},{
    mandatory: {
        OfferToReceiveAudio: true,
        OfferToReceiveVideo: true,
        iceRestart: false // not supported in Firefox
    }
});

// In Firefox constraints would look like this:
// {
//     offerToReceiveAudio: true,
//     offerToReceiveVideo: true
// }
// without additional "mandatory": {} wrapper

Offer

var pc = new webkitRTCPeerConnection({iceServers: []});

// For Firefox please use mozRTCSessionDescription
pc.setLocalDescription(new RTCSessionDescription(offer), 
    callback, errorCallback);

After setting local description Chrome assumes that connection will be established within 30 seconds.

If connection establishment takes longer it's not a big deal but it might cause video artefacts. 

So it is better to exchange offer answer beforehand.

Answer

var pc = new webkitRTCPeerConnection({iceServers: []});

// handle offer on "second" peer
function onOfferReceived(offer) {

    // offer should look like {"type":"offer","sdp":"..."}
    pc.setRemoteDescription(new RTCSessionDescription(offer), function(){
        pc.createAnswer(function(answer){
            pc.setLocalDescription(new RTCSessionDescription(answer), function(){
                
                // signal (send) answer back

            }, function(err){
                console.error(err);
            });
        }, function(err){
            console.error(err);
        });
    }, function(err){
        console.error(err);
    });
}

Offer-Answer

var pc = new webkitRTCPeerConnection({iceServers: []});

// handle answer on "first" peer
function onAnswerReceived(answer) {

    // answer should look like {"type":"offer","sdp":"..."}
    pc.setRemoteDescription(new RTCSessionDescription(answer), callback, errorCallback);

}

ICE Candidates

var pc = new webkitRTCPeerConnection({iceServers: []});

pc.onicecandidate = function(candidate){

    if (!candidate.candidate){
        // if candidate property is null then it was last candidate
        // and there is no need to signal it
        return;
    }

    // signal ice candidate to remote peer

};

function onRemoteIceCandidate(candidate){

    // in case of Firefox use mozRTCIceCandidate
    pc.addIceCandidate(new RTCIceCandidate(candidate));

}

Signalling / ICE states

var pc = new webkitRTCPeerConnection({iceServers: []});

// current state of signalling
console.log(pc.signalingState);

pc.onsignalingstatechange = function(state){ ... };

// current state of ice
console.log(pc.iceConnectionState);

pc.oniceconnectionstatechange = function(state){ ... };

ICE

• gathering / checking
• completed / connected
• failed / disconnected
• closed
• etc.

Signalling

• stable
• closed
• etc.

Streams

var pc = new webkitRTCPeerConnection({iceServers: []});

// add local stream (upstream) to local peer connection
pc.addStream(stream);

// remove local stream from local peer connection
pc.removeStream(stream);

// after each add/remove operation peer connection should renegotiate handshake
// you can do it either manually or based on peer connection callback
pc.onnegotiationneeded = function(){};

pc.onaddstream = function(stream){
    // remote peer added stream
};

pc.onremovestream = function(stream){
    // remote peer removed stream
};

Streams

One peer connection can have many streams.

In Chrome it is possible to add/remove streams at any time.

In Firefox all streams should be added before handshake.

In case of additional streams peer connection should be closed and created again.

Same streams can be added to multiple peer connections.

Data channel

var pc = new webkitRTCPeerConnection({iceServers: []});

var dataChannel = pc.createDataChannel("my-channel", {
    ordered: false, // do not guarantee order
    maxRetransmitTime: 3000, // ms    
});

dataChannel.onopen = function() {
    dataChannel.send("Hi.");
};

dataChannel.onmessage = function(msg) {
    console.log("New message received:", msg.data);
};

dataChannel.onerror = function(error){ ... };

dataChannel.onclose = function(){ ... };

Very similar to WebSocket API.

Closing Peer Connection

var pc = new webkitRTCPeerConnection({iceServers: []});

// remote peer will be notified via signalling and ice state changes
pc.close();

Connection strategies

In p2p world you connect everyone with everyone.

It works fine for 3-5 participants but not more than that.

So it is easier to get server involved. 

Each peer connects to server peer so for local peer it is 1:1 connection, but for server it is 1:n.

This way you keep signalling on client side easier and get better scalability. 

Available WebRTC relays

Many...

Here are some of them:

• FreeSwitch
• OpenTalk
• Jitsi Videobridge
• RTCGW by Citrix
  (not yet available to 3rd parties)

Available "p2p" servers

Many...

Here are some of them:

• Firebase
• PubNub
• Pusher
• Any pubsub server you like
  or can build with whatever

Some handy tools

• chrome://webrtc-internals
• apprtc.appspot.com

• test.webrtc.org

Some showcases

• free.gotomeeting.com (GoToMeeting Free)

• app.gotomeeting.com (GoToMeeting Pro)

• talky.io

• UberConference

• Twilio

• etc.

GoToMeeting Free

GoToMeeting Free

GoToMeeting Pro

Questions?

Thank you!

Tweet me at @z0r1k