HTML5Rocks@SG

Using  HTML5 audio & video elements

. Audio & Video APIs

. Features functionnalities : 

Media controls

Media properties

Basic HTML5 media events: Buffering, seeking & time range

. What about media streaming ?

. Error handling

. Annexes & Tools 

. What's more ?

Audio & Video APIs

Audio & video media are embedding fairly simply as following:

<audio src="audio.ogg" controls autoplay loop>
<p> Jimi Hendrix, Little wing </p>
</audio> 

<video src="http://v2v.sg/~j/theora_testsuite/pink-pigs.ogg" controls type="video/ogg; codecs=dirac, speex">
  Pink Floyd, Pigs
// the following lines takes into account fallback options

   <object data="flvplayer.swf" type="application/x-shockwave-flash">
      <param value="flvplayer.swf" name="movie"/>
    </object>
</video>

. controls : Displays the standard HTML5 controls for the audio on the web page
. autoplay : Makes the audio play automatically
. loop : Make the audio repeat (loop) automatically
. preload: Used in the audio element for buffering large files

. autobuffer: 

. poster // which specific to <video> tag

. type : specify which codec are required for the media

Here is important to take into consideration the MIME type as it is specified on both the client & server side, and generally the container and its associated formats & file extensions.

Features associated: 

Media attributes constitute the core of the features providing functionnalities such as:

Media controls

var v = document.getElementsByTagName("video")[0];
v.play();

<audio id="demo" src="audio.mp3"></audio>
<div>
  <button onclick="document.getElementById('demo').play()">Play the Audio</button>
  <button onclick="document.getElementById('demo').pause()">Pause the Audio</button>
  <button onclick="document.getElementById('demo').volume+=0.1">Increase Volume</button>
  <button onclick="document.getElementById('demo').volume-=0.1">Decrease Volume</button>
</div>

API methods defined by the spec:
.play() — plays the audio
.pause() — pauses the audio
.canPlayType() — interrogates the browser to establish whether the given mime type can be played
.buffered() — attribute that specifies the start and end time of the buffered part of the file

Features associated:

 media PROPERTIES

. currentTime : playhead position
. duration: media duration
. muted: is volume muted?
. paused: is media paused?

. volume: volume level

var audio = new Audio();
var duration = audio.duration;

Features associated:

BASIC HTML5 MEDIA EVENTS: Media BUFFERING, SEEKING, TiME RANGE :

Buffered  ATTRIBUTE

// returns TimeRanges object of buffered media
var buffered = audio.buffered;

// returns time in seconds of the last buffered TimeRange
var bufferedEnd = audio.buffered.end();

  TimeRanges OBJECT

    audio.buffered.length
audio.buffered.start(0)
audio.buffered.end(0)  

Seeking & seekable

    // Is the player currently seeking?
var isSeeking = audio.seeking;
    
// Is the media seekable?
var isSeekable = audio.seekable && audio.seekable.length > 0;
// Time in seconds within which the media is seekable.
var seekableEnd = audio.seekable.end(); 

WHAT ABOUT MEDIA STREAMING ?

For now media streaming is still a well-dominated features by licenced technologies such as RTMP protocol  by Adobe.
Nonetheless streaming is more and more the subject of open-source project such as Red5 on the back-end side & constitute a promise to further HTML5 speciftions.

Error handling

Error handling has been revised to match the latest version of the HTML5 specification. Instead of the error event being dispatched to the media element itself, it now gets delivered to the child <source> elements corresponding to the sources resulting in the error.

To detect that all child <source> elements have failed to load, check the value of the media element's networkState attribute. If this is HTMLMediaElement.NETWORK_NO_SOURCE, you know that all the sources failed to load.

If at that point you add another source by inserting a new <source> element as a child of the media element, the browser should attempt to load the specified resource.

ANNEXES & TOOLS

Media Support in desktop

ANNEXES & TOOLS

Media Support in mobile is still to be improved & requires therefore extensive functionnal testing

Tools for HTML5 level of implementation:

http://html5please.com

ANNEXES & TOOLS

HTML5 specifications are evolving constantly so pay attention to them .

SPECIFICATIONS:

W3C Audio : http://www.w3.org/TR/html5/video.html#audio

W3C Video: http://www.w3.org/TR/html5/video.html#video

W3C Media: http://www.w3.org/TR/html5/video.html#media-elements

WHAT MORE ?

. TRACK ELEMENT :

FOR MEDIA subtitles, captions, screen reader descriptions and chapters

http://www.html5rocks.com/en/tutorials/track/basics/

. WEB SPEECH RECOGNITION:

http://updates.html5rocks.com/2013/01/Voice-Driven-Web-Apps-Introduction-to-the-Web-Speech-API

WEBRTC: What's that all about ?

. WebRTC APIs

WebRTC APIs

WebRTC, standing for Web Real Time Communications, relies on 3 core APIs:

. MediaStream : getUserMedia

. RTCPeerConnection:

webkitRTCPeerConnection on Chrome

mozRTCPeerConnection on Mozilla Firefox

. RTCDataChannel

Real-Time Communication topology:

 Signalisation & data

Behind the use of these APIs the notion of Real Time Communication is based on an understanding of the communication topology & its implementation throught the differenciation of signalling & data channels as key of a communication establishment:

RTC & servers

RTC therefore relies often on a architecture requiring back-ends such as NAT , STUN servers:

BEHING THE SCENE OF THE WEBRTC APIs

is the reality of the meaning of communication in its core:

signal capture, its transcoding & its transport:

Indepently of the network issues impacting the communication such as :
. latency ,
. packet loss,
. jitter,
. noise,
. echo

Implementation:  STEP 1:

. CAPTURING AUDIO & VIDEO :

function gotStream(stream) {
  var audioContext = new webkitAudioContext();
  var mediaStreamSource = audioContext.createMediaStreamSource(stream);
  mediaStreamSource.connect(audioContext.destination);
}

navigator.webkitGetUserMedia({audio:true}, gotStream);

Implementation: STEP 2:

. DISCOVER PEER & ESTABLISH PEER CONNECTION :

function start(isCaller) {
    pc = new RTCPeerConnection(configuration);

// send any ice candidates to the other peer
    pc.onicecandidate = function (evt) {
        signalingChannel.send(JSON.stringify({ "candidate": evt.candidate }));
    };

// once remote stream arrives, show it in the remote video element
    pc.onaddstream = function (evt) {
        remoteView.src = URL.createObjectURL(evt.stream);
    };

// get the local stream, show it in the local video element and send it
    navigator.getUserMedia({ "audio": true, "video": true }, function (stream) {
        selfView.src = URL.createObjectURL(stream);
        pc.addStream(stream);

        if (isCaller)
            pc.createOffer(gotDescription);
        else
            pc.createAnswer(pc.remoteDescription, gotDescription);

        function gotDescription(desc) {
            pc.setLocalDescription(desc);
            signalingChannel.send(JSON.stringify({ "sdp": desc }));
        }
    });
}

Implementation:  STEP 3:

. STREAMING DATA :

// RTCPeerConnection setup and offer-answer exchange omitted
var dc1 = pc1.createDataChannel("mylabel");  // create the sending RTCDataChannel (reliable mode)
var dc2 = pc2.createDataChannel("mylabel");  // create the receiving RTCDataChannel (reliable mode)

// append received RTCDataChannel messages to a textarea
var receiveTextarea = document.querySelector("textarea#receive");
dc2.onmessage = function(event) {
  receiveTextarea.value += event.data;
};

var sendInput = document.querySelector("input#send");
// send message over the RTCDataChannel
function onSend() {
  dc1.send(sendInput.value);
}

ANNEXES & TOOLS:

W3C SPECIFICATIONS:  

http://www.w3.org/TR/webrtc/

Google I/O 2012 WebRTC conference by Justin Uberti:

http://www.youtube.com/watch?v=E8C8ouiXHHk&feature=player_embedded