What do I do?

• Work with JavaScript
• "If it doesn't break on webkit, don't fix it"
• Lots of R&D
• Observing different behavior over the wide variety of devices

Evolution Gaming

Which of those are shared in WebKit-based browsers?

• Parsing (HTML, XML, CSS, JavaScript)
• Layout
• Text and graphics rendering
• Image decoding
• GPU interaction
• Network access
• Hardware acceleration

Some of  devices in our office

You may already be familiar with the curious case of client side persistent data solutions

• globalStorage: Firefox 2.0+, Internet Explorer 8
• localStorage: development WebKit
• openDatabase: Safari 3.1+
• userdata behavior: Internet Explorer 5.5+
• Adobe Flash
• Google Gears

Fortunately

The challenges

• Desktop browsers mostly don't support any of the native live delivery formats
• Modernizr & feature detection (canplayType etc) can't be trusted
• Mobile networks may have a limited bandwidth
• Multiple models can lie under the same device name (such as Samsung Galaxy S4)
  (http://en.wikipedia.org/wiki/Samsung_Galaxy_S4#Model_variants)

HLS playback capabilities in desktop browsers

1. HLS is built into the Android OS since version 3. It has severe issues on Ice Cream Sandwich (4.0). Subsequent Android releases improved upon HLS, but even Lollipop (5.0) has issues with e.g. accurate seeking and updating of controls.

(http://www.jwplayer.com/html5/hls/)

canPlayType

(https://developer.mozilla.org/en/docs/Web/API/HTMLMediaElement)

canPlayType(in DOMStringtype)

• probably: if the specified type appears to be playable.
• maybe: if it's impossible to tell whether the type is playable without playing it.
• The empty string: if the specified type definitely cannot be played.

Mostly for HLS you will get maybe, which is not very reliable data

Possible approaches to test bandwidth issues in mobile devices

• Charles proxy
• Dedicated router for testing (http://www.dd-wrt.com/help/english/HQos.asp)
• Network link conditioner (Only for iOS devices, and requires developer mode enabled on the device)

Possible approaches in delivery

Flash

• Not acceptable according to cool kids
• Deep control over latency & buffering
• Not an option for mobile devices
• Less cross-platform differences

RTSP in android native browser

• Not supported in google chrome
• Initialization over tcp may take 15 seconds
• Has a rather low latency
• Quite stable (but unsupported)

HLS (HTTP live streaming)

• Plays flawlessly in apple devices
• Offers adaptive streaming
• Unacceptable minimal latency
• Different m3u8 formats, with variable support across devices
• Upcoming Android devices are quite demanding in regard for the encoder settings, and may require mystical tweaks
• In some cases it's impossible to detect that the delivery has failed.
• May stall on androids due to insufficient bandwidth

HLS (HTTP live streaming)

Binary operations in javascript to remux video

http://rreverser.github.io/mpegts/

Also plugins for flash players available

http://osmfhls.kutu.ru/

WebSockets + Canvas

• Comparatively reliable in the client side
• High CPU & Battery usage for MPEG1 and MP3 decoding in JS
• Not all mobile devices offer access to Audio API context
• Comparatively high bandwidth usage
• Limited control over potential congestion issues in client side
• Some mobile network operators (Vodafone) may not allow usage of WS
• WSS results in even higher bandwidth and CPU consumption

MSE (Media Source Extensions)

Specification extends HTMLMediaElement to allow JavaScript to generate media streams for playback. Allowing JavaScript to generate streams facilitates a variety of use cases like adaptive streaming and time shifting live streams. (http://w3c.github.io/media-source/)

var ms = new MediaSource();

var video = document.querySelector('video');
video.src = window.URL.createObjectURL(ms);

ms.addEventListener('sourceopen', function(e) {
  //...
  var sourceBuffer = ms.addSourceBuffer('video/webm; codecs="vorbis,vp8"');
  sourceBuffer.appendBuffer(oneVideoWebMChunk);
  //....
}, false);

MSE (Media Source Extensions)

• Not supported in Safari
• For now various experiments have been made, using the MPEG-DASH protocol.
  It works fine in Chrome, however impossible to configure it with low latency
• Potentially an option for video streaming over websockets, however that requires server side solution.

MSE (Media Source Extensions)

http://dashif.org/reference/players/javascript/1.2.0/index.html

WebRTC

http://www.html5rocks.com/en/tutorials/webrtc/basics/

• Also utilizing MSE
• Similar to WebSockets
• Promise for very low latency
• Multiple use cases
• Simple and complicated at the same time
• Unconfirmed stability

How does youtube Make use of MSE?

<video
 class="video-stream html5-main-video"
 x-webkit-airplay="allow"
 data-youtube-id="5yCGRy_dR30"
 style="width: 640px; height: 360px; left: 0px; top: 0px; transform: none;"
 src="blob:https%3A//www.youtube.com/ced8792d-6d88-4a75-bee5-56592ca676c3">
</video>

So what's the blob in the video elements source?

• The source of the video element is a blob url of a MediaSource
• Periodic XHR requests are being made for the respective video ranges
• Player gains full control over the content download throttling, and the chunks can also be limited to a session

Ok, So what's the point?

Lessons learned

• When it comes to live video, native apps have an advantage
• Not always the cool kids can be trusted
• The player must be able to detect the available and failing playback methods
• Our job is interesting
• YouTube can be reverse-
  engineered