Moving Pictures

On web video and Interfaces

@opherv / Oct 2016 / CPHFront

Hi! I'm Opher

I work with developers, designers and video makers to create some pretty cool experiences

I'm a creative developer at Interlude


Interactive video can be...

A game

A presentation

An interview

An app

and anything else, really


Changing/switching the video stream content

Manipulating the video stream display (postprocessing, shaders)



Graphic overlays ("buttons")

Interface (gestures, phone sensors, web services)

Implementing Overlays

Sounds simple...?

  • Placement is important
  • GUI is time based and context sensitive
  • GUI size needs to fit video (embedded, full screen)
  • Performance considerations (GPUs, decoders, devices)

Video GUI Challenges

<div class="videoContainer">
  <video controls id="video" muted autoplay loop>
    <source src="" type="video/mp4">
  <button>Do something</button>

Basic Layout

#1: GUI is Time Based and context sensitive

Video GUI Challenges

video "timeupdate" event to the rescue!

  if (videoEl.currentTime > 1 
      && videoEl.currentTime < 2){ = 1; = "all";
  else if (videoEl.currentTime > 6){ = 0; = "none";


Example:  That Moment When

  if (videoEl.currentTime > startTime 
      && videoEl.currentTime < endTime){ = SOME_CALCULATION;


Moving GUI

//will get called on each frame render
var onRender = function(){
    if (videoEl.currentTime > startTime 
        && videoEl.currentTime < endTime){ = CURRENT_TIME_BASED_CALCULATION




#2: GUI needs to correspond

to your video size

Video GUI Challenges



  position: absolute;

  width: 100%;
  padding-top: 1/$aspectRatio * 100%;

  left: 0;
  top: 50%;
  transform: translateY(-50%);
function calculateGuiContainerSize(containerWidth, containerHeight){
      var isPillarBoxing = containerWidth/containerHeight > aspectRatio;

      if (isPillarBoxing){
        var scale = containerHeight/originalHeight; = "100%"; = containerHeight * aspectRatio + "px"; = 
            ( containerWidth - containerHeight * aspectRatio ) / 2 + "px"; =  0;
        var scale = containerWidth/originalWidth;"100%"; = containerWidth / aspectRatio + "px"; = 
            ( containerHeight - containerWidth / aspectRatio ) / 2 + "px"; = 0;

Example: Trending Loop

 var isPillarBoxing = containerWidth/containerHeight > aspectRatio;
 if (isPillarBoxing){

   var scale = containerHeight/originalHeight; = "scaleX("+scale+") scaleY("+scale+")"; = 0; = ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";


  var scale = containerWidth/originalWidth;"scaleX("+scale+") scaleY("+scale+")"; = ( containerHeight - containerWidth / aspectRatio ) / 2 + "px"; = 0;


GUI Scaling

Example: Mind Blown

Another option:

Baking GUI into video

example: Clash Up


Moving GUI can pose a challenge 

requestAnimationFrame to the rescue


GUI needs to adapt to your video size

account for letterboxing/pillarboxing - scale or respond

Video GUI Challenges


Account for imprecisions!

Some tips

Prefer animation using hardware accelerated properties:

CSS3 3D Transforms, CSS3 Transitions (not "left" or "top")

Disable interactions when not shown

pointerEvents: "none"/ display: "none"

Test test test test test (did I say test?)

Devices fragmentation = nightmare / spec implementations / network conditions


Chrome, FF, Safari, Edge

Mobile Chrome / Android

Mobile Safari / iPad

Mobile Safari / iPhone




>iOS 10

Why is iOS<10 evil challenging?

Video is played in its own quicktime context,

on top of the browser


All UI events are captured by this context

On iOS-based devices with small screens—such as iPhone and iPod touch—video always plays in fullscreen mode, so the canvas cannot be superimposed on playing video. On iOS-based devices with larger screens, such as iPad, you can superimpose canvas graphics on playing video, just as you can on the desktop"

The magic keyword - use a UKWebView 

Roll your own app!

the video element in the HTML document must also include the webkit-playsinline attribute.


What can you do?

We solved a lot of this stuff so you don't have to

Want to learn more?

(tell them @OpherV sent you)