Moving Pictures

On web video and Interfaces

@opherv / Oct 2016 / CPHFront

Hi! I'm Opher

I work with developers, designers and video makers to create some pretty cool experiences

I'm a creative developer at Interlude

What's

interactive video?

Interactive video can be...

A game

A presentation

An interview

An app

and anything else, really

Manipulation

Changing/switching the video stream content

Manipulating the video stream display (postprocessing, shaders)

Control

Graphic overlays ("buttons")

Interface (gestures, phone sensors, web services)

Implementing Overlays

Sounds simple...?

• Placement is important
   
• GUI is time based and context sensitive
   
• GUI size needs to fit video (embedded, full screen)
   
• Performance considerations (GPUs, decoders, devices)

Video GUI Challenges

<div class="videoContainer">
  <video controls id="video" muted autoplay loop>
    <source src="http://opherv.com/cartest.mp4" type="video/mp4">
  </video>
  <button>Do something</button>
</div>

Basic Layout

#1: GUI is Time Based and context sensitive

Video GUI Challenges

Example:  That Moment When

videoEl.addEventListener("timeupdate",function(e){
  if (videoEl.currentTime > startTime 
      && videoEl.currentTime < endTime){

      buttonEl.style.left = SOME_CALCULATION;

  }
});

Moving GUI

//will get called on each frame render
var onRender = function(){
    if (videoEl.currentTime > startTime 
        && videoEl.currentTime < endTime){

    buttonEl.style.left = CURRENT_TIME_BASED_CALCULATION

    }
  requestAnimationFrame(onRender)
}

requestAnimationFrame(onRender)

#2: GUI needs to correspond

to your video size

Video GUI Challenges

.guiContainer{
  position: absolute;

  width: 100%;
  padding-top: 1/$aspectRatio * 100%;

  left: 0;
  top: 50%;
  transform: translateY(-50%);
}

function calculateGuiContainerSize(containerWidth, containerHeight){
      var isPillarBoxing = containerWidth/containerHeight > aspectRatio;

      if (isPillarBoxing){
        var scale = containerHeight/originalHeight;
        guiContainer.style.height = "100%";
        guiContainer.style.width = containerHeight * aspectRatio + "px";
        guiContainer.style.left = 
            ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";
        guiContainer.style.top =  0;
      }
      else{
        var scale = containerWidth/originalWidth;
        guiContainer.style.width="100%";
        guiContainer.style.height = containerWidth / aspectRatio + "px";
        guiContainer.style.top = 
            ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
        guiContainer.style.left = 0;
      }
}

Example: Trending Loop

 var isPillarBoxing = containerWidth/containerHeight > aspectRatio;
 if (isPillarBoxing){

   var scale = containerHeight/originalHeight;
   guiContainer.style.transform = "scaleX("+scale+") scaleY("+scale+")";
   guiContainer.style.top = 0;
   guiContainer.style.left = ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";

}
else{

  var scale = containerWidth/originalWidth;
  guiContainer.style.transform="scaleX("+scale+") scaleY("+scale+")";
  guiContainer.style.top = ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
  guiContainer.style.left = 0;

}

GUI Scaling

Example: Mind Blown

Another option:

Baking GUI into video

example: Clash Up

Moving GUI can pose a challenge 

requestAnimationFrame to the rescue

GUI needs to adapt to your video size

account for letterboxing/pillarboxing - scale or respond

Video GUI Challenges

recap

Some tips

Why is iOS<10 evil challenging?

Video is played in its own quicktime context,

on top of the browser

All UI events are captured by this context

On iOS-based devices with small screens—such as iPhone and iPod touch—video always plays in fullscreen mode, so the canvas cannot be superimposed on playing video. On iOS-based devices with larger screens, such as iPad, you can superimpose canvas graphics on playing video, just as you can on the desktop"

What can you do?

A parting gift

developer.interlude.fm

We solved a lot of this stuff so you don't have to

Want to learn more?

(tell them @OpherV sent you)

Thanks!

@OpherV

opherv.com

opherv@gmail.com

github.com/opherv