Moving Pictures

On web video and Interfaces

@opherv / Oct 2016 / CPHFront

Hi! I'm Opher

I work with developers, designers and video makers to create some pretty cool experiences

I'm a creative developer at Interlude

What's

interactive video?

Interactive video can be...

A game

A presentation

An interview

An app

and anything else, really

Manipulation

Changing/switching the video stream content

Manipulating the video stream display (postprocessing, shaders)

Control

Graphic overlays ("buttons")

Interface (gestures, phone sensors, web services)

Implementing Overlays

Sounds simple...?

Placement is important
GUI is time based and context sensitive
GUI size needs to fit video (embedded, full screen)
Performance considerations (GPUs, decoders, devices)

Video GUI Challenges

<div class="videoContainer">
  <video controls id="video" muted autoplay loop>
    <source src="http://opherv.com/cartest.mp4" type="video/mp4">
  </video>
  <button>Do something</button>
</div>

Basic Layout

#1: GUI is Time Based and context sensitive

Video GUI Challenges

video "timeupdate" event to the rescue!

videoEl.addEventListener("timeupdate",function(e){

  if (videoEl.currentTime > 1 
      && videoEl.currentTime < 2){
    buttonEl.style.opacity = 1;
    buttonEl.style.pointerEvents = "all";
  }
  else if (videoEl.currentTime > 6){
    buttonEl.style.opacity = 0;
    buttonEl.style.pointerEvents = "none";
  }

});

Example: That Moment When

videoEl.addEventListener("timeupdate",function(e){
  if (videoEl.currentTime > startTime 
      && videoEl.currentTime < endTime){

      buttonEl.style.left = SOME_CALCULATION;

  }
});

Moving GUI

//will get called on each frame render
var onRender = function(){
    if (videoEl.currentTime > startTime 
        && videoEl.currentTime < endTime){

    buttonEl.style.left = CURRENT_TIME_BASED_CALCULATION

    }
  requestAnimationFrame(onRender)
}

requestAnimationFrame(onRender)

#2: GUI needs to correspond

to your video size

Video GUI Challenges

responsive

scaling

.guiContainer{
  position: absolute;

  width: 100%;
  padding-top: 1/$aspectRatio * 100%;

  left: 0;
  top: 50%;
  transform: translateY(-50%);
}

function calculateGuiContainerSize(containerWidth, containerHeight){
      var isPillarBoxing = containerWidth/containerHeight > aspectRatio;

      if (isPillarBoxing){
        var scale = containerHeight/originalHeight;
        guiContainer.style.height = "100%";
        guiContainer.style.width = containerHeight * aspectRatio + "px";
        guiContainer.style.left = 
            ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";
        guiContainer.style.top =  0;
      }
      else{
        var scale = containerWidth/originalWidth;
        guiContainer.style.width="100%";
        guiContainer.style.height = containerWidth / aspectRatio + "px";
        guiContainer.style.top = 
            ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
        guiContainer.style.left = 0;
      }
}

Example: Trending Loop

 var isPillarBoxing = containerWidth/containerHeight > aspectRatio;
 if (isPillarBoxing){

   var scale = containerHeight/originalHeight;
   guiContainer.style.transform = "scaleX("+scale+") scaleY("+scale+")";
   guiContainer.style.top = 0;
   guiContainer.style.left = ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";

}
else{

  var scale = containerWidth/originalWidth;
  guiContainer.style.transform="scaleX("+scale+") scaleY("+scale+")";
  guiContainer.style.top = ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
  guiContainer.style.left = 0;

}

GUI Scaling

Example: Mind Blown

Another option:

Baking GUI into video

example: Clash Up

Moving GUI can pose a challenge

requestAnimationFrame to the rescue

GUI needs to adapt to your video size

account for letterboxing/pillarboxing - scale or respond

Video GUI Challenges

recap

Account for imprecisions!

Some tips

Prefer animation using hardware accelerated properties:

CSS3 3D Transforms, CSS3 Transitions (not "left" or "top")

Disable interactions when not shown

pointerEvents: "none"/ display: "none"

Test test test test test (did I say test?)

Devices fragmentation = nightmare / spec implementations / network conditions

Support

Chrome, FF, Safari, Edge

Mobile Chrome / Android

Mobile Safari / iPad

Mobile Safari / iPhone

Yeah!

sorta...

>iOS 10

Why is iOS<10 evil challenging?

Video is played in its own quicktime context,

on top of the browser

All UI events are captured by this context

On iOS-based devices with small screens—such as iPhone and iPod touch—video always plays in fullscreen mode, so the canvas cannot be superimposed on playing video. On iOS-based devices with larger screens, such as iPad, you can superimpose canvas graphics on playing video, just as you can on the desktop"

The magic keyword - use a UKWebView

Roll your own app!

the video element in the HTML document must also include the webkit-playsinline attribute.

What can you do?

A parting gift

developer.interlude.fm

We solved a lot of this stuff so you don't have to

Want to learn more?

(tell them @OpherV sent you)

Thanks!

@OpherV

opherv.com

opherv@gmail.com

github.com/opherv