Moving Pictures

On web video and Interfaces

@opherv / Oct 2016 / CPHFront

Hi! I'm Opher

I work with developers, designers and video makers to create some pretty cool experiences

I'm a creative developer at Interlude

 

Interactive video can be...

A game

A presentation

An interview

An app

and anything else, really

Manipulation

Changing/switching the video stream content

Manipulating the video stream display (postprocessing, shaders)

 

Control

Graphic overlays ("buttons")

Interface (gestures, phone sensors, web services)

Implementing Overlays

Sounds simple...?

  • Placement is important
     
  • GUI is time based and context sensitive
     
  • GUI size needs to fit video (embedded, full screen)
     
  • Performance considerations (GPUs, decoders, devices)

Video GUI Challenges

<div class="videoContainer">
  <video controls id="video" muted autoplay loop>
    <source src="http://opherv.com/cartest.mp4" type="video/mp4">
  </video>
  <button>Do something</button>
</div>

Basic Layout

#1: GUI is Time Based and context sensitive

Video GUI Challenges

video "timeupdate" event to the rescue!
videoEl.addEventListener("timeupdate",function(e){

  if (videoEl.currentTime > 1 
      && videoEl.currentTime < 2){
    buttonEl.style.opacity = 1;
    buttonEl.style.pointerEvents = "all";
  }
  else if (videoEl.currentTime > 6){
    buttonEl.style.opacity = 0;
    buttonEl.style.pointerEvents = "none";
  }

});

Example:  That Moment When

videoEl.addEventListener("timeupdate",function(e){
  if (videoEl.currentTime > startTime 
      && videoEl.currentTime < endTime){

      buttonEl.style.left = SOME_CALCULATION;

  }
});

Moving GUI

//will get called on each frame render
var onRender = function(){
    if (videoEl.currentTime > startTime 
        && videoEl.currentTime < endTime){

    buttonEl.style.left = CURRENT_TIME_BASED_CALCULATION

    }
  requestAnimationFrame(onRender)
}

requestAnimationFrame(onRender)

 

#2: GUI needs to correspond

to your video size

Video GUI Challenges

responsive

scaling

.guiContainer{
  position: absolute;

  width: 100%;
  padding-top: 1/$aspectRatio * 100%;

  left: 0;
  top: 50%;
  transform: translateY(-50%);
}
function calculateGuiContainerSize(containerWidth, containerHeight){
      var isPillarBoxing = containerWidth/containerHeight > aspectRatio;

      if (isPillarBoxing){
        var scale = containerHeight/originalHeight;
        guiContainer.style.height = "100%";
        guiContainer.style.width = containerHeight * aspectRatio + "px";
        guiContainer.style.left = 
            ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";
        guiContainer.style.top =  0;
      }
      else{
        var scale = containerWidth/originalWidth;
        guiContainer.style.width="100%";
        guiContainer.style.height = containerWidth / aspectRatio + "px";
        guiContainer.style.top = 
            ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
        guiContainer.style.left = 0;
      }
}

Example: Trending Loop

 var isPillarBoxing = containerWidth/containerHeight > aspectRatio;
 if (isPillarBoxing){

   var scale = containerHeight/originalHeight;
   guiContainer.style.transform = "scaleX("+scale+") scaleY("+scale+")";
   guiContainer.style.top = 0;
   guiContainer.style.left = ( containerWidth - containerHeight * aspectRatio ) / 2 + "px";

}
else{

  var scale = containerWidth/originalWidth;
  guiContainer.style.transform="scaleX("+scale+") scaleY("+scale+")";
  guiContainer.style.top = ( containerHeight - containerWidth / aspectRatio ) / 2 + "px";
  guiContainer.style.left = 0;

}

GUI Scaling

Example: Mind Blown

Another option:

Baking GUI into video

example: Clash Up

 

Moving GUI can pose a challenge 

requestAnimationFrame to the rescue

 

GUI needs to adapt to your video size

account for letterboxing/pillarboxing - scale or respond

Video GUI Challenges

recap

Account for imprecisions!

Some tips

Prefer animation using hardware accelerated properties:

CSS3 3D Transforms, CSS3 Transitions (not "left" or "top")

Disable interactions when not shown

pointerEvents: "none"/ display: "none"

Test test test test test (did I say test?)

Devices fragmentation = nightmare / spec implementations / network conditions

Support

Chrome, FF, Safari, Edge

Mobile Chrome / Android

Mobile Safari / iPad

Mobile Safari / iPhone

Yeah!

Yeah!

sorta...

>iOS 10

Why is iOS<10 evil challenging?

Video is played in its own quicktime context,

on top of the browser

 

All UI events are captured by this context

On iOS-based devices with small screens—such as iPhone and iPod touch—video always plays in fullscreen mode, so the canvas cannot be superimposed on playing video. On iOS-based devices with larger screens, such as iPad, you can superimpose canvas graphics on playing video, just as you can on the desktop"

The magic keyword - use a UKWebView 

Roll your own app!

the video element in the HTML document must also include the webkit-playsinline attribute.

 

What can you do?

We solved a lot of this stuff so you don't have to

Want to learn more?

(tell them @OpherV sent you)

Thanks!

@OpherV

opherv.com

opherv@gmail.com

github.com/opherv

Moving Pictures - on video and interfaces

By Opher Vishnia

Moving Pictures - on video and interfaces

  • 872

More from Opher Vishnia