WebXR API

Virtual - Augmented - Mixed reality and the

immersive web

Vincent Ogloblinsky

Compodoc maintainer

Google Developer Expert on Web Technologies

Software architect

Agenda

1.

The immersive web

2.

AR in the web now

3.

WebXR API

4.

Progressive interfaces

Immersive web

Virtual reality

" Virtual reality (VR), which can be referred to as immersive multimedia or computer-simulated reality, replicates an environment that simulates a physical presence in places in the real world or an imagined world, allowing the user to interact in that world. "

Wikipedia

Completely digital environment

Augmented reality

" Augmented reality (AR) is a live, direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. "

Wikipedia

Real world with digital information overlay

Mixed reality

" Mixed reality (MR) is the merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real time. "

Wikipedia

Real and the virtual are intertwined

AR & MR experiences are constrained by

Knowledge of the world relative to the viewer

We can only augment what we know

The real world

We can't manipulate user and his space like in VR

VR + AR + the web = Immersive web

Reality

Augmented reality

Virtual reality

Real world

Computer-generated

2018

Now in a mature state

Augmented & virtual reality market projections - Statista

AR in the web today

Current ecosystem

Millions of phones and tablets support AR

Web browser inside

Web is the most "opened" platform

Advantages of the web

Open

Connected

Instant

AR enhance web pages

Add an AR experience in a web page for :

- create engagement

- create empathy

- create action

AR enhance web pages

Shopping

Education

Entertainement

AR enhance web pages

AR on the web : markers

Very fast

 

Web-based : three.js + jsartoolkit5

 

Open Source

 

Standards : webgl + webrtc

 

Depends on a real marker

AR on the web : markers

DEMO : Let's rebuild Ikea Place ©

AFRAME

Physical marker

Device camera

JS ARToolKit

AR.js

Aframe

Tooling / Editor

AR on the web : markerless

ARKit

ARCore

AR on the web : markerless

Features

Positional tracking of the phone

Surface detection (horizontal & vertical)

Light estimation

Image & object detection

AR on the web : AR browsers

WebARonARKit

WebARonARCore

Mozilla WebXR Viewer

AR on the web : markerless

DEMO : Let's rebuild again Ikea Place©

AFRAME

ARKit

Device camera

ARkit / ARCore

Aframe-ar

Aframe

three.ar.js

WebXR Viewer

WebXR

API

In the beginning there was WebVR API

First generation JS API for Virtual reality started in 2014-2016

Work in progress API, not supported everywhere

and in 2017 : two big players in AR

ARKit

ARCore

Why ?

Maslow’s hierarchy of a Progressive Enhancement project, by Arturo Paracuellos

Why ?

The Web is the most opened platform

The Web makes easier for developers to target all devices

For any platform to succeed, the barrier needs to be sufficiently low that artists can experiment with it freely, and expanded tooling is essential to attracting more people into creating for the medium.

 

WebXR is going to bring VR and AR to the masses. Here’s why. by Owen Williams

so...

WebVR 1.1

WebXR Device API

WebVR

Community Group

Immersive Web

Community Group

What is WebXR Device API ?

No backward compatibility with WebVR

Existing WebVR apps "should port easily"

Cleaner API

More compatible devices

Support for AR & VR in the same API

Faster ! ! ! ~50% more pixels

Goals

Detect available VR/AR devices

Query the devices capabilities

Poll the device’s position and orientation

Display imagery on the device at the appropriate frame rate

Specifications / draft

State of WebXR

Polyfill

Limitations for AR

Modules

DOM Overlay

Anchors

AR Lighting Estimation

Proposal

Experimentation

Origin Trial / flag

Shipped

WebXR API

Hit testing

Depth API

Lifetime of a WebXR application

Check XR support

Request XR session

Run render loop

Getting poses

Add anchors

End XR session

1.   Check XR support

navigator.xr.isSessionSupported('immersive-ar')
  	.then((supported) => {
          if (supported) {
            // 'immersive-ar' sessions are supported.
            // Page should advertise support to the user.
          } else {
            // 'immersive-ar' sessions are not supported.
          }
        });

2.   Request XR Session

navigator.xr.requestSession('immersive-ar', {requiredFeatures: ['hit-test']})
        .then(xrSession => {
  		console.log('requestSession immersive-ar ok');
  		// Do necessary session setup here.
  		// xrSession = session;
      		onSessionStarted(xrSession);
        })
        .catch(err => {
            // May fail for a variety of reasons.
    	});

Requires user interaction

3.   Prepare rendering

function onSessionStarted(xrSession) {

    xrSession.addEventListener('end', onSessionEnded);

    let canvas = document.createElement('canvas');
    gl = canvas.getContext('webgl', { xrCompatible: true });

    xrSession.updateRenderState({
      baseLayer: new XRWebGLLayer(xrSession, gl)
    });

    xrSession.requestReferenceSpace('local')
      .then((refSpace) => {
        xrRefSpace = refSpace;
        xrSession.requestAnimationFrame(onXRFrame);
      });

}

4.   Start rendering

function onXRFrame(time, xrFrame) {
    let xrSession = xrFrame.session;
    session.requestAnimationFrame(onXRFrame);
    // Render a frame.
}

5.  Getting poses & prepare anchors

function onXRFrame(time, xrFrame) {
    let xrSession = xrFrame.session;
    // Queue up the next frame
    xrSession.requestAnimationFrame(onXRFrame);

    let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
  
    if (xrViewerPose) {
        // Our XRFrame has an array of views. In the VR case, we'll have
        // two views, one for each eye. In mobile AR, however, we only
        // have one view.
        for (let view of xrViewerPose.views) {
            // Render something
            render(scene, camera);
            // prepare anchors
            processXRInput(xrFrame);
        }
    }
}

6.   Add anchors

function processXRInput(xrFrame) {
    let xrSession = xrFrame.session;
    const sources = Array.from(xrSession.inputSources).filter(input => input.targetRayMode === 'screen');

    const pose = xrFrame.getPose(sources[0].targetRaySpace, xrRefSpace);
    if (pose) {
        placeModel();
    }
}

function placeModel() {
    let hits, raycaster = new THREE.Raycaster();
    raycaster.setFromCamera({0,0}, camera);
  
    const ray = raycaster.ray;
    
    let xrray = new XRRay(ray.origin, ray.direction);
    hits = xrSession.requestHitTest(xrray, xrRefSpace);

    if (hits && hits.length) {
        const hit = hits[0];

        const hitMatrix = new THREE.Matrix4().fromArray(hit.hitMatrix);

        model.position.setFromMatrixPosition(hitMatrix);
    }
}

7.   Close session

function onSessionEnded(event) {

    xrSession = null;

}

WebXR API support

Not supported everywhere

In-development : Firefox / WebKit

Safari implementation : non-official in-progress (Fernando Serrano recruited at Apple)

Want to play with it today ?

WebXR API enabled by default

Google Chrome Beta 79

WebXR API Emulator

Demo time : with WebXR API

DEMO : Let's rebuild again Ikea Place ©

ARCore

Device camera

ARCore

three.js

Google Chrome Canary

Google WebXR Experiments

Google WebXR Experiments

<model-viewer>

UX design in AR

UX design for WebXR apps

UX design for WebXR apps

UX design for WebXR apps

Designing the Wider Web - Trevor Smith

UX design for WebXR apps

PotassiumES

Wider web app framework

Create once for flat, portal, and immersive displays

Want some inspiration ?

Tooling / Editor

Tooling / Editor

Codelab

Codelab

People to follow

Brandon Jones

WebXR developer at Google

Blair MacIntyre

Research Scientist at Mozilla

Jordan Santell

Indie engineering consultant

Ada Rose Cannon

Developer Advocate at Samsung

Josh Carpenter

Indie UX designer VR/AR

Diego Gonzalez

PM Edge for PWA

Want some news ? 

American computer scientist and Internet pioneer, widely regarded as the "father of computer graphics".

Quote of the day

The screen is a window through which one sees a virtual world. The challenge is to make that world look real, act real, sound real, feel real.

- Ivan Sutherland -

Helpful resources - Articles

Helpful resources - Presentations

WebXR Primer by Brandon Jones

 

Writing Vr and Ar apps with web technology by Michaela Lehr

 

WebVR, WebXR and A-Frame - Intro to the Next-Gen Web by Dr. Pete Markiewicz

 

WebXR : Introducting Mixed reality and the immersive web by Peter O’Shaughnessy & Diego González, Ph.D

Helpful resources - Channels

The future of the web is immersive / Google I/O '18

Brandon Jones & John Pallet

Helpful resources - Videos

WebXR : Introducting Mixed reality and the immersive web

Peter O'Shaughnessy

Augmented Reality: Past, Present, Future | JSConf Iceland 2018

Raisa Cuevas

Helpful resources - Videos

A Roadmap for Future AR Technology in WebXR

Blair MacIntyre

Exploring Augmented Reality on the web - JSConf EU 2018

Jordan Santell

Helpful resources - Videos

Thanks for listening !

Ask me anything during the conference

Slides : bit.ly/webxr-api

WebXR API - Virtual/Augmented/Mixed reality and the immersive web

By Vincent Ogloblinsky

WebXR API - Virtual/Augmented/Mixed reality and the immersive web

After virtual reality & augmented reality, here is the third : mixed reality. In this fusion of real and virtual world, mixed reality adds objects in our real environment which user can interact with. WebXR in an official API draft from Mozilla to extend WebVR API for augmented reality, with support of frameworks like ARKit (Apple) et ARCore (Google). In this talk, discover how we can build right now with web languages such interactive web experiences.

  • 4,398