WebXR API

Virtual - Augmented - Mixed reality and the

immersive web

Vincent Ogloblinsky

Frontend software architect

Tweet at @vogloblinsky

Compodoc Core team

Open source at vogloblinsky

Agenda

1.

The immersive web

2.

AR in the web now

3.

WebXR API

4.

Progressive interfaces

Immersive web

Virtual reality

" Virtual reality (VR), which can be referred to as immersive multimedia or computer-simulated reality, replicates an environment that simulates a physical presence in places in the real world or an imagined world, allowing the user to interact in that world. "

Wikipedia

Completely digital environment

Augmented reality

" Augmented reality (AR) is a live, direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. "

Wikipedia

Real world with digital information overlay

Mixed reality

" Mixed reality (MR) is the merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real time. "

Wikipedia

Real and the virtual are intertwined

AR & MR experiences are constrained by

Knowledge of the world relative to the viewer

We can only augment what we know

 

The real world

We can't manipulate user and his space like in VR

VR + AR + the web = Immersive web

Reality

Augmented reality

Virtual reality

Real world

Computer-generated

AR in the web today

AR now...

Millions of phones and tablets support AR

Web browser inside

Web is the most "opened" platform

Advantages of the web

Open

Connected

Instant

AR enhance web pages

Add an AR experience in a web page for :

- create engagement

- create empathy

- create action

AR enhance web pages

Shopping

Education

Entertainement

AR enhance web pages

AR on the web : markers

Very fast

 

Web-based : three.js + jsartoolkit5

 

Open Source

 

Standards : webgl + webrtc

 

Depends on a real marker

AR on the web : markers

DEMO : Let's rebuild Ikea Place©

AFRAME

Physical marker

Device camera

JS ARToolKit

AR.js

Aframe

AR on the web : markerless

ARKit

ARCore

AR on the web : markerless

Features

Positional tracking of the phone

Surface detection (horizontal & vertical)

Light estimation

Image & object detection

ARKit 2

ARCore 1.5

AR on the web : AR browsers

WebARonARKit

WebARonARCore

Mozilla WebXR Viewer

AR on the web : three.ar.js

A helper three.js library for building AR web experiences that run in WebARonARKit and WebARonARCore.

https://github.com/google-ar/three.ar.js

three.js

three.ar

WebARonARKit

WebARonARCore

WebVR Extension for AR

WebVR Extension for AR helps :

- motion tracking

- rendering the pass through camera

- basic understanding of the real world

AR on the web : markerless

DEMO : Let's rebuild again Ikea Place©

AFRAME

ARKit 2

Device camera

ARkit / ARCore

Aframe-ar

Aframe

three.ar.js

WebXR Viewer

WebXR

API

In the beginning there was WebVR API

First generation JS API for Virtual reality

Work in progress API, not supported everywhere

and in 2017 : two big players in AR

ARKit

ARCore

Why ?

Maslow’s hierarchy of a Progressive Enhancement project, by Arturo Paracuellos

Why ?

The Web is the most opened platform

The Web makes easier for developers to target all devices

For any platform to succeed, the barrier needs to be sufficiently low that artists can experiment with it freely, and expanded tooling is essential to attracting more people into creating for the medium.

 

WebXR is going to bring VR and AR to the masses. Here’s why. by Owen Williams

so...

WebVR 1.1

WebXR Device API

WebVR

Community Group

Immersive Web

Community Group

What is WebXR Device API ?

No backward compatibility with WebVR

Existing WebVR apps "should port easily"

Cleaner API

More compatible devices

Support for AR & VR in the same API

Faster ! ! ! ~50% more pixels

Goals

Detect available VR/AR devices

Query the devices capabilities

Poll the device’s position and orientation

Display imagery on the device at the appropriate frame rate

Specifications / draft

Specifications / draft

Session.requestHitTest

AR Sessions

Anchors

Proposal

Experimentation

Origin Trials

Shipped

Polyfill

Lifetime of a WebXR application

Request XR device

Reveal XR functionality

Request XR session

Run render loop

End session

1.   Request device

class App {
    constructor() {
        this.init();
    }

    async init() {
        navigator.xr.requestDevice().then(device => {
            this.xrDevice = device; // type of XRDevice
            this.detectXRMode();
        }, err => {
            if (err.name === 'NotFoundError') {
                console.error('No XR devices available:', err);
            } else if (err.name === 'NotAllowedError') {
                console.error('No permission for accessing XR capabilities:', err);
            } else {
                console.error('An error occurred while requesting an XRDevice:', err);
            }
        });
    }
}

2.   Detect XR mode

async detectXRMode() {
    this.xrDevice.supportsSession({ exclusive: true }).then(() => {
        var enterXrBtn = document.createElement('button');
        enterXrBtn.innerHTML = 'Enter AR';
        enterXrBtn.addEventListener('click', this.beginXRSession);
        document.body.appendChild(enterVrBtn);
    }).catch((reason) => {
        console.log("Session not supported: " + reason);
    });
}

Sessions have two levels of access :

- exclusive : present content directly to XRDevice. Need user gesture

- non-exclusive : !present, but access tracking information of XRDevice.

3.   Request a XRSession

async beginXRSession() {

    const outputCanvas = document.createElement('canvas');
    
    const ctx = outputCanvas.getContext('xrpresent');
    
    this.xrDevice.requestSession({ exclusive: true, outputContext: ctx })
        .then(onSessionStarted)
        .catch(err => {
            // May fail for a variety of reasons. Probably just want to
            // render the scene normally without any tracking at this point.
            window.requestAnimationFrame(onDrawFrame);
    });
}

4.1.   Prepare rendering

async onSessionStarted(session) {

    this.xrSession = session;

    this.renderer = new THREE.WebGLRenderer({alpha: true, preserveDrawingBuffer: true});

    this.gl = this.renderer.getContext();

    await this.gl.setCompatibleXRDevice(this.session.device);

    this.xrSession.baseLayer = new XRWebGLLayer(this.xrSession, this.gl);

    this.frameOfRef = await this.xrSession.requestFrameOfReference('eye-level');

    this.xrSession.requestAnimationFrame(this.onXRFrame);

}

4.2.   Start rendering

onXRFrame(time, frame) {
    let session = frame.session;
    let pose = frame.getDevicePose(this.frameOfRef);

    // Queue up the next frame
    session.requestAnimationFrame(this.onXRFrame);

    if (pose) {
        // Our XRFrame has an array of views. In the VR case, we'll have
        // two views, one for each eye. In mobile AR, however, we only
        // have one view.
        for (let view of frame.views) {
            // Render someting 
        }
    }
}

5.   Add anchors

async onClick(e) {

    const x = 0;
    const y = 0;

    this.raycaster = this.raycaster || new THREE.Raycaster();
    this.raycaster.setFromCamera({ x, y }, this.camera);
    
    const ray = this.raycaster.ray;

    const origin = new Float32Array(ray.origin.toArray());
    
    const direction = new Float32Array(ray.direction.toArray());
    
    const hits = await this.session.requestHitTest(origin, direction, this.frameOfRef);

    if (hits.length) {
      // We can have multiple collisions per hit test. Take the first hit, the nearest, for now.
      const hit = hits[0];

      const hitMatrix = new THREE.Matrix4().fromArray(hit.hitMatrix);

      // Now apply the position from the hitMatrix onto our model.
      this.model.position.setFromMatrixPosition(hitMatrix);
    }

}

6.   Close session

async endXRSession() {

    await this.xrSession.end();
    
    this.xrSession = null;

}

How today ?

Enable flags in version 72+

chrome://flags/

Google Chrome Canary

Demo time : with WebXR API

DEMO : Let's rebuild again Ikea Place©

ARCore 1.5

Device camera

ARCore

three.js

Google Chrome Canary

UX design in AR

UX design for WebXR apps

UX design for WebXR apps

UX design for WebXR apps

Designing the Wider Web - Trevor Smith

UX design for WebXR apps

PotassiumES - Create once for flat, portal, and immersive displays

WebXR domains - work in progress

Persistance

Geospatial

3D graphics

Multi-user

Computer vision

Browsers

Tooling

Want some inspiration ?

Codelab

People to follow

Brandon Jones

WebXR developer at Google

Blair MacIntyre

Research Scientist at Mozilla

Jordan Santell

WebXR engineer at Google

Ada Rose Cannon

Developer Advocate at Samsung

Josh Carpenter

UX Lead, WebVR/AR at Google

Diego Gonzalez

Developer Advocate at Samsung

Want some news ? 

Firefox Reality

American computer scientist and Internet pioneer, widely regarded as the "father of computer graphics".

Quote of the day

The screen is a window through which one sees a virtual world. The challenge is to make that world look real, act real, sound real, feel real.

- Ivan Sutherland -

Helpful resources - Articles

Helpful resources - Presentations

WebXR Primer by Brandon Jones

 

Writing Vr and Ar apps with web technology by Michaela Lehr

 

WebVR, WebXR and A-Frame - Intro to the Next-Gen Web by Dr. Pete Markiewicz

 

WebXR : Introducting Mixed reality and the immersive web by Peter O’Shaughnessy & Diego González, Ph.D

The future of the web is immersive / Google I/O '18

Brandon Jones & John Pallet

Helpful resources - Videos

WebXR : Introducting Mixed reality and the immersive web

Peter O'Shaughnessy

Augmented Reality: Past, Present, Future | JSConf Iceland 2018

Raisa Cuevas

Helpful resources - Videos

A Roadmap for Future AR Technology in WebXR

Blair MacIntyre

Exploring Augmented Reality on the web - JSConf EU 2018

Jordan Santell

Helpful resources - Videos

Thanks for listening !

Ask me anything during the conference

Slides : bit.ly/webxr-api

WebXR API - Virtual/Augmented/Mixed reality and the immersive web

By Vincent Ogloblinsky

WebXR API - Virtual/Augmented/Mixed reality and the immersive web

After virtual reality & augmented reality, here is the third : mixed reality. In this fusion of real and virtual world, mixed reality adds objects in our real environment which user can interact with. WebXR in an official API draft from Mozilla to extend WebVR API for augmented reality, with support of frameworks like ARKit (Apple) et ARCore (Google). In this talk, discover how we can build right now with web languages such interactive web experiences.

  • 901
Loading comments...

More from Vincent Ogloblinsky