WebXR API
Virtual - Augmented - Mixed reality and the
immersive web
Vincent Ogloblinsky
Compodoc maintainer
Google Developer Expert on Web Technologies
Software architect
Agenda
1.
The immersive web
2.
AR in the web now
3.
WebXR API
4.
Progressive interfaces
Immersive web
Virtual reality
" Virtual reality (VR), which can be referred to as immersive multimedia or computer-simulated reality, replicates an environment that simulates a physical presence in places in the real world or an imagined world, allowing the user to interact in that world. "
Wikipedia
Completely digital environment
Augmented reality
" Augmented reality (AR) is a live, direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. "
Wikipedia
Real world with digital information overlay
Mixed reality
" Mixed reality (MR) is the merging of real and virtual worlds to produce new environments and visualisations where physical and digital objects co-exist and interact in real time. "
Wikipedia
Real and the virtual are intertwined
AR & MR experiences are constrained by
Knowledge of the world relative to the viewer
We can only augment what we know
The real world
We can't manipulate user and his space like in VR
VR + AR + the web = Immersive web
Reality
Augmented reality
Virtual reality
Real world
Computer-generated
2018
Now in a mature state
Augmented & virtual reality market projections - Statista
AR in the web today
Current ecosystem
Millions of phones and tablets support AR
Web browser inside
Web is the most "opened" platform
Advantages of the web
Open
Connected
Instant
AR enhance web pages
Add an AR experience in a web page for :
- create engagement
- create empathy
- create action
AR enhance web pages
Shopping
Education
Entertainement
AR enhance web pages
AR on the web : markers
Very fast
Web-based : three.js + jsartoolkit5
Open Source
Standards : webgl + webrtc
Depends on a real marker
AR on the web : markers
DEMO : Let's rebuild Ikea Place ©
AFRAME
Physical marker
Device camera
JS ARToolKit
AR.js
Aframe
Tooling / Editor
AR on the web : markerless
ARKit
ARCore
AR on the web : markerless
Features
Positional tracking of the phone
Surface detection (horizontal & vertical)
Light estimation
Image & object detection
AR on the web : AR browsers
WebARonARKit
WebARonARCore
Mozilla WebXR Viewer
AR on the web : markerless
DEMO : Let's rebuild again Ikea Place©
AFRAME
ARKit
Device camera
ARkit / ARCore
Aframe-ar
Aframe
three.ar.js
WebXR Viewer
WebXR
API
In the beginning there was WebVR API
First generation JS API for Virtual reality started in 2014-2016
Work in progress API, not supported everywhere
and in 2017 : two big players in AR
ARKit
ARCore
Why ?
Maslow’s hierarchy of a Progressive Enhancement project, by Arturo Paracuellos
Why ?
The Web is the most opened platform
The Web makes easier for developers to target all devices
For any platform to succeed, the barrier needs to be sufficiently low that artists can experiment with it freely, and expanded tooling is essential to attracting more people into creating for the medium.
WebXR is going to bring VR and AR to the masses. Here’s why. by Owen Williams
so...
WebVR 1.1
WebXR Device API
WebVR
Community Group
Immersive Web
Community Group
What is WebXR Device API ?
No backward compatibility with WebVR
Existing WebVR apps "should port easily"
Cleaner API
More compatible devices
Support for AR & VR in the same API
Faster ! ! ! ~50% more pixels
Goals
Detect available VR/AR devices
Query the devices capabilities
Poll the device’s position and orientation
Display imagery on the device at the appropriate frame rate
Specifications / draft
State of WebXR
Polyfill
Limitations for AR
Modules
DOM Overlay
Anchors
AR Lighting Estimation
Proposal
Experimentation
Origin Trial / flag
Shipped
WebXR API
Hit testing
Depth API
Lifetime of a WebXR application
Check XR support
Request XR session
Run render loop
Getting poses
Add anchors
End XR session
1. Check XR support
navigator.xr.isSessionSupported('immersive-ar')
.then((supported) => {
if (supported) {
// 'immersive-ar' sessions are supported.
// Page should advertise support to the user.
} else {
// 'immersive-ar' sessions are not supported.
}
});2. Request XR Session
navigator.xr.requestSession('immersive-ar', {requiredFeatures: ['hit-test']})
.then(xrSession => {
console.log('requestSession immersive-ar ok');
// Do necessary session setup here.
// xrSession = session;
onSessionStarted(xrSession);
})
.catch(err => {
// May fail for a variety of reasons.
});Requires user interaction
3. Prepare rendering
function onSessionStarted(xrSession) {
xrSession.addEventListener('end', onSessionEnded);
let canvas = document.createElement('canvas');
gl = canvas.getContext('webgl', { xrCompatible: true });
xrSession.updateRenderState({
baseLayer: new XRWebGLLayer(xrSession, gl)
});
xrSession.requestReferenceSpace('local')
.then((refSpace) => {
xrRefSpace = refSpace;
xrSession.requestAnimationFrame(onXRFrame);
});
}4. Start rendering
function onXRFrame(time, xrFrame) {
let xrSession = xrFrame.session;
session.requestAnimationFrame(onXRFrame);
// Render a frame.
}5. Getting poses & prepare anchors
function onXRFrame(time, xrFrame) {
let xrSession = xrFrame.session;
// Queue up the next frame
xrSession.requestAnimationFrame(onXRFrame);
let xrViewerPose = xrFrame.getViewerPose(xrRefSpace);
if (xrViewerPose) {
// Our XRFrame has an array of views. In the VR case, we'll have
// two views, one for each eye. In mobile AR, however, we only
// have one view.
for (let view of xrViewerPose.views) {
// Render something
render(scene, camera);
// prepare anchors
processXRInput(xrFrame);
}
}
}6. Add anchors
function processXRInput(xrFrame) {
let xrSession = xrFrame.session;
const sources = Array.from(xrSession.inputSources).filter(input => input.targetRayMode === 'screen');
const pose = xrFrame.getPose(sources[0].targetRaySpace, xrRefSpace);
if (pose) {
placeModel();
}
}
function placeModel() {
let hits, raycaster = new THREE.Raycaster();
raycaster.setFromCamera({0,0}, camera);
const ray = raycaster.ray;
let xrray = new XRRay(ray.origin, ray.direction);
hits = xrSession.requestHitTest(xrray, xrRefSpace);
if (hits && hits.length) {
const hit = hits[0];
const hitMatrix = new THREE.Matrix4().fromArray(hit.hitMatrix);
model.position.setFromMatrixPosition(hitMatrix);
}
}7. Close session
function onSessionEnded(event) {
xrSession = null;
}WebXR API support
Not supported everywhere
In-development : Firefox / WebKit
Safari implementation : non-official in-progress (Fernando Serrano recruited at Apple)
Want to play with it today ?
WebXR API enabled by default
Google Chrome Beta 79
WebXR API Emulator
Demo time : with WebXR API
DEMO : Let's rebuild again Ikea Place ©
ARCore
Device camera
ARCore
three.js
Google Chrome Canary
Google WebXR Experiments
Google WebXR Experiments
<model-viewer>
UX design in AR
UX design for WebXR apps
UX design for WebXR apps
UX design for WebXR apps
Designing the Wider Web - Trevor Smith
UX design for WebXR apps
Want some inspiration ?
Tooling / Editor
Tooling / Editor
Codelab
Codelab
People to follow
Brandon Jones
WebXR developer at Google
Blair MacIntyre
Research Scientist at Mozilla
Jordan Santell
Indie engineering consultant
Ada Rose Cannon
Developer Advocate at Samsung
Josh Carpenter
Indie UX designer VR/AR
Diego Gonzalez
PM Edge for PWA
Want some news ?
American computer scientist and Internet pioneer, widely regarded as the "father of computer graphics".
Quote of the day
The screen is a window through which one sees a virtual world. The challenge is to make that world look real, act real, sound real, feel real.
- Ivan Sutherland -
Helpful resources - Articles
Helpful resources - Presentations
WebXR Primer by Brandon Jones
Writing Vr and Ar apps with web technology by Michaela Lehr
WebVR, WebXR and A-Frame - Intro to the Next-Gen Web by Dr. Pete Markiewicz
WebXR : Introducting Mixed reality and the immersive web by Peter O’Shaughnessy & Diego González, Ph.D
Helpful resources - Channels
The future of the web is immersive / Google I/O '18
Brandon Jones & John Pallet
Helpful resources - Videos
WebXR : Introducting Mixed reality and the immersive web
Peter O'Shaughnessy
Augmented Reality: Past, Present, Future | JSConf Iceland 2018
Raisa Cuevas
Helpful resources - Videos
A Roadmap for Future AR Technology in WebXR
Blair MacIntyre
Exploring Augmented Reality on the web - JSConf EU 2018
Jordan Santell
Helpful resources - Videos
Thanks for listening !
Ask me anything during the conference
Slides : bit.ly/webxr-api