SIGGRAPH Asia 2018 | Khronos BOF
4 December 2018
Xavier Ho
CSIRO's Data61
xavier.ho@data61.csiro.au
Juan Miguel de Joya
United Nations International Telecommunication Union
juandejoya@gmail.com
Germain Rufflé
Google Daydream
germain.ruffle@gmail.com
Neil Trevett
NVIDIA, Khronos Group
ntrevett@nvidia.com
Mike Alger
Google Daydream
mikealger@google.com
To present the latest work and techniques from the WebGL community
To cast a light on relevant standard groups (e.g. WebGPU)
To discuss effective user experience design in digital and augmented spaces
Technologists
UX Design
Standards
This presentation assumes a beginner's understanding of graphics rendering
WebGL: latest and shiniest features
Optimisation techniques
Industry and community highlights
User experience design methods
WebGL Public Wiki (Khronos), Long Live WebGL (Wired)
WebGL 1.0 (2011) was widely adopted in a variety of applications:
WebGL 2.0 is growing!
WebGL Stats:
44-68% global (up from 37%, 2017)
Can I Use:
67-69% global (up from 29%, 2017)
Khronos WebGL 2.0 Webinar (Apr 2017),
WebGL2Fundamentals, Mozilla Hacks,
WebGL2 Basics (Real-time Rendering)
WebGL 1.0
WebGL 2.0
feature guarantees
Many WebGL 2.0 features are available in WebGL 1.0 as extensions.
Engines will provide fallback solutions.
You can use WebGL 2.0 today!
WebGL Extension Registry (Khornos)
MDN, Unity Benchmarks, Emscripten Docs
Demos: Tanks!, wasm-particles
WebGL rendering off the main thread
Permits smooth rendering with CPU-intensive user interactions!
Demo: Three.js
CON
Difficult to synchronise exact rendering frame with user interactions
PRO
Near 1:1 API compatibility with <canvas> as progressive enhancement
A new API for accessing GPU features on the web, based on designs from Vulkan, Metal, and Direct3D 12
Safari has an initial implementation behind flags. Chrome intends to impl.
Brings support for OpenGL ES 3.1 Compute Shaders, and GLSL 3.10
No browser currently implements this.
Chrome is working on an implementation.
Unity Compute Shader example :)
Increases performance for web apps that are CPU-bound with multiple viewports, e.g. WebVR/WebXR
Compatible with WebGL 1.0
WebGL Aquarium Demo (Chromium only, setup instructions)
(If you're lucky.)
(Doesn't work on Mac when we tried yesterday.)
A single API to conquer all VR/AR on the web, including 6DOF and controller tracking (40% adoption)
Experimental binding to native mobile VR/AR APIs - available now
Join the discussion:
There are no one-trick-fits-all optimisation techniques, due to hardware differences
Instrument profiling often, automate if possible. Save screenshots.
And remember: good enough is good enough
Debugging and Optimising WebGL Applications (New Game 2011), FBOs (Song Ho), Porting Source to Linux (Valve)
IndexDB for WebGL (David Rousset), requestAnimationFrame (Paul Irish)
Optimisation for Game Assets Guide (Samsung Developers), WebGL2Samples, Using Textures (MDN)
Use instancing as much as you can.
WebGL Techniques (Google I/O 2011) , Crowd demo (tojicode), WebGL Instancing (tojicode), Instancing demo (three.js)
Transform Feedback: a vertex post-processing step for recording values output from the vertex process into the Buffer Objects.
Computations and data from shaders stay on GPU.
Post-transform attributes can be easily read from the buffer.
Particle Systems: composed of a large amount of small particles that simulate physically-driven phenomena, such as smoke, dust, fireworks, rain, etc.
Without Transform Feedback...
1. WebGL copies particle contents of the VBO from GPU memory to CPU memory.
2. Calculations on particle attributes.
3. Reload information onto the GPU to be rendered.
Managing data transfer between GPU and CPU and redundant complex computation takes time and bandwidth!
With Transform Feedback...
1. Connect w/ Transform Feedback Buffer.
2. Apply further transformations on VBOs before rendering, all computations done on the GPU.
3. No application involvement other than connecting buffers and setting state.
Good for doing complex computations on VBOs, offload CPU and memory read times.
Caching and reusing GPU computation
Stateful particle systems
Non-WebGL implementations
Capture and reuse for geometric deformations
Terrain generation calculations
Caching and reusing GPU computation
Stateful particle systems
Non-WebGL implementations
Capture and reuse for geometric deformations
Terrain generation calculations
WebGL Best Practices (MDN)
Using Emscripten? Turn on WebGL 2.0 by using -s USE_WEBGL2=1
Check out the WebGL 2 Examples by Tarek Sherif:
State-of-the-Art WebGL 2.0 (SIGGRAPH Asia 2017)
Forward Rendering: pass geometry data into the vertex shader(s) for per-vertex operations, have these rasterized into fragments, do per-pixel operations via the fragment shader(s) before render.
Deferred Rendering: rendering is deferred until all primitives have been passed down the pipeline.
Forward Rendering…
Pros
1. Polygon data are passed per vertex and fragment, drawn over the screen.
2. Preserves true transparency and provides anti-aliasing out of the box.
Cons
1. Linear, heavy on performance
2. Wasted fragment shader calculations
3. Limited number of lights available for rendering.
Deferred Rendering…
Pros
1. Actual fragment information as a screen pixel, only one fragment shader pass.
2. Great for dynamic lights and optimizing for a larger amount of light sources and post-processing.
Cons
1. Requires high bandwidth, memory.
2. Requires transparency, anti-aliasing workarounds.
Real-time lighting via light linked list
Physically correct lighting models
SPU-based deferred shading
Extensive particle systems
Post-processing effects (bloom, FXAA, AO)
Available as WebGL 1.0 as extension, now core
Allow rendering to multiple framebuffers in one pass
Big performance boost for:
Deferred rendering
Multipass post-processing effects
Other techniques that use depth, color, normals, and so on into multiple buffers
Applications of Multiple Draw Buffers
Occluded scattering resulting in volumetric shadows and light shafts
Heterogeneous participating media
Sun/Particle interaction, physically-based sky and atmosphere simulation, large scale fog
Art directable clouds for Horizon Zero Dawn, Pixar’s The Good Dinosaur
Volumetric materials for character pipeline.
Volumetric rendering simulates the propagation of light rays into a material’s volume (a.k.a a participating media), which allow for a broad range of rendering effects.
Volume ray marching is a technique where a ray is shot/cast from the camera into the material’s volume for rendering calculations.
Images: Nop Jiarathanakul, Almar Klein
First line always starts with #version 300 es
Compiler needs to know if you’re using the new version.
Some of the new features:
flat/smooth Interpolators
Flexible loop lengths in GLSL
Dynamic indexing
See GLSL ES 3.00 Specification for the complete feature set.
Images: Shuai Shao, Trung Le, Patrick Cozzi
Three.js: r95 supports WebGL 2.0
Babylon.js: Microsoft's real-time 3D engine
PicoGL.js: minimal WebGL 2 rendering library
Unity/Unreal: uses Emscripten to port C++ code
PlayCanvas: Commercial game engine, supports most WebGL 2.0 feature
Shadertoy: Supports GLSL 3.00 fragment shader with WebGL 2
Blend4Web: Blender-interoperable engine, supports most WebGL 2.0 features
Verge3D: Blend4Web's founders' new framework
A-Frame: Open-sourced WebVR framework
Whitestorm.js: based on Three.js that simplify code, adds physics, and post-effects
After the Flood [article][video]
Voxels_WebGL2: Notch's current project [link]
Voxel engine running at 99fps
60+km view distance
Real-time terrain generation
Modifiable voxels at runtime
Facebook 3D Photos and Posts [link]
SketchFab 3D Remix Challenge [link]
Just A Reflektor (2013), Arcade Fire + Google Data Arts [link]
Project SUN (2017), Philip Schütte + Venice Art Projects [link]
Stream (2016), Ezra Miller + Houston's Day for Night Festival [link]
HTC Vive Projection Mapping in WebVR (2016), Hien Huynh + Marpi @ DanceHackDay [link]
Virtual Art Sessions, Google Tilt Brush team [link]
Codevember Examples (2018), Jaume Sanchez Elias [link]
ShaderToy Examples (2018), Inigo Quilez [link]
Data visualisation of > 2mil datasets at 60fps.
Cesium: 3D Geospatial Mapping & Analysis
Web-based visualisation for time-dynamic autonomous car simulation [link]
Regl: Declarative and stateless WebGL
Pixi.js: Fast, flexible, 2D WebGL renderer
How Uber Uses Psychological Tricks to Push Its Drivers’ Buttons, New York Times [link]
Luma.gl: GPU-powered data visualisation/computation
Deck.gl: Large-scale data visualisation
Three.js: High-level scene graph
Bear 71 by Jam3 [link]
CryptoVoxels: Blockchain visualisation [link]
Illuminated City: Crime data visualisation [link]
Healthcare visualization
Experience Insight: NASA's JPL visualisation of the Mars Rover Insight [link]
TensorFlow.js: In-browser machine learning that is deployed and executed with WebGL2 [link]
Punk Office: 3D Avatars on the Web [link]
RDF Graphs visualisation: Visualisation of Global Cargo Ships [link]
Google Earth VR: Virtual reconstruction of Earth [link]
Shipmap: Visualisation of Global Cargo Ships [link]
D3.js and Three.js: GeoJSON interoperation [link]
Device adaptations are growing (96% for WebGL 1.0, 37% for WebGL 2.0, and growing).
WebVR Specification v2 Draft: [link], slated for Q3 2018 recommendation
Firefox v55 shipped WebVR by default (8th August, 2017)
Image: Don McCurdy
Different setups
Mobile XR (Oculus Go/Quest, ARCore, ARKit)
Room-scale XR (Oculus Rift, Vive, Hololens, Magic Leap)
Full body tracking (HoloSuit)
Cinematic and/or narrative experiences
Interaction/Gaming Experiences
Virtual training/simulation
Art
Visualization
Media
Medical/Health
Academic Research
Putting yourself in the skin of a black avatar reduces implicit racial bias, Consciousness and Cognition (Vol 22, Issue 3; August 2013).
First Person Experience of Body Transfer in Virtual Reality, PLOS One (May 2010)
A Survey of Augmented Reality, Foundations and Trends in Human-Computer Interaction (Vol 8, Issue 2-3, March 2015)
Headset “Removal” for Virtual and Mixed Reality, Google Research and Daydream Labs (2017)
The Coming Age of Empathic Computing, Medium (2017)
Industry Standardization
OpenXR, Khronos Group
WebXR important part of ecosystem
Benefits of WebGL for widespread cross-compatibility and adoption
Works on every platform. WebVR specification does not distinguish what device you use.
WebVR becoming more readily available:
Firefox shipped WebXR for everyone for v55 (August 8th, 2017)
Image: SantaWinsAgain
Frameworks include:
WebVR, Google’s JavaScript API for creating immersive VR experiences [demo]
A-Frame: Make WebVR with HTML and Entity-Component. [demo]
JSARtoolkit: Emscripten port of ARToolKit to JavaScript. [demo]
Argon.js: Adding augmented reality content to web applications [demo]
Awe.js: Mixed reality on the web [link]
Image: Mozilla
VR Web Browser
WebVR NYC Snow Globe, Ronik Design
Broken Night VR, Eko
WebScreenVR, Samuel Cadillo
WebVR experiments, Google
Google I/O 2017 Prototype, Wayfair [Github]
Posture Recognition using Kinect, Azure IoT, ML and WebVR, Microsoft
BandExplorerVR, The Tin
Virtuleap WebVR Hackathon
Image: Khronos Group
WebGL 2.0 is growing fast -
we may hit 65%+ adoption next year! *
WebGL 1.0 already universal -
as seen in the state of industry survey
2018 has been a year of complementary Web API developments for WebGL.
2019 will hopefully see more WebGL extensions moving into implementation.
* based on linear projection, WebGL 2.0 may be universally supported by 2022.
Neil Trevett
Emily Stearns
Angela Cheng
Kris Rose
Takashi Umezawa
Hitoshi Kasai
Kenneth Russell
Zhenyao Mo
Kai Ninomiya
Brandon Jones
Ricardo Cabello
Diego Marcos
Don McCurdy
Kevin Ngo
Tomasz Bednarz
Felix Maya
Jaume Sanchez Elias
Leonardo Carrion
Tarek Sherif
...and more!
Pol Jeremias
Inigo Quilez
Yang Gu
Jie Chen
Jeff Lang
Olli Etuaho
Gregg Tavares
Damon Hernandez
Will Eastcott
Josh Carpenter
Iker Jamardo
Google WebXR Team
MDN Contributors
Emscripten Team
Samsung Developers