Real time 3d
IN A BROWSER
Will Adams
Designer / Developer
Interactive Director at Psyop
Introduction
Real time 3D has been around for a while
even in our browsers. Now with support for GPU acceleration we are able to create visuals in browser
that rival console graphics.
History
We (web devs) got our start with flash.
Eco Zoo - Masayuki Kido
Drawing Images
Flash's bitmap API allowed us to translate 3D
objects onto a 2D layer.
Flash 3d Libraries
The Popular Ones
- Papervision 3D (Dead)
- Away3D (Most Popular)
- Flare3D
- Yogurt3D
In Use
3D in Flash was pretty neat but viewed as gimmicky.
It was too slow to make rich game environments.
You occasionally saw some impressive things but mostly
lots of 3D carousels, tilting images, low poly models,
and peoples faces stretched onto a mesh.
It Got Better
Hardware acceleration showed up. And now we had
three new, really awesome, 3D tools:
- Stage3D (Flash)
- CSS3 with 3D Transforms
- WebGL
Which One To Use
There are great reasons to use each technology.
(And some not so great ones)
Flash
- Most interactive devs were already using it.
- A huge amount of community support.
- By comparison AS3 is awesome.
- Really high penetration rate (twss).
- Platform independent.
- Compiles to Desktop and Mobile Apps.
css3
- Really fast to do simple things.
- 60fps sub-pixel animations.
- Works great on most devices.
- No plugins required.
WebGL (JS)
- Very fast drawing of 3D graphics.
- Increasing browser penetration.
- No plugins required.
- Ultra trendy.
- Imminent mobile support.
Webgl Seems Pretty cool
Not if you are not interested in programming CG.
Luckily for us there a few people have abstracted that
layer away. Popular 3D JS Libraries for WebGL:
- ThreeJS (The standard)
- PhiloGL (Looks promising)
- CopperLicht (Meh)
Three JS
Created by Mr Doob (ex flash guy). Currently the
most popular javascript 3D library.
PhiloGL
Created by Nicolas Garcia Belmonte. Not very popular
but has some very impressive demos.
Copperlicht
Created by Ambiera. Has a world editor built in
so thats kind of neat. Seems like they are
targeting the gaming market.
WebGL Demos
There are a ton out there. One great thing about javascript
is that the source is easily accessible (unless they were dicks and minified it in their demo).
Character Animation
Three js
Since this is the most popular (and the only JS 3D library
I have used in depth), I will run through some fundamentals of creating 3D scenes with ThreeJS.
Basic Scenes
All 3D scenes are comprised of a couple
fundamental elements.
- Cameras
- Lights
- Geometry
- Renderer
Implementation
The scene is your container for all 3D objects. Items in your scene define what is rendered.
this.scene = new THREE.Scene();
this.scene.add( this.light );
this.scene.add( this.camera );
this.scene.add( this.mesh );
Cameras
Cameras define the viewing angle of elements in the scene. There are two main types of cameras:
Perspective
- Objects further away from the camera appear smaller.
- Imagine the eye is at a fixed point.
- Similar to how we see things in real life.
Orthographic
- Objects appear at the same scale no matter how far away.
- Imagine the eye is at infinity.
- Great for grid / tile based games.
Implementation
The Perspective Camera has 4 main attributes
- Field of View
- Aspect Ratio
- Near Clipping Plane
- Far Clipping Plane
this.camera = new THREE.PerspectiveCamera( 35, this.wndw.width / this.wndw.height, 1, 5000 );
Geometry
The Geometry is the basic collection vertices, faces, and normals that make up your 3D shapes and models.
Object 3d
Your geometry is wrapped in an object that allows one to apply transformations to rotation and position.
Implementation
You can use primitive objects in ThreeJS but most likely
you will be loading in models instead.
var loader = new THREE.OBJLoader();
loader.addEventListener('load', this.onModelLoadComplete);
loader.load(this.modelLocation);
function onModelLoadCompelte(event) {
var model = event.content;
model.traverse(this.traverseModel);
this.scene.add(model);
}
Lights
Lights are used to determine the color of your objects in your 3D scene. They affect the texture of the object alter how it is displayed in the render.
MAin Types of Lights
- Ambient: Applied to everything evenly.
- Area: Light that illuminates from a surface.
- Directional: An infinite light source in one direction.
- Point: Illuminates outward from a single point.
- Spot: Point light that casts a shadow in one direction.
Implementation
var light = new THREE.DirectionalLight( 0xAAAAAA, 1.5 );
light.position.set(0, 300, 500);
light.castShadow = true;
light.shadowMapWidth = 4096;
light.shadowMapHeight = 4096;
scene.add(light);
Renderers
The renderer is what translates all of your 3D scene information and draws it onto a canvas. There are two main types of renderers for ThreeJS:
- CanvasRenderer
- WebGL Renderer
Implementation
this.renderer = new THREE.WebGLRenderer( { antialias: true } );
this.renderer.setSize( this.windondwWidth, this.windowHeight );
this.renderer.shadowMapEnabled = true;
this.renderer.shadowMapSoft = true;
this.renderer.shadowMapDarkness = 0.5;
this.renderer.shadowMapWidth = 2048;
this.renderer.shadowMapHeight = 2048;
this.renderer.shadowMapType = THREE.PCFSoftShadowMap;
this.element.appendChild( this.renderer.domElement );
So that's cool...
Nothing really new there to anyone who has worked with any web based 3D library. ThreeJS has a very similar feel to Away3D and Papervision3D.
Here are a few things that I found to be not so straight forward when working with ThreeJS & WebGL.
Textures
When adding textures to your mesh, the GPU stores them uncompressed in memory for fast random access. Mobile phones have similar guidelines; you generally have megapixel limits instead of megabyte limits for browser memory.
Compressed Vs Uncompressed
When working with a large number of high quality textures, you have to be very careful with your memory management.
Normal web files like PNG and JPG work fine in ThreeJS however the file size in memory is significantly larger than it is on disk.
RGB Memory = Width * Height * 3 bytes
RGBA Memory = Width * Height * 4 bytes
DXT1 Memory = Width * Height * 4 bytes / 8
DXT5 Memory = Width * Height * 4 bytes / 4
DDS Files
DDS is the file extension of the DXT compressed textures. They are quite big on disk which makes transfers and storage more of an issue than with PNG and JPG.
DDS MIME-TYPE
When supplying DDS files from your server, you're most likely going to need to add the MIME-Type.
And since these files are pretty big you should probably enable GZip compression / deflation to that file type as well.
AddOutputFilterByType DEFLATE image/x-dds
Creating Compressed Textures
Word on the street is Gimp has a DXT compression plugin. But who the fuck uses Gimp? For those of us who use Photoshop, Squish DDS for OSX is a handy little app. Almost no interface, you just open a file and pick a compression type.
Automating the Process
One thing we were curious about when exploring this was: how could one have user-generated content with DXT compression. NVidia has a library that can be used in conjunction with ImageMagick to take care of converting PNG and JPG files into DDS.
Additional Notes
Two things you want to keep in mind when making and using compressed textures with ThreeJS:
- Compressed textures are read in upside down.
- ThreeJS has a separate loader for compressed textures.
THREE.ImageUtils.loadCompressedTexture(source, mapType, loadHandler);
Monitoring Memory
The browser doesn't give you any warning when you are running out of memory on the GPU. It will just start to slow down and eventually crash. You need to make sure you are properly managing your textures and geometry in you site.
Apple has a great tool for their developers to monitor the communication of the video driver and the GPU.
OpenGL Driver Monitor
This is a great tool for monitoring whats going on with your GPU. It doesn't isolate each process but it you can monitor the memory change as you progress through the site. Two things you should look out for are:
- Current Free Video Memory
- GPU Core Utilization
Interacting With 3D
Since 3D objects are not DOM elements default mouse events don't exist. To track if your mouse is hitting an element you need to cast a ray. This creates an infinite straight line projected from the camera to the X and Y coordinates of the window.
Implementation
To create your ray, you need the mouse coordinates which come from mouse events. Down, up, click, and move will give you the x and y location to create your ray.
var vector = new THREE.Vector3(( x / windowWidth ) * 2 - 1, - ( y / windowHeight ) * 2 + 1, 1 );
var projector = new THREE.Projector();
projector.unprojectVector( vector, this.camera );
var raycaster = new THREE.Raycaster( this.camera.position, vector.sub( this.camera.position ).normalize() );
var intersects = raycaster.intersectObjects( objects, true);
Moving in 3d Space
Working in 3D space allows us to move in three dimensions. We generally interface with screens in a 2D fashion — with our mice, tablets, touch screens, etc.
How can we get people to control all three axis of a camera or object?
Mouse / Track Pad / Stylus
Two axis movement for X, Y or X, Z.
Mouse wheel offers third axis of movement.
-
Everyone has one.
Smart Phones / Tablets
- Gyroscope provides three axis rotation.
- Accelerometer provides three axis of position.
- Touch displays allow for custom UI controls.
- An existing user understanding of basic gestures.
- A lot of people have at least one.
Leap Motion
-
Three dimensional finger and hand tracking.
-
Tracks both hands and all fingers individually.
Gestures can be assigned as input methods.
-
SDKs available including a JavaScript version.
Limited distance away from device.
-
Slightly wonky gesture capturing.
Kinect (Xbox one)
-
Three dimensional depth input through 1080p TOF camera.
-
Tracks up to 6 skeletons, 25 joints, heart rates, facial expressions, and more.
-
Large variety of gesture support.
-
Larger coverage area for capturing.
-
Must buy a special version for PC connectivity.
-
SDK comes out sometime in 2014.
MYO
-
Captures electrical activity from your muscles.
-
Detects hand / finger / wrist movement independently.
-
Worn on your arm and connects via bluetooth, so very mobile.
-
Not out yet, so who knows how well it will work. Q1 2014
-
Closed SDK currently.
Downside
Every new technology has its downside. Being aware of what something can and can't do guides creative without wasting time on failed attempts.
WebGL in General
-
Current Apple / NVidia security error kills antialiasing.
-
Inconsistent rendering across multiple browsers.
- Huge variety of hardware manufacturers / profiles to build for.
- Javascript fucking sucks.
Three JS
Though there are a few problems with it, hats off to Mr. Doob and all the guys that have contributed to getting it where it is. They may not have made it perfect but they did it first and better than any of have so far.
Thanks For Coming
Thank you all for coming out. I'm going to show you some demos that I've been doing here at Psyop, and then we can all get drunk!