Real time 3d

IN A BROWSER



Will Adams

Designer / Developer

Interactive Director at Psyop



Introduction

Real time 3D has been around for a while
even in our browsers. Now with support for GPU acceleration we are able to create visuals in browser
that rival console graphics.

History

We (web devs) got our start with flash. 

Eco Zoo - Masayuki Kido



Drawing Images

Flash's bitmap API allowed us to translate 3D
objects onto a 2D layer.


Flash 3d Libraries

The Popular Ones

  • Papervision 3D (Dead)
  • Away3D (Most Popular)
  • Flare3D
  • Yogurt3D


In Use

3D in Flash was pretty neat but viewed as gimmicky. 
It was too slow to make rich game environments.

You occasionally saw some impressive things but mostly
lots of 3D carousels, tilting images, low poly models,
and peoples faces stretched onto a mesh.





It Got Better

Hardware acceleration showed up. And now we had
three new, really awesome, 3D tools:

  1. Stage3D (Flash)
  2. CSS3 with 3D Transforms
  3. WebGL



Which One To Use

There are great reasons to use each technology.
(And some not so great ones) 


Flash

  • Most interactive devs were already using it.
  • A huge amount of community support.
  • By comparison AS3 is awesome.
  • Really high penetration rate (twss).
  • Platform independent.
  • Compiles to Desktop and Mobile Apps.


css3

  • Really fast to do simple things.
  • 60fps sub-pixel animations.
  • Works great on most devices.
  • No plugins required.


WebGL (JS)

  • Very fast drawing of 3D graphics.
  • Increasing browser penetration.
  • No plugins required.
  • Ultra trendy.
  • Imminent mobile support.


Webgl Seems Pretty cool

Not if you are not interested in programming CG.
Luckily for us there a few people have abstracted that
layer away. Popular 3D JS Libraries for WebGL:

  • ThreeJS (The standard)
  • PhiloGL (Looks promising)
  • CopperLicht (Meh)



Three JS

Created by Mr Doob (ex flash guy). Currently the
most popular javascript 3D library.
www.threejs.org



PhiloGL

Created by Nicolas Garcia Belmonte. Not very popular
but has some very impressive demos.
www.senchalabs.org/philogl/



Copperlicht

Created by Ambiera. Has a world editor built in
so thats kind of neat. Seems like they are
targeting the gaming market.
www.ambiera.com/copperlicht/



WebGL Demos

There are a ton out there. One great thing about javascript
is that the source is easily accessible (unless they were dicks and minified it in their demo).

Smoke Particles

Character Animation

Area Lights


Water & Light Refraction


Volumetric Light




Three js

Since this is the most popular (and the only JS 3D library
I have used in depth), I will run through some fundamentals of creating 3D scenes with ThreeJS.


Basic Scenes

All 3D scenes are comprised of a couple
fundamental elements.

  • Cameras
  • Lights
  • Geometry
  • Renderer


Implementation

The scene is your container for all 3D objects. Items in your scene define what is rendered.

this.scene = new THREE.Scene();this.scene.add( this.light );
this.scene.add( this.camera );
this.scene.add( this.mesh );



Cameras

Cameras define the viewing angle of elements in the scene. There are two main types of cameras:

  • Perspective
  • Orthographic



Perspective

  • Objects further away from the camera appear smaller.
  • Imagine the eye is at a fixed point.
  • Similar to how we see things in real life.



Orthographic

  • Objects appear at the same scale no matter how far away.
  • Imagine the eye is at infinity.
  • Great for grid / tile based games.


Implementation

The Perspective Camera has 4 main attributes

  1. Field of View
  2. Aspect Ratio
  3. Near Clipping Plane
  4. Far Clipping Plane


this.camera = new THREE.PerspectiveCamera( 35, this.wndw.width / this.wndw.height, 1, 5000 );



Geometry

The Geometry is the basic collection vertices, faces, and normals that make up your 3D shapes and models.



Object 3d

Your geometry is wrapped in an object that allows one to apply transformations to rotation and position.


Implementation

You can use primitive objects in ThreeJS but most likely
you will be loading in models instead.

var loader = new THREE.OBJLoader();
loader.addEventListener('load', this.onModelLoadComplete);
loader.load(this.modelLocation);
function onModelLoadCompelte(event) { var model = event.content; model.traverse(this.traverseModel); this.scene.add(model);}



Lights

Lights are used to determine the color of your objects in your 3D scene. They affect the texture of the object alter how it is displayed in the render. 



MAin Types of Lights

  • Ambient: Applied to everything evenly.
  • Area: Light that illuminates from a surface.
  • Directional: An infinite light source in one direction.
  • Point: Illuminates outward from a single point.
  • Spot: Point light that casts a shadow in one direction.


Implementation


var light = new THREE.DirectionalLight( 0xAAAAAA, 1.5 );
light.position.set(0, 300, 500); light.castShadow = true;light.shadowMapWidth = 4096;light.shadowMapHeight = 4096;scene.add(light);



Renderers

The renderer is what translates all of your 3D scene information and draws it onto a canvas. There are two main types of renderers for ThreeJS:

  • CanvasRenderer
  • WebGL Renderer


Implementation


this.renderer = new THREE.WebGLRenderer( { antialias: true } );
this.renderer.setSize( this.windondwWidth, this.windowHeight );
			
this.renderer.shadowMapEnabled = true;
this.renderer.shadowMapSoft = true;

this.renderer.shadowMapDarkness = 0.5;
this.renderer.shadowMapWidth = 2048;
this.renderer.shadowMapHeight = 2048;
this.renderer.shadowMapType = THREE.PCFSoftShadowMap;

this.element.appendChild( this.renderer.domElement );



So that's cool...

Nothing really new there to anyone who has worked with any web based 3D library. ThreeJS has a very similar feel to Away3D and Papervision3D.

Here are a few things that I found to be not so straight forward when working with ThreeJS & WebGL.



Textures

When adding textures to your mesh, the GPU stores them uncompressed in memory for fast random access. Mobile phones have similar guidelines; you generally have megapixel limits instead of megabyte limits for browser memory.


Compressed Vs Uncompressed

When working with a large number of high quality textures, you have to be very careful with your memory management.
Normal web files like PNG and JPG work fine in ThreeJS however the file size in memory is significantly larger than it is on disk.

RGB Memory = Width * Height * 3 bytes

RGBA Memory = Width * Height * 4 bytes
DXT1 Memory = Width * Height * 4 bytes / 8
DXT5 Memory = Width * Height * 4 bytes / 4



DDS Files

DDS is the file extension of the DXT compressed textures. They are quite big on disk which makes transfers and storage more of an issue than with PNG and JPG.


DDS MIME-TYPE

When supplying DDS files from your server, you're most likely going to need to add the MIME-Type.

AddType image/x-dds .dds 

And since these files are pretty big you should probably enable GZip compression / deflation to that file type as well.

AddOutputFilterByType DEFLATE image/x-dds 



Creating Compressed Textures

Word on the street is Gimp has a DXT compression plugin. But who the fuck uses Gimp? For those of us who use Photoshop, Squish DDS for OSX is a handy little app. Almost no interface, you just open a file and pick a compression type.

SquishDDS 1.1.1



Automating the Process

One thing we were curious about when exploring this was: how could one have user-generated content with DXT compression. NVidia has a library that can be used in conjunction with ImageMagick to take care of converting PNG and JPG files into DDS. 

DDS Texture Conversion



Additional Notes

Two things you want to keep in mind when making and using compressed textures with ThreeJS:

  1. Compressed textures are read in upside down.
  2. ThreeJS has a separate loader for compressed textures.


THREE.ImageUtils.loadCompressedTexture(source, mapType, loadHandler);


Monitoring Memory

The browser doesn't give you any warning when you are running out of memory on the GPU. It will just start to slow down and eventually crash. You need to make sure you are properly managing your textures and geometry in you site.

Apple has a great tool for their developers to monitor the communication of the video driver and the GPU.

Graphics Tools for Xcode


OpenGL Driver Monitor

This is a great tool for monitoring whats going on with your GPU. It doesn't isolate each process but it you can monitor the memory change as you progress through the site. Two things you should look out for are:

  1. Current Free Video Memory
  2. GPU Core Utilization



Interacting With 3D

Since 3D objects are not DOM elements default mouse events don't exist. To track if your mouse is hitting an element you need to cast a ray. This creates an infinite straight line projected from the camera to the X and Y coordinates of the window.


Implementation

To create your ray, you need the mouse coordinates which come from mouse events. Down, up, click, and move will give you the x and y location to create your ray.

var vector = new THREE.Vector3(( x / windowWidth ) * 2 - 1, - ( y / windowHeight ) * 2 + 1, 1 );

var projector = new THREE.Projector();
projector.unprojectVector( vector, this.camera );
            
var raycaster = new THREE.Raycaster( this.camera.position, vector.sub( this.camera.position ).normalize() );
var intersects = raycaster.intersectObjects( objects, true);



Moving in 3d Space

Working in 3D space allows us to move in three dimensions. We generally interface with screens in a 2D fashion — with our mice, tablets, touch screens, etc. 

How can we get people to control all three axis of a camera or object?



Mouse / Track Pad / Stylus


  • Two axis movement for X, Y or X, Z.

  • Mouse wheel offers third axis of movement.


  • Everyone has one.



Smart Phones / Tablets


  • Gyroscope provides three axis rotation.
  • Accelerometer provides three axis of position. 
  • Touch displays allow for custom UI controls.
  • An existing user understanding of basic gestures.
  • A lot of people have at least one.


Leap Motion


  • Three dimensional finger and hand tracking.


  • Tracks both hands and all fingers individually.


  • Gestures can be assigned as input methods.

  • SDKs available including a JavaScript version.


  • Limited distance away from device.

  • Slightly wonky gesture capturing.


Kinect (Xbox one)


  • Three dimensional depth input through 1080p TOF camera.


  • Tracks up to 6 skeletons, 25 joints, heart rates, facial expressions, and more.


  • Large variety of gesture support.


  • Larger coverage area for capturing.


  • Must buy a special version for PC connectivity.


  • SDK comes out sometime in 2014.


MYO


  • Captures electrical activity from your muscles.


  • Detects hand / finger / wrist movement independently.


  • Worn on your arm and connects via bluetooth, so very mobile.


  • Not out yet, so who knows how well it will work. Q1 2014


  • Closed SDK currently.



Downside

Every new technology has its downside. Being aware of what something can and can't do guides creative without wasting time on failed attempts. 



WebGL in General


  • Current Apple / NVidia security error kills antialiasing.


  • Inconsistent rendering across multiple browsers.


  • Huge variety of hardware manufacturers / profiles to build for.
  • Javascript fucking sucks.

Three JS


  • Built by one developer, managed by a few.


  • Documentation is terrible.

  • Poorly architected and organized.
  • No funding or sponsorship.


Though there are a few problems with it, hats off to Mr. Doob and all the guys that have contributed to getting it where it is. They may not have made it perfect but they did it first and better than any of have so far. 



Thanks For Coming

Thank you all for coming out. I'm going to show you some demos that I've been doing here at Psyop, and then we can all get drunk!