Lindsay Kay

BioDigital Tech Talk

September 22, 2017

Picking

3D Objects

within

Human

Four Flavors of Picking

Human's picking system supports four types of picking:

  1. Pick object at canvas coords
  2. Pick object surface position at canvas coords
  3. Pick object with arbitrary 3D ray
  4. Pick object surface position with arbitrary 3D ray

Flavor 1: Pick Object at Canvas Coords 

Find object at mouse coordinates:

Flavor 1: Pick Object at Canvas Coords (cont)

Usage:

var hit = Human.renderer.pick({
    canvasPos: [500,400]
});

if (hit) { // Picked an object
    var object = Human.scene.objects[hit.objectId];
    //...
} 

Flavor 1: Pick Object at Canvas Coords (cont)

Algorithm:

  1. User picks at canvas coordinates.
  2. Render scene to auxiliary image buffer, filling each object with a unique color that is the RBGA-encoded index of its position within the internal display list.
  3. Read color from the image buffer at the canvas coordinates, map the color back to the object in the display list.
  4. Return a hit result containing ID of the object.

Flavor 2: Pick Object Surface at Canvas Coords

Find surface intersection at mouse coordinates:

Flavor 2: Pick Object Surface at Canvas Coords (cont)

var hit = Human.renderer.pick({
    canvasPos: [500,400],
    pickSurface: true,    // <<----- Indicates that we want to pick on surface
});

if (hit) { // Picked an object

    var object = hit.objectId;      // ID of object we picked
    var primitive = hit.primitive;  // Type of primitive, usually "triangles"
    var primIndex = hit.primIndex;  // Index of triangle within object's indices 
    var indices = hit.indices;      // Value of each of the triangle's indices
    var localPos = hit.localPos;    // Local-space coordinates
    var worldPos = hit.worldPos;    // World-space coordinates
    var viewPos = hit.viewPos;      // View-space coordinates
    var bary = hit.bary;            // Barycentric coordinates within the triangle
    var normal = hit.normal;        // Interpolated normal vector within the triangle
    var uv = hit.uv;                // Interpolated UVs within the triangle
}

Usage:

Flavor 2: Pick Object Surface at Canvas Coords (cont)

Algorithm:

  1. User ray-picks at canvas coordinates.
  2. Render scene to auxiliary image buffer, filling each object with unique color that is the RBGA-encoded index of its position within the display list.
  3. Read color from image buffer at canvas coords, map to object in the display list.
  4. Clear image buffer, render object's triangles, filling each with unique color as RBGA-encoded index of triangle within object’s geometry.
  5. Read color from image buffer at canvas coords, map to the picked triangle.
  6. Make ray originating at eye and passing through near projection plane where we picked, then unproject to get ray in object’s local coordinate space.
  7. Find local-space intersection of ray with triangle.

Flavor 3: Pick Objects with Arbitrary Ray

Same as Flavor 1, but using a ray in 3D space:

Flavor 3: Pick Objects with Arbitrary Ray (cont)

var hit = Human.renderer.pick({
    origin: [0,0,-5],   // Ray origin
    direction: [0,0,1], // Ray direction
    pickSurface: false  // Don't want intersect info
});

if (hit) { // Picked an object with the ray
     var object = Human.scene.objects[hit.objectId]
}

Usage:

Flavor 3: Pick Objects with Arbitrary Ray (cont)

Algorithm:

  1. User picks using a ray in World-space.
  2. Position the camera to look along the ray.
  3. Render the scene to an auxiliary image buffer, filling each entity with a unique color that is the RBGA-encoded index of its position within the internal display list.
  4. Restore the camera to its previous position.
  5. Read the color from the image buffer at the canvas coordinates, map the color back to an object in the display list.
  6. Return a hit result containing the object.

Flavor 4: Pick Object Surfaces with Arbitrary Ray

Same as Flavor 2, but using a ray in 3D space:

Flavor 4: Pick Object Surfaces with Arbitrary Ray (cont)

var hit = Human.renderer.pick({
    origin: [0,0,-5],   // Ray origin
    direction: [0,0,1], // Ray direction
    pickSurface: true   // Pick on surface
});

if (hit) { // Picked an object with the ray

     var objectId = hit.objectId;    
     var primitive = hit.primitive;  
     var primIndex = hit.primIndex; 
     var indices = hit.indices;      
     var localPos = hit.localPos;    
     var worldPos = hit.worldPos;    
     var viewPos = hit.viewPos;      
     var bary = hit.bary;            
     var normal = hit.normal;        
     var uv = hit.uv;                
}

Usage:

Flavor 4: Pick Object Surfaces with Arbitrary Ray (cont)

Algorithm:

  1. User picks using a ray in World-space.
  2. Position camera to look along the ray.
  3. Render scene to image buffer, filling each entity with unique color that is RBGA-encoded index of its position within display list.
  4. Restore camera to previous position.
  5. Read color from image buffer at canvas coord, map to object in display list.
  6. Clear image buffer, render triangles of object, each with unique color mapping to triangle within geometry indices.
  7. Read color from the image buffer at the canvas coords, map back to the triangle.
  8. Restore camera to previous position.
  9. With object and the triangle, make ray in clip-space, originating at the eye position and passing through the near projection plane at a position corresponding to where we picked, then unproject that ray to get a ray in the entity’s local coordinate space.
  10. Find the intersection of the ray with the triangle in local space.
  11. Find the barycentric coordinates of the local-space intersection, then use those to interpolate within the triangle to find the normal vector and UV coordinates at that position.
  12. Return a hit result containing the picked object, the triangle, and the ray-triangle intersection info.

Flavor 4: Pick Object Surfaces with Arbitrary Ray (cont)

Advantages:

  1. Linear time complexity O(n) - time taken increases linearly with increase in number of objects and triangles
  2. Done mostly on GPU - way faster than JS on CPU

Disadvantages:

  1. Only finds closest intersection
  2. Requires dedicated lazy-computed vertex colors and indices to support individually-colored triangles for picking

Application of Ray Picking: Drawing

Four flavors of picking used in BioDigital Human

By xeolabs

Four flavors of picking used in BioDigital Human

Four flavors of picking used in BioDigital Human

  • 1,648