OpenGL and Shaders

CPU Workloads

Think about many of the programs you wrote before this class.

def func1(my_list, target):
  while len(my_list) > target:
    el_last = find_thing(my_list)
    my_list.remove(el_last)
  return my_list

def func2(tgt):
  if tgt.left is not None:
    func2(tgt)
  print(tgt.data)
  if tgt.right is not None:
    funct(tgt.right)

Which happens next?

Known as branches

Properties of (most) Programs

Execute one thing at a time
Have to resolve branches (don't know what's going to happen next)
Have to do each individual thing very fast

def func1(my_list, target):
  while len(my_list) > target:
    el_last = find_thing(my_list)
    my_list.remove(el_last)
  return my_list

CPUs are generally very good at handling the unpredictable, branch-dependent workloads of typical programs.

Graphics Programs

Drawing Shapes

void setup(){
  size(500, 500); 
}

void draw(){
  rect(20,50,100,100);
}

Processing Images

-1	0	1
-2	0	2
-1	0	1

Apply this kernel to each pixel in the source image to get the result.

Any branches?

Drawing in 3D

Ambient depends on just ambient coefficient
Diffuse value at a pixel depends on the surface normal of the shape at that pixel, and the direction of the light.
Specular value at a pixel depends on surface normal, direction of light, and direction of camera.

Drawing in 3D

No branches! Color of pixels can be computed totally independently!

No if-else or loops!

Drawing in 3D

Rasterization is the process of taking shapes and turning them into pixels.

This is a more general form of the square-drawing exercise we looked at earlier, and it is also highly independent and branch-free.

This pixel's appearance has nothing to do with what that pixel looks like.

Transforms

Recall that transforms are performed as matrix-vector or matrix-matrix multiplications.

Each matrix-multiply is branch-free and can be done independently.

What about transforming 1000 points?

Graphics Workloads are different!

Unlike most CPU workloads, workloads in graphics tend to be

Low-branch
Data-independent
Parallel

but involve large amounts of data movement.

We can exploit this by using special hardware which specializes in this kind of workload to speed up graphics!

Graphics Processing Unit

GPUs are processing units which specialize in the types of computations needed for graphics.

Sacrifice the ability to deal effectively (or at all!) with branches and other issues common to CPU code.

In exchange, gain the ability to move massive amounts of data and execute on it very quickly (as long as you don't have code with the things it can't deal with!)

Typical CPU: ~100 independent operations per cycle.

Typical GPU: ~10,000 independent operations per cycle.

GPU Speed Demo

void setup(){
  // size(1200, 800);
  // size(1200, 800, P3D);
}

void draw(){
   int NUM_CIRCLES = 500;
   int R = 100;
   for(int i = 0; i < NUM_CIRCLES; i++){
     int ix = int(random(0, 1200));
     int iy = int(random(0, 800));
     ellipse(ix, iy, R, R);
   }
   println(frameRate);
}

How do we tell the GPU what to do?

Some of the GPU's functions are fixed in the hardware, and there's nothing we can do to change them.

The Graphics Pipeline

Application sends scene data from CPU to GPU.
GPU transforms the data into geometry.
Geometry is rasterized.
Image data is transformed into screen space.

OpenGL

An open graphics programming system supported by almost all major vendors.

Provides a programmable pipeline for graphics applications.

Requires support from multiple parties:

Hardware needs to support the draw commands used
Operating system needs to provide the appropriate libraries, software, and interfaces to support OpenGL usage.
Application needs to provide the programs to run.
Everyone needs to support the use of shaders so that custom rendering rules can be implemented.

Shaders

What are Shaders?

Shaders are programs that are run on the GPU.

Written in a special shading language. In our case, this will be the OpenGL Shading Language (GLSL), but can also be others, e.g. HLSL, RenderManSL, etc.

GLSL

A language which is most similar to C (but pretty similar to Java).

Because it runs on the GPU, don't have access to things like print, or the ability to read files.

A few oddities like swizzling and varying/uniform variables.

varying vec4 thing;
uniform mat4 matt;

void main(){
  vec3 v1 = vec3(1.0, 2.0, 3.0);
  vec4 v2;
  
  // These are not members!
  v2.xyz = v1.xxy;
  v2.w = v1.x;
}

Vertex Shader

Fragment Shader

Vertex Shader

Takes in vertices and applies an operation to every vertex.

Examples:

Transform each vertex to a new position
Compute lighting/color data for each vertex
Can displace vertices to make a bumpy/textured surface (though in classic OpenGL, this is more commonly done in the tessellation shader).

Vertex Shader

uniform mat4 transform;
attribute vec4 position;
void main() {
  //Must set gl_Position in vertex shader
  //for each vertex processed
  gl_Position = transform * position;
}

Shaders run main() for each input they get! Inputs are passed in by shader-global variables.

For the vertex shader:

uniform variables are the same across all inputs.
attribute variables may change per-input, but are readonly.
varying variables may change per-input and are writeable.

Fragment Shader

Takes in fragments (pieces of shapes that the pipeline has rendered).

Also takes in any outputs the fragment shader may have created.

Outputs the color to use on the fragment.

void main() {
  // Color components are in [0.0, 1.0]
  // instead of [0, 255]
  gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0);
}

Moving Data

In our example vertex shader, we took a transform and position as input, and computed the transformed position of the vertex.

uniform mat4 transform;
attribute vec4 position;
void main() {
  //Must set gl_Position in vertex shader
  //for each vertex processed
  gl_Position = transform * position;
}

Question: Where does this data come from?

Question: How does this data get there?

vertexPositions = [[1,2,3,1], [1,3,4,1], [1,4,7,1], ...]
transformMat = Mat4.identity()
# When shader runs on vertex 1, vertexPositions[1]
# is availble as the variable "positions"
glBindAttribute("position", vertexPositions)
# transformMat will be availble to all shader invocations
# under the variable name "transform"
glBindUniform("transform", transformMat)

uniform mat4 transform;
attribute vec4 position;
void main() {
  //Must set gl_Position in vertex shader
  //for each vertex processed
  gl_Position = transform * position;
}

Manually moving data between the CPU and GPU is error prone. Let OpenGL handle it instead.

Moving Data Between Shaders

attribute vec3 position;
attribute float size;
varying vec3 vert_Output;
void main() {
  //Must set gl_Position in vertex shader
  //for each vertex processed
  gl_Position = vec4(position, 1.0);
  vert_Output = vec3(gl_Position.xyz)
}

Shaders run in a fixed order. In our system, vertex runs first, then fragment.

To pass a variable between shaders, declare it in the first shader and assign a value to it. It will be available in the second shader.

varying vec3 vert_Output;
void main() {
  // We have access to vert_Output
  // in the fragment shader!
}

Overall Setup Steps

Compile shader programs (e.g. vertex, fragment, etc.)
Link shader programs together to make one big shader
Tell the GPU to use our shader program
Specify data location on the CPU so that the GPU knows how to find it
Set output data location
Draw stuff!

Shaders in Processing

Shaders in Processing are Moderately Cursed

No way to control varying variables fed into the shader
Have to load vertex and fragment shaders together
Basically completely undocumented---only way to know what happens is to read the Processing source code.
Shaders are basically un-debuggable.

For Processing, stick to modifying in-class examples or existing shading code that works.

Processing API

PShader blur = loadShader("blur.glsl");

Just like most other load_ functions. First filename is always a fragment shader, pass optional second filename to load a vertex shader as well.

We can use the .set() method of PShader to set uniforms within the shader.

Shaders are hard!

Visual scripting languages can make it much easier to program shaders. Example: Substance allows us to program shaders for materials in a visual manner.

Engines like Unreal 4 also allow us to visually script shaders, along with providing things like previews of the material results

Demo: Shaders

Hands-On: Shaders

Download the hands-on skeleton from Canvas. This contains enough skeleton code to do a very basic rendering of the famous Utah teapot.
Modify the code so that the ambient light and directional light work, by following the hints in the shader files.
OPTIONAL: Try adding another directional light, either with the same color or a different color.

Index Cards!

Your name and EID.
One thing that you learned from class today. You are allowed to say "nothing" if you didn't learn anything.
One question you have about something covered in class today. You may not respond "nothing".
(Optional) Any other comments/questions/thoughts about today's class.