Images

Last Time

Attributes/Modes: Function calls that modify all subsequent draw calls.

Attributes

Change how results look.

  • fill()
  • stroke()

Modes

Change what arguments mean.

  • rectMode()
  • colorMode()
  • blendMode()

Last Time

Digital color uses an additive model, usually with 8 bits (0-255) each for RGB.

Can use attributes like fill, stroke, and background to affect color of what's drawn.

color yellow = color(255.0, 255.0, 0.0);
fill(yellow);
rect(0, 0, 200, 200);

Multiple ways to use these functions: can specify RGB or pass a color primitive

fill(255.0, 255.0, 0.0);
rect(0, 0, 200, 200);

Last Time

Remember, draw() is called in a loop!

void draw(){
  rect(10, 10, 20, 20);
  fill(blue);
  ellipse(100, 100, 200, 200);
}

Attributes and modes can bleed through the bottom of the loop and start affecting things before them if you're not careful!

Most functions that take color in Processing will have these three variants:

  • Take a single color or hex argument
  • Take a single numerical grayscale argument
  • Take three separate arguments (RGB)

and for each of these, there will usually be a variant that takes an additional alpha (transparency) argument.

Office Hours!

Third floor bridge of GDC. (Enter the main atrium of GDC and take the staircase up one flight).

Mine: Monday and Wednesday 11:30-12:30 (before class)

Annabel's: Monday and Friday 3:00-4:00 (after class)

Questions

Why do we use RGB over CYMK?

Alt: how do you pick a color model?

Depends on how you get your color!

Projector: throw light at the screen, all of it reflects. Additive model.

Paper: throw light at paper, some of it gets absorbed. Subtractive model.

How do places like YouTube manage the size of videos?

What does the next frame look like?

a

b

c

d

Fundamental idea: the better you can predict something, the less information you need to describe it.

The dog falls in front of the tree.

The scene suddenly cuts away to several rows of rubber ducks, of various colors, shapes and designs. They appear to be laid out on a long strip of concrete which curves slightly as it moves away from the camera. There are chairs and tables in the background....

void drawRect(int a, int b, 
              int c, int d){
  fill(BLACK);
  stroke(BLACK);
  rect(a,b,c,d);
}

int WHITE = 255;
int BLACK = 0;

void draw(){
  fill(WHITE);
  stroke(BLACK);
  rect(0, 0, 50, 50);
 
  drawRect(0, 0, 100, 100);

  // What color does this draw in?
  rect(100, 100, 100, 100);
}

Why does the blendMode matter? Can't we just change the color of the shape instead?

Yes, but the complexity can rapidly get out of hand.

Have you ever personally gone down a color theory rabbithole?

Can I use burnt orange without getting cancelled by UT copyright?

It turns out you can't copyright colors!

Copyright

 

Protects a specific artistic or intellectual expression of an idea.

Trademark

 

Protects a recognizable design which identifies a particular product or service.

Also: patent and trade secrets (but let's not turn this into a law class)

Hands-On Review

More About Colors

RGB Color

We have defined color as a triplet of numbers, representing the red, green, and blue components of the color.

color yellow = color(255.0, 255.0, 0.0);

What if I told you there was another way?

HSV/HSB

Hue-Saturation-Value (or Brightness) is commonly used in color pickers

  • Hue: pure color
  • Saturation: amount of color
  • Value: darkness or lightness of color

How do we change the interpretation of colors between HSV/RGB?

colorMode(RGB, 255, 255, 255);
colorMode(HSB, 360, 100, 100);
colorMode(RGB, 1.0, 1.0, 1.0);
colorMode(HSB, 100);

Extracting Data from a color 

float r = red(color c);
float g = green(color c);
float b = blue(color c);
float h = hue(color c);
float s = saturation(color c);
float v = brightness(color c);
colorMode(RGB, 255, 255, 255);
fill(50, 100, 100);
rect(0, 0, 50, 50); //Rect1
colorMode(HSB, 360, 100, 100);
fill(50, 100, 100);
rect(50, 50, 50, 50); //Rect2

Image Storage

Pixels

CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1477223

Each pixel has some bits that determine the color.

Images are composed of pixels.

Image Buffer

Screen pixel data is stored in a buffer (array), which allows us to access per-pixel information.

0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
18 19 20 21 22 23
24 25 26 27 28 29
30 31 32 33 34 35

How the pixels look on screen

0 1 2 3 4 5 6 7 8 9

...

How the pixels are stored in memory

Images in Processing

  • Images in processing are stored in the PImage type.
  • PImage allows us to load and display data.
  • We can also manipulate data:
    • Size
    • Position
    • Opacity
    • Tint
  • image(PImage img, float x, float y, float width, float height) to display an image.
  • Use loadImage(string fname) to load an image.
void setup(){
  size(100, 100);
}

void draw(){
  PImage img;
  image(img, 0, 0);
}
void setup(){
  size(100, 100);
}

void draw(){
  PImage img = loadImage("foo.png");
  image(img, 0, 0);
}
void setup(){
  PImage img = loadImage("foo.png");
  size(100, 100);
}

void draw(){
  image(img, 0, 0);
}
PImage img;

void setup(){
  img = loadImage("foo.png");
  size(100, 100);
}

void draw(){
  image(img, 0, 0);
}

Displaying an Image

Adjusting Window to Image Size

void setup(){
  surface.setResizable(true);
  img = loadImage("foo.png");
  surface.setSize(img.width, img.height);
}

Modifying Pixels

In principle, we can modify pixel data just by modifying the pixels member of PImage.

PImage img;

void setup(){
  // Do setup + loading
}

void draw(){
  for (int i = 0; i < img.pixels.length; i++){
    color c = color(/* Some initialization */);
    img.pixels[i] = c;
  }
}

But this won't always work!

Modifying Pixels

Processing maintains two copies of the data in the image:

  • One is stored in the PImage variable
  • The other is used to actually display the image.

PImage Data

Screen Data

loadPixels()

updatePixels()

PImage Data

Screen Data

loadPixels()

updatePixels()

PImage img;

void setup(){
  // Do setup + loading
}

void draw(){
  img.loadPixels(); // Read updated pixel data into img
  for (int i = 0; i < img.pixels.length; i++){
    color c = color(/* Some initialization */);
    img.pixels[i] = c;
  }
  img.updatePixels(); // Write updated img data to screen
}

Image manipulation might not work without these calls (depends on OS)

How can we access a pixel by its (x,y) value?

0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
18 19 20 21 22 23
24 25 26 27 28 29
30 31 32 33 34 35

How the pixels look on screen

0 1 2 3 4 5 6 7 8 9

How the pixels are stored in memory

  • Perform a stride into the 1D array to find the row we're looking for.
  • Then use the column to find the final placement in the 1D array.
  • We need to know the image width to do this.
0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
18 19 20 21 22 23
24 25 26 27 28 29
30 31 32 33 34 35
0
1
2
3
4
5
0 1 2 3 4 5

Where is (3,5)?

Where is (5,2)?

Where is (2,0)?

Where is (x,y)?

Traversing by (x,y)

img.loadPixels();
for(int x = 0; x < img.width; x++){
  for(int y = 0; y < img.height; y++){
    int index = x + y * img.width;
    color c = img.pixels[index];
  }
}

Tint

tint() modifies the color of a displayed image.

noTint() disables tint modifications.

size(400,400);
PImage img;
img = loadImage("yuya-onsen.jpg");
image(img, 0, 0);
tint(0, 153, 204);  // Tint blue
image(img, width/2, 0);

The Data Directory

Processing needs one of two things to be true before it loads an image:

  1. You need to give an absolute path to the file, i.e. a filepath that starts with '/' on MacOS + Linux, or a drive letter on Windows.
  2. The file needs to be in a data directory within the Processing sketch.
PImage img1;

void setup(){
  img1 = loadImage("your_image.jpeg");
}

Hands-On: Creating Tint

Recreate Processing's tint() function using a method you create called myTint(). Do not use the existing tint() method.

  1. The method should take RGB data. (OPTIONAL): make it so that you can take either separate RGB arguments or a single color with the same function.
  2. myTint() should be "per image" rather than "per-screen" so you should have an argument for the PImage you will tint.
  3. You do not need to worry about undoing the tint.

If you don't have a suitable picture, check on Canvas for a repository of puppy pictures.

Take the directory which contains your .pde file and your data directory, and ZIP that into an archive. Submit this archive to Canvas.

Kernels and Convolution

Last Time

We can think of colors either in RGB space or in HSV space.

colorMode() switches between the two interpretations.

Last Time

0 1 2 3 4 5
6 7 8 9 10 11
12 13 14 15 16 17
18 19 20 21 22 23
24 25 26 27 28 29
30 31 32 33 34 35
0 1 2 3 4 5 6 7 8 9

Pixels are stored in a linear array in memory but displayed in a 2D grid on screen. We need to do some work to interconvert between the two.

Last Time

The PImage class is used to work with images in Processing. We need to call loadPixels or updatePixels to make sure changes propogate to the screen.

PImage img;

void setup(){
  // Do setup + loading
}

void draw(){
  img.loadPixels(); // Read updated pixel data into img
  for (int i = 0; i < img.pixels.length; i++){
    color c = color(/* Some initialization */);
    img.pixels[i] = c;
  }
  img.updatePixels(); // Write updated img data to screen
}

Pixels are stored in a pixel array.

Questions

Can you rearrange the pixels in an image?

Yes, we can do things like randomize the order of pixels in an image by just randomly swapping pairs of pixels.

Of course, this won't look much like an image anymore.

A lot of image modifications (e.g. image rotation, cropping, etc) rely on either changing the order of pixels or getting rid of some pixels altogether.

More Image Questions

How do you access the width or height of an image?

Can we tint only a certain part of an image?

// PixelSet is a custom structure (pseudocode)
void myTint(PImage img, color tintColor, PixelSet tintTargets){
  for (int i = 0; i < img.width; i++){
    for (int j = 0; j < img.height; j++){
      if (tintTargets.contains(i,j)){
        // Tint pixel
      } else {
        // Do nothing
      }}}}  // Vertical space saving

When do you need to call updatePixels()?

Whenever you modify the pixels array of an image.

Can we have images that aren't squares or rectangles?

Yes and no.

 

  • Yes: We can have transparent pixels, or refuse to draw some of the pixels in an image.
  • No: an image is fundamentally a layout of pixels. If you don't have that, you still have a picture, but you're no longer discussing an image.

What do the numbers in the kernels mean?

In some sense, an image kernel tries to extract some information from the neighborhood of a kernel.

Examples:

  • Is this area part of an edge?
  • What is the average value in this area?
  • Is the center pixel brighter than its neighbors?

We then draw this information out as a new image.

The numbers in the kernel are chosen based off of what information we want to extract.

More Kernel Questions

Do kernels make adjustments to the saturation of an image? Or maybe brightness?

Fantastic question! I've reworked this part of class to hopefully be more clear---please re-ask if you're still confused.

Are there kernels of different sizes? Are there kernels that have different width/height?

Yes. You decide the kernel parameters---you could make it 1 tall and 4 wide if you wanted to.

How do negative values in the kernel work?

We need to modify them somehow before displaying.

Are there any other kernels useful in image processing?

Miscellaneous

Why does draw() seem to return stuff if it's a void function?

Does excusing an attendance excuse the hands-on as well?

No.

void printFive(){
  println(5);
}

int x = printFive(); // ???

The Values In Your Neighborhood

They're the values that you meet when you're walking down the street...

The manipulations we have seen so far are per-pixel. Output pixel values only depend on the input pixel values.

Example: Grayscale image. How can we find a single value that captures the information of three color channels?

Some manipulations are local: require information about neighboring pixels as well.

Example: Increase contrast between pixels.

Image Kernel

Can go by several names:

  • Kernel
  • Convolution Matrix
  • Mask

A small matrix which is used to compute some information about the image in a local region.

Almost always small (3x3, 5x5) and square.

Image Kernel

Once we have this information, we will often re-display the image using the information computed by the kernel.

This leads to a discussion of image kernels as "image modification"....which it is!

 

But the kernel itself is just trying to compute local image information.

Convolution

  1. Multiply corresponding cells
  2. Sum the resulting values

NOT a matrix multiply!!

Reasoning About Kernels

Step 1: Grayscale

Think of an image as an array of grayscale pixels: each pixel has a single value in the range 0-255.

1 1 1
1 1 1
1 1 1

What Does This Kernel Compute?

\(\frac{1}{9}\)

Averages all pixels in the neighborhood: blurs the image slightly. This kernel is known as the box blur.

-1 0 1
-2 0 2
-1 0 1
-1 0 1
-2 0 2
-1 0 1
-1 0 1
-2 0 2
-1 0 1
-1 0 1
-2 0 2
-1 0 1
-1 0 1
-2 0 2
-1 0 1

Blacks out the result unless there is a vertical edge in the image. This is known as a Sobel filter or an edge-detection filter.

-1 -2 -1
0 0 0
1 2 1
-1 -2 -1
0 0 0
1 2 1
-1 -2 -1
0 0 0
1 2 1
-1 -2 -1
0 0 0
1 2 1
-1 -2 -1
0 0 0
1 2 1

Horizontal edge detector

-1 -2 -1
0 0 0
1 2 1
-1 0 1
-2 0 2
-1 0 1
1 2 1
2 4 2
1 2 1

\(\frac{1}{16}\)

Known as a Gaussian blur

0 -1 0
-1 5 -1
0 -1 0

What does this kernel do?

If center pixel is brighter than others, boost its brightness.

0 -1 0
-1 5 -1
0 -1 0
0 -1 0
-1 5 -1
0 -1 0

If center pixel is darker than others, lower its brightness.

Effect: emphasizes pixels that are brighter than their neighbors.

Kernels on Color Images

How can we deal with convolutions on color images if we can't add or multiply colors?

These are all grayscale images!

Applying Convolutions: Code

Step 1: Apply kernel to single pixel

void applyKernelTo(PImage img, int x, int y){
  // ?
}

Need to access neighborhood of x, y to obtain image values, and same for kernel.

float[] kernel = {0, -1, 0, -1, 5, -1, 0, -1, 0};

void applyKernelTo(PImage img, int x, int y){
  for(int x_off = -1; x_off <= 1; x_off++){
    for(int y_off = -1; y_off <= 1; y_off++ ){
      
    }
  }
}

float[] kernel = {0, -1, 0, -1, 5, -1, 0, -1, 0};

void applyKernelTo(PImage img, int x, int y){
  for(int x_off = -1; x_off <= 1; x_off++){
    for(int y_off = -1; y_off <= 1; y_off++ ){
      int img_index = (y + y_off) * img.width + (x + x_off);
      int ker_index = (1 + y_off) * 3 + (1 + x_off);

      // ?
    }
  }
}

float[] kernel = {0, -1, 0, -1, 5, -1, 0, -1, 0};

void applyKernelTo(PImage img, int x, int y){
  // Declare and initialize final_red, final_green, final_blue
  for(int x_off = -1; x_off <= 1; x_off++){
    for(int y_off = -1; y_off <= 1; y_off++ ){
      int img_index = (y + y_off) * img.width + (x + x_off);
      int ker_index = (1 + y_off) * 3 + (1 + x_off);
      
      float red = img.pixels[img_index];
      final_red += red * kernel[ker_index];
      
      // Perform similar operations for green and blue;
    }
  }
  final_red = constrain(red, 0, 255);  // Why?
  // Also clamp green and blue
  color final_color = color(final_red, final_green, final_blue);
  img[x + img.width * y] = final_color;
}

Why can't we store the new value in the existing image?

Intermediate Buffer

  • Array of pixels which matches the size of the image
  • Provides a safe location for storing image data
  • Allows program to preserve original image data
  • Common trick to increase speed of rendering (double buffering)

Arithmetic happens here

Write back to intermediate buffer

Creating a Buffer

There are a few ways to create a duplicate image.

 

  • The simplest (when possible) is just to load the image twice:
    PImage img = loadImage(image_file);
    PImage buf = loadImage(image_file);
     
  • You can also create a blank image:
    PImage buf = createImage(width, height, ARGB);
     
  • If necessary, copy pixel values between the images:
    copy(img, x, y, width, height, x, y, width, height);

Beginner's Trap

PImage img = loadImage(image_file);
PImage buf = img;

This does not create a full copy of the image! This is called a shallow copy of the data. Use the techniques on the previous slide to get a deep copy.

img

buf

Shallow Copy

Deep Copy

data

img

buf

data

img

buf

data

data

So Far

  • Use function to compute the convolution at a single pixel location.
  • Within function, use for-loops over offsets to compute result of convolution.
  • Use 2D array math to compute indices into image and kernel.
  • Read data from image and compute convolution value.
  • Clamp final values into valid range.
  • Write final value to intermediate buffer (you will need to change the code to accomplish this!).
  • Call this function on every location in the image.
float[] kernel = {0, -1, 0, -1, 5, -1, 0, -1, 0};

void applyKernelTo(PImage img, int x, int y){
  // Declare and initialize final_red, final_green, final_blue
  for(int x_off = -1; x_off <= 1; x_off++){
    for(int y_off = -1; y_off <= 1; y_off++ ){
      int img_index = (y + y_off) * img.width + (x + x_off);
      int ker_index = (1 + y_off) * 3 + (1 + x_off);
      
      float red = img.pixels[img_index];
      final_red += red * kernel[ker_index];
      
      // Perform similar operations for green and blue;
    }
  }
  final_red = constrain(red, 0, 255);  // Why?
  // Also clamp green and blue
  color final_color = color(final_red, final_green, final_blue);
  img[x + img.width * y] = final_color; // Broken
}

Literal Edge Cases

Strategies for dealing with this:

  • Don't apply convolution to edges or corners
  • Missing values are filled in with a default (0 or 255)
  • Wrap missing pixels from the other side of the image
  • Mirror missing pixels from the other side of the kernel

applyKernelTo(img, 0, 0)

Hands-On: Image Kernels

  1. Take your "sharpen" kernel and store it in a variable.
     
  2. Create a new image buffer to store the final, convolved image data.
     
  3. Apply the sharpen kernel to an image and store the convolved data into your secondary image buffer.
     
  4. Display the convolved buffer to the screen.

Index Cards!

  1. Your name and EID.
     
  2. One thing that you learned from class today. You are allowed to say "nothing" if you didn't learn anything.
     
  3. One question you have about something covered in class today. You may not respond "nothing".
     
  4. (Optional) Any other comments/questions/thoughts about today's class.