Seam Carving for Content-Aware Image Resizing

Shai Avidan, Ariel Shamir

SIGRAPH '07

DOI 10.1145/1275808.1276390

Joshua Horacsek

Motivation

Resizing an image is a very common task
Huge diversity of display devices makes content adaption non-obvious and non-trivial
Dynamic layouts - think HTML web pages
Uniform scaling is not the answer (distortions galore)
We want something like this...

Thoughts?

How would you resize an image?

Scale
Crop
Look for some "salience" information, then scale or crop according to that?

Towards a Solution

This can either add "new" information, or remove information. Focus for now, on removing pixels
We should really only remove pixels from an image that don't contribute to an image
That is, we're free to remove things that we won't "notice". What does a pixel need to be "noticable"?

Define an "energy" term for pixels, but what's an appropriate "energy" for a pixel? Low if it blends in with its surrounding area, high if it doesn't

Towards a Solution

e_1(I) = |\frac{\partial}{\partial x} I| + |\frac{\partial}{\partial y} I|

e_1(I) = |\frac{\partial}{\partial x} I| + |\frac{\partial}{\partial y} I|

This helps us quantify energy, but is not a strategy for removing pixels

Towards a Solution

A few other energy terms

L2 Norm of the gradient
Histogram of gradient normalized energy
Entropy

Towards a Solution

Example image and energy map

Strategies for Removal

Crop to the region of highest energy
Remove low energy columns
Remove low energy pixels from rows
Remove any low energy pixels from the image

Crop

Column

Row

Pixel

Column is pretty good, but motivates a better approach

Strategies for Removal

(Ignore the indigo line for now)

Column is pretty good, but motivates a better approach

Seams

A connected paths down the image sounds like a good idea.

s^x = \{(x(i),i)\}^n_{i=1}\}

s^x = \{(x(i),i)\}^n_{i=1}\}

\forall i, |x(i) - x(i-1)| \le 1

\forall i, |x(i) - x(i-1)| \le 1

such that

Formally, for an n by m image, a horizontal seam is defined as

Seams

Vertical seams are similar

s^y = \{(j,y(j))\}^m_{j=1}\}

s^y = \{(j,y(j))\}^m_{j=1}\}

\forall j, |y(j) - y(j-1)| \le 1

\forall j, |y(j) - y(j-1)| \le 1

such that

Seams

Using the intuition from the energy , should find a seam that minimizes energy along said seam

s^* = \ \displaystyle\min_s \sum^n_{i=1}e(I(s_i))

s^* = \ \displaystyle\min_s \sum^n_{i=1}e(I(s_i))

Done via dynamic programming. Similar for vertical seams

Let's focus on shrinking images...

Aspect Ratio Changes

Changes along one axis, i.e. an n by m image becomes an n' by m image, where (n-n') > 0

Remove low energy seams

Retargeting

Both axes,an n by m image becomes an n' by m' image, where (n-n') > 0, (m-m') > 0

They have a non-zero intersection, which do you remove first? Is that other seam still a seam after removal?

Retargeting

Redefine this as a minimization process

\displaystyle\min_{s^x, s^y, \alpha} \sum^k_{i=1} E(\alpha_is^x_i + (1-\alpha_i)s^y_i)

\displaystyle\min_{s^x, s^y, \alpha} \sum^k_{i=1} E(\alpha_is^x_i + (1-\alpha_i)s^y_i)

Where

k = r + c, r=(m-m^\prime), c=(n-n^\prime)

k = r + c, r=(m-m^\prime), c=(n-n^\prime)

\alpha_i \in \{0,1\}, \sum^k_{i=1}{\alpha_i}=r \text{ and } \sum^k_{i=1}(1-\alpha_i)=c

\alpha_i \in \{0,1\}, \sum^k_{i=1}{\alpha_i}=r \text{ and } \sum^k_{i=1}(1-\alpha_i)=c

Again, a dynamic programming problem

Enlarging Content

Find the first k optimal seams, then average the content

Find a seam, average content

More applications/variations

Content amplification (seam carve then scale)
Gradient domain seam carving (seam carve with the gradient, then use a Poisson solver to reconstruct the image)
Object removal (tag areas as low or high energy, use in conjunction with face detection algorithms)
Multi-size images (precompute all seams)