Seam Carving for Content-Aware Image Resizing
Shai Avidan, Ariel Shamir
SIGRAPH '07
Joshua Horacsek
Motivation
- Resizing an image is a very common task
- Huge diversity of display devices makes content adaption non-obvious and non-trivial
- Dynamic layouts - think HTML web pages
- Uniform scaling is not the answer (distortions galore)
- We want something like this...
Thoughts?
How would you resize an image?
- Scale
- Crop
- Look for some "salience" information, then scale or crop according to that?
Towards a Solution
- This can either add "new" information, or remove information. Focus for now, on removing pixels
- We should really only remove pixels from an image that don't contribute to an image
- That is, we're free to remove things that we won't "notice". What does a pixel need to be "noticable"?
Define an "energy" term for pixels, but what's an appropriate "energy" for a pixel? Low if it blends in with its surrounding area, high if it doesn't
Towards a Solution
e_1(I) = |\frac{\partial}{\partial x} I| + |\frac{\partial}{\partial y} I|
This helps us quantify energy, but is not a strategy for removing pixels
Towards a Solution
A few other energy terms
- L2 Norm of the gradient
- Histogram of gradient normalized energy
- Entropy
Towards a Solution
Example image and energy map
Strategies for Removal
- Crop to the region of highest energy
- Remove low energy columns
- Remove low energy pixels from rows
- Remove any low energy pixels from the image
Crop
Column
Row
Pixel
Column is pretty good, but motivates a better approach
Strategies for Removal
(Ignore the indigo line for now)
Column is pretty good, but motivates a better approach
Seams
A connected paths down the image sounds like a good idea.
s^x = \{(x(i),i)\}^n_{i=1}\}
\forall i, |x(i) - x(i-1)| \le 1
such that
Formally, for an n by m image, a horizontal seam is defined as
Seams
Vertical seams are similar
s^y = \{(j,y(j))\}^m_{j=1}\}
\forall j, |y(j) - y(j-1)| \le 1
such that
Seams
Using the intuition from the energy , should find a seam that minimizes energy along said seam
s^* = \ \displaystyle\min_s \sum^n_{i=1}e(I(s_i))
Done via dynamic programming. Similar for vertical seams
Let's focus on shrinking images...
Aspect Ratio Changes
Changes along one axis, i.e. an n by m image becomes an n' by m image, where (n-n') > 0
Remove low energy seams
Retargeting
Both axes,an n by m image becomes an n' by m' image, where (n-n') > 0, (m-m') > 0
They have a non-zero intersection, which do you remove first? Is that other seam still a seam after removal?
Retargeting
Redefine this as a minimization process
\displaystyle\min_{s^x, s^y, \alpha} \sum^k_{i=1} E(\alpha_is^x_i + (1-\alpha_i)s^y_i)
Where
k = r + c, r=(m-m^\prime), c=(n-n^\prime)
\alpha_i \in \{0,1\}, \sum^k_{i=1}{\alpha_i}=r \text{ and } \sum^k_{i=1}(1-\alpha_i)=c
Again, a dynamic programming problem
Enlarging Content
Find the first k optimal seams, then average the content
Find a seam, average content
More applications/variations
- Content amplification (seam carve then scale)
- Gradient domain seam carving (seam carve with the gradient, then use a Poisson solver to reconstruct the image)
- Object removal (tag areas as low or high energy, use in conjunction with face detection algorithms)
- Multi-size images (precompute all seams)
Results
Content amplification
Enlargement
Results
Gradient Domain Carving
Results
Person removal
Results
Face detection
Results
Future Work
Possible extensions of the work
- Resizing video
- Combining seam carving and scaling
- Better seam choice for multi-size images
- Better perceptually based energy function (based on cone response)
- Scale based carving (maybe based on some wavelet decomposition?)
- Carving on indicator functions? In 3D, what does this do to a model
Discussion
Seam Carving
By Joshua Horacsek
