Day 38:

Convolution operators

Fourier Transform of a Cyclic Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]

then we already computed

Now, we take the Fourier Transform of \(x\circledast y\) and we get

\[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 & 1 & 1\\ 1 & \omega & \omega^2\\ 1 & \omega^2 & \omega^4\end{bmatrix} \begin{bmatrix} 11\\ 11\\ 5\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 11+ 11+ 5\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega^4\end{bmatrix}\]

\[=\frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]

Remember:

  • \(\omega^3=1\)
  • \(1+\omega+\omega^2=0\)

Example continued. Now, we take the Fourier Transform of \(x\circledast y\) and we get \[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]

\[F_{3}x = \frac{1}{\sqrt{3}} \begin{bmatrix} 3\\ 1+2\omega^2\\ 1+2\omega \end{bmatrix}\ \text{and}\ F_{3}y = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 1+5\omega+3\omega^2\\ 1+5\omega^2+3\omega \end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 4\omega+2\omega^2\\ 4\omega^2+2\omega \end{bmatrix}\]

\((1+2\omega^2)(4\omega+2\omega^2) = 4\omega+2\omega^2+8\omega^3+4\omega^4 = 8+8\omega+2\omega^2 = -6\omega^2\)

\((1+2\omega)(4\omega^2+2\omega) = 4\omega^2+2\omega+8\omega^3+4\omega^2 = 8+2\omega+8\omega^2 = -6\omega\)

\[F_{3}x\odot F_{3}y \]

Fourier Transform of a Cyclic Convolution

This is the entrywise product of two matrices of the same size. This is sometimes called the Hadamard product.

\[= \frac{1}{3}\begin{bmatrix} 3\cdot 9\\ (1+2\omega^2)(4+2\omega^2)\\ (1+2\omega)(4+2\omega)\end{bmatrix} = \frac{1}{3}\begin{bmatrix} 27\\ -6\omega^2\\ -6\omega\end{bmatrix} = \frac{1}{\sqrt{3}}F_{3}(x\circledast y)\]

Lemma. Let 

\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]

\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]

Define the polynomials

\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]

\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}\]

Then,

\[f(t)g(t) = c_{0} + c_{1}t+c_{2}t^2+\cdots + c_{2N-2}t^{2N-2}\]

where

\[[c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2N-2}]^{\top} = x\ast y.\]

If \(\omega = e^{-2\pi i/N}\), then

\[f(\omega^{k})g(\omega^{k}) = d_{0} + d_{1}\omega^{k} + d_{2}\omega^{2k} + \cdots + d_{N-1}\omega^{(N-1)k}\]

where

\[[d_{0}\ \ d_{1}\ \ d_{2}\ \ \cdots\ \ d_{N-1}]^{\top} = x\circledast y.\]

Theorem. If \(x,y\in\mathbb{C}^{N}\), then

\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]

Fourier Transform of a Cyclic Convolution

Proof. Let 

\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]

\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]

\[x\circledast y = \begin{bmatrix} d_{0} & d_{1} & d_{2} & \cdots & d_{N-1}\end{bmatrix}^{\top}\]

Define the polynomials

\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]

\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}.\]

Notice that 

\[F_{N}x = \frac{1}{\sqrt{N}}\begin{bmatrix} f(1) & f(\omega) & f(\omega^{2}) & \cdots & f(\omega)^{N-1}\end{bmatrix}^{\top}\]

\[F_{N}y = \frac{1}{\sqrt{N}}\begin{bmatrix} g(1) & g(\omega) & g(\omega^{2}) & \cdots & g(\omega)^{N-1}\end{bmatrix}^{\top}\]

Fourier Transform of a Cyclic Convolution

Proof continued. Hence, if we define

\[h(t) = d_{0} + d_{1}t + d_{2}t^2 + \cdots + d_{N-1}t^{N-1},\]

then by the lemma we have

\[F_{N}x\odot F_{N}y= \]

\[= \frac{1}{N}\begin{bmatrix} f(1)g(1) & f(\omega)g(\omega) & f(\omega^{2})g(\omega^{2}) & \cdots & f(\omega^{N-1})g(\omega^{N-1})\end{bmatrix}^{\top}\]

\[= \frac{1}{\sqrt{N}}\left(\frac{1}{\sqrt{N}}\begin{bmatrix} h(1) & h(\omega) & h(\omega)^{2} & \cdots & h(\omega^{N-1})\end{bmatrix}^{\top}\right)\]

\[= \frac{1}{\sqrt{N}}\left(F_{N}(x\circledast y)\right).\ \Box\]

Convolution vs. Cyclic Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

\[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]

 

\[\begin{array}{ccc} 1 &  0 &  2\\ 1 &  5 &  3\\ \hline\end{array}\]

\[\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}\]

Convolution:

\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]

 

\[\begin{array}{ccc} 1 &  0 &   2\\ 1 &  5 &   3\\ \hline\end{array}\]

\[\begin{array}{ccc} 0 & 6 & 3\\ 10 & 5 & 0\\ 1 & 0 & 2\\\hline 11 & 11 & 5\end{array}\]

Cyclic convolution:

Convolution vs. Cyclic Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

Convolution:

Cyclic convolution: Let \(P\) be the \(3\times 3\) upward shift matrix:

\[(1+0x+2x^2)(1+5x+3x^2) = 1+5x+5x^2+10x^3+6x^4\]

\[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]

\[(1I+0P+2P^2)(I+5P+3P^2) = 1I+5P+5P^2+10P^3+6P^4\]

\[ = 1I+5P+5P^2+10I + 6P\]

\[ = 11I+11P+5P^2\]

\[x\circledast y = \begin{bmatrix} 11 & 11 & 5 \end{bmatrix}^{\top}\]

Example continued. Set

\[\tilde{x}=\begin{bmatrix} 1\\ 0\\ 2\\ 0\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} 1\\ 5\\ 3\\ 0\\ 0\end{bmatrix}\]

Let \(\tilde{P}\) be the \(5\times 5\) upward shift matrix:

\((1I+0\tilde{P}+2\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4)(1I+5\tilde{P}+3\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4) \)

\( = (1I+0\tilde{P}+2\tilde{P}^2)(1I+5\tilde{P}+3\tilde{P}^{2})\)

\( = 1I + 5\tilde{P} + 5\tilde{P}^2 + 10\tilde{P}^3+6\tilde{P}^4\)

\[\tilde{x}\circledast \tilde{y} = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top} = x\ast y\]

Convolution vs. Cyclic Convolution

Let \(\tilde{x},\tilde{y}\in \mathbb{C}^{2N-1}\) be the "zero padded" versions of \(x\) and \(y\), that is,

Given \(x,y\in\mathbb{C}^{N}\), where

\[x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\  \vdots\\ a_{N-1}\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\  \vdots\\ b_{N-1}\end{bmatrix}\]

\[\tilde{x}=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\  \vdots\\ a_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\  \vdots\\ b_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}\]

Then, \(\tilde{x}\circledast\tilde{y} = x\ast y\).

Using the notation from the previous slide allows us to compute the convolution:

\[x\ast y = \tilde{x}\circledast\tilde{y} = F_{2N-1}^{\ast}F_{2N-1}(\tilde{x}\circledast\tilde{y}) = \sqrt{2N-1}F_{2N-1}^{\ast}\big(F_{2N-1}\tilde{x}\odot F_{2N-1}\tilde{y}\big)\]

Computing convolutions with the Fourier Transform

Theorem (The Convolution Theorem). If \(x,y\in\mathbb{C}^{N}\), then

\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]

Hence, we can compute the cyclic convolution using:

\[x\circledast y = \sqrt{N}F_{N}^{\ast}\big(F_{N}x\odot F_{N}y\big).\]

Convolutions in Pictures

Given a vector \(x\in\R^{N}\) we sometimes think of \(x\) as a function with domain \(\{0,1,\ldots,N-1\}\). That is, if

\[x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\  \vdots\\ a_{N-1}\end{bmatrix}\]

then we write \(x[i] = a_{i}\). Thus, we can graph \(x\).

Convolutions in Pictures

Example. Let \(x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\).

Convolutions in Pictures

Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}\]

Convolutions in Pictures

Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]\]

\(0\cdot \)

\(0\cdot \)

\(1\cdot \)

\(1\cdot \)

\(2\cdot \)

\(1\cdot \)

\(0\cdot\)

To compute \(x\ast y\) we add up the following vectors:

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0\end{array}\right]^{\top}\]

\({}^{\top}\)

Convolutions in Pictures

Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}\]

\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 2 & 4 & 4 & 6 & 2 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]

To compute \(x\ast y\) we add up the following vectors:

\[x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}\]

\(+\)

Convolutions in Pictures

\[x =[0\ \ \ \ 1\ \ \ \ 2\ \ \ \ 2\ \ \ \ 3\ \ \ \ 1\ \ \ \ 0]^{\top}\]

\[y =[0\ \ \ \ 0\ \ \ \ 1\ \ \ \ 1\ \ \ \ 2\ \ \ \ 1\ \ \ \ 0]^{\top}\]

\[x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}\]

Walls are Convolutions

So, the sound on the other side of the wall is a sum of time shifted and muffled (scaled) versions of the same sound.

It's a convolution!

Walls are Convolutions

If \(x\) is a sound on one side of the wall, then there is some \(y\) such that \(x\ast y\) is the sound on the other side.

Recall that \(x\ast y = \tilde{x}\circledast \tilde{y}\)

Given \[\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\]

 

Recall that

\[C(\tilde{y}) = \begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\\ a_{N-1} & a_{0} & a_{1} & \cdots & a_{N-2}\\ a_{N-2} & a_{N-1} & a_{0} & \cdots & a_{N-3}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ a_{1} & a_{2} & a_{3} & \cdots & a_{0} \end{bmatrix}\]

and hence 

\[C(\tilde{y})^{\top}\tilde{x} = \tilde{x}\circledast \tilde{y} = x\ast y\]

Walls are Convolutions

If \(x\) is a sound on one side of the wall, then there is some \(y\) such that  \[x\ast y = \tilde{x}\circledast \tilde{y} = C(\tilde{y})^{\top}\tilde{x}\] is the sound on the other side.

 

\[C(\tilde{y}) = F_{N}\begin{bmatrix} f(\omega^{0}) & 0 & 0 & 0 & \cdots & 0\\ 0 & f(\omega) & 0 & 0 & \cdots & 0\\ 0 & 0 & f(\omega^2) & 0 & \cdots & 0\\ 0 & 0 & 0 & f(\omega^3) & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & f(\omega^{N-1})\end{bmatrix}F_{N}^{\ast}\]

If we define the polynomial

\[f(x) = a_{0} + a_{1}x+a_{2}x^2+a_{3}x^3+\cdots+a_{N-1}t^{N-1},\]

 

where \(\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]\)

Linear Algebra Day 38

By John Jasper

Linear Algebra Day 38

  • 271