# Day 38:

Convolution operators

### Fourier Transform of a Cyclic Convolution

Example. If

$x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}$

$x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}$

Now, we take the Fourier Transform of $$x\circledast y$$ and we get

$F_{3}(x\circledast y) = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 & 1 & 1\\ 1 & \omega & \omega^2\\ 1 & \omega^2 & \omega^4\end{bmatrix} \begin{bmatrix} 11\\ 11\\ 5\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 11+ 11+ 5\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega^4\end{bmatrix}$

$=\frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}$

Remember:

• $$\omega^3=1$$
• $$1+\omega+\omega^2=0$$

Example continued. Now, we take the Fourier Transform of $$x\circledast y$$ and we get $F_{3}(x\circledast y) = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}$

$F_{3}x = \frac{1}{\sqrt{3}} \begin{bmatrix} 3\\ 1+2\omega^2\\ 1+2\omega \end{bmatrix}\ \text{and}\ F_{3}y = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 1+5\omega+3\omega^2\\ 1+5\omega^2+3\omega \end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 4\omega+2\omega^2\\ 4\omega^2+2\omega \end{bmatrix}$

$$(1+2\omega^2)(4\omega+2\omega^2) = 4\omega+2\omega^2+8\omega^3+4\omega^4 = 8+8\omega+2\omega^2 = -6\omega^2$$

$$(1+2\omega)(4\omega^2+2\omega) = 4\omega^2+2\omega+8\omega^3+4\omega^2 = 8+2\omega+8\omega^2 = -6\omega$$

$F_{3}x\odot F_{3}y$

### Fourier Transform of a Cyclic Convolution

This is the entrywise product of two matrices of the same size. This is sometimes called the Hadamard product.

$= \frac{1}{3}\begin{bmatrix} 3\cdot 9\\ (1+2\omega^2)(4+2\omega^2)\\ (1+2\omega)(4+2\omega)\end{bmatrix} = \frac{1}{3}\begin{bmatrix} 27\\ -6\omega^2\\ -6\omega\end{bmatrix} = \frac{1}{\sqrt{3}}F_{3}(x\circledast y)$

Lemma. Let

$x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}$

$y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}$

Define the polynomials

$f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}$

$g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}$

Then,

$f(t)g(t) = c_{0} + c_{1}t+c_{2}t^2+\cdots + c_{2N-2}t^{2N-2}$

where

$[c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2N-2}]^{\top} = x\ast y.$

If $$\omega = e^{-2\pi i/N}$$, then

$f(\omega^{k})g(\omega^{k}) = d_{0} + d_{1}\omega^{k} + d_{2}\omega^{2k} + \cdots + d_{N-1}\omega^{(N-1)k}$

where

$[d_{0}\ \ d_{1}\ \ d_{2}\ \ \cdots\ \ d_{N-1}]^{\top} = x\circledast y.$

Theorem. If $$x,y\in\mathbb{C}^{N}$$, then

$F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).$

### Fourier Transform of a Cyclic Convolution

Proof. Let

$x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}$

$y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}$

$x\circledast y = \begin{bmatrix} d_{0} & d_{1} & d_{2} & \cdots & d_{N-1}\end{bmatrix}^{\top}$

Define the polynomials

$f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}$

$g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}.$

Notice that

$F_{N}x = \frac{1}{\sqrt{N}}\begin{bmatrix} f(1) & f(\omega) & f(\omega^{2}) & \cdots & f(\omega)^{N-1}\end{bmatrix}^{\top}$

$F_{N}y = \frac{1}{\sqrt{N}}\begin{bmatrix} g(1) & g(\omega) & g(\omega^{2}) & \cdots & g(\omega)^{N-1}\end{bmatrix}^{\top}$

### Fourier Transform of a Cyclic Convolution

Proof continued. Hence, if we define

$h(t) = d_{0} + d_{1}t + d_{2}t^2 + \cdots + d_{N-1}t^{N-1},$

then by the lemma we have

$F_{N}x\odot F_{N}y=$

$= \frac{1}{N}\begin{bmatrix} f(1)g(1) & f(\omega)g(\omega) & f(\omega^{2})g(\omega^{2}) & \cdots & f(\omega^{N-1})g(\omega^{N-1})\end{bmatrix}^{\top}$

$= \frac{1}{\sqrt{N}}\left(\frac{1}{\sqrt{N}}\begin{bmatrix} h(1) & h(\omega) & h(\omega)^{2} & \cdots & h(\omega^{N-1})\end{bmatrix}^{\top}\right)$

$= \frac{1}{\sqrt{N}}\left(F_{N}(x\circledast y)\right).\ \Box$

### Convolution vs. Cyclic Convolution

Example. If

$x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}$

$x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}$

$\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}$

$\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}$

Convolution:

$x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}$

$\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}$

$\begin{array}{ccc} 0 & 6 & 3\\ 10 & 5 & 0\\ 1 & 0 & 2\\\hline 11 & 11 & 5\end{array}$

Cyclic convolution:

### Convolution vs. Cyclic Convolution

Example. If

$x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}$

Convolution:

Cyclic convolution: Let $$P$$ be the $$3\times 3$$ upward shift matrix:

$(1+0x+2x^2)(1+5x+3x^2) = 1+5x+5x^2+10x^3+6x^4$

$x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}$

$(1I+0P+2P^2)(I+5P+3P^2) = 1I+5P+5P^2+10P^3+6P^4$

$= 1I+5P+5P^2+10I + 6P$

$= 11I+11P+5P^2$

$x\circledast y = \begin{bmatrix} 11 & 11 & 5 \end{bmatrix}^{\top}$

Example continued. Set

$\tilde{x}=\begin{bmatrix} 1\\ 0\\ 2\\ 0\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} 1\\ 5\\ 3\\ 0\\ 0\end{bmatrix}$

Let $$\tilde{P}$$ be the $$5\times 5$$ upward shift matrix:

$$(1I+0\tilde{P}+2\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4)(1I+5\tilde{P}+3\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4)$$

$$= (1I+0\tilde{P}+2\tilde{P}^2)(1I+5\tilde{P}+3\tilde{P}^{2})$$

$$= 1I + 5\tilde{P} + 5\tilde{P}^2 + 10\tilde{P}^3+6\tilde{P}^4$$

$\tilde{x}\circledast \tilde{y} = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top} = x\ast y$

### Convolution vs. Cyclic Convolution

Let $$\tilde{x},\tilde{y}\in \mathbb{C}^{2N-1}$$ be the "zero padded" versions of $$x$$ and $$y$$, that is,

Given $$x,y\in\mathbb{C}^{N}$$, where

$x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\end{bmatrix}$

$\tilde{x}=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}$

Then, $$\tilde{x}\circledast\tilde{y} = x\ast y$$.

Using the notation from the previous slide allows us to compute the convolution:

$x\ast y = \tilde{x}\circledast\tilde{y} = F_{2N-1}^{\ast}F_{2N-1}(\tilde{x}\circledast\tilde{y}) = \sqrt{2N-1}F_{2N-1}^{\ast}\big(F_{2N-1}\tilde{x}\odot F_{2N-1}\tilde{y}\big)$

### Computing convolutions with the Fourier Transform

Theorem (The Convolution Theorem). If $$x,y\in\mathbb{C}^{N}$$, then

$F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).$

Hence, we can compute the cyclic convolution using:

$x\circledast y = \sqrt{N}F_{N}^{\ast}\big(F_{N}x\odot F_{N}y\big).$

### Convolutions in Pictures

Given a vector $$x\in\R^{N}$$ we sometimes think of $$x$$ as a function with domain $$\{0,1,\ldots,N-1\}$$. That is, if

$x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}$

then we write $$x[i] = a_{i}$$. Thus, we can graph $$x$$.

### Convolutions in Pictures

Example. Let $$x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}$$.

### Convolutions in Pictures

Example. Let $x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}$

### Convolutions in Pictures

Example. Let $x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]$

$$0\cdot$$

$$0\cdot$$

$$1\cdot$$

$$1\cdot$$

$$2\cdot$$

$$1\cdot$$

$$0\cdot$$

To compute $$x\ast y$$ we add up the following vectors:

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0\end{array}\right]^{\top}$

$${}^{\top}$$

### Convolutions in Pictures

Example. Let $x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}$

$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 2 & 4 & 4 & 6 & 2 & 0 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}$$\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}$

To compute $$x\ast y$$ we add up the following vectors:

$x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}$

$$+$$

### Convolutions in Pictures

$x =[0\ \ \ \ 1\ \ \ \ 2\ \ \ \ 2\ \ \ \ 3\ \ \ \ 1\ \ \ \ 0]^{\top}$

$y =[0\ \ \ \ 0\ \ \ \ 1\ \ \ \ 1\ \ \ \ 2\ \ \ \ 1\ \ \ \ 0]^{\top}$

$x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}$

### Walls are Convolutions

So, the sound on the other side of the wall is a sum of time shifted and muffled (scaled) versions of the same sound.

It's a convolution!

### Walls are Convolutions

If $$x$$ is a sound on one side of the wall, then there is some $$y$$ such that $$x\ast y$$ is the sound on the other side.

Recall that $$x\ast y = \tilde{x}\circledast \tilde{y}$$

Given $\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}$

Recall that

$C(\tilde{y}) = \begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\\ a_{N-1} & a_{0} & a_{1} & \cdots & a_{N-2}\\ a_{N-2} & a_{N-1} & a_{0} & \cdots & a_{N-3}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ a_{1} & a_{2} & a_{3} & \cdots & a_{0} \end{bmatrix}$

and hence

$C(\tilde{y})^{\top}\tilde{x} = \tilde{x}\circledast \tilde{y} = x\ast y$

### Walls are Convolutions

If $$x$$ is a sound on one side of the wall, then there is some $$y$$ such that  $x\ast y = \tilde{x}\circledast \tilde{y} = C(\tilde{y})^{\top}\tilde{x}$ is the sound on the other side.

$C(\tilde{y}) = F_{N}\begin{bmatrix} f(\omega^{0}) & 0 & 0 & 0 & \cdots & 0\\ 0 & f(\omega) & 0 & 0 & \cdots & 0\\ 0 & 0 & f(\omega^2) & 0 & \cdots & 0\\ 0 & 0 & 0 & f(\omega^3) & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & f(\omega^{N-1})\end{bmatrix}F_{N}^{\ast}$

If we define the polynomial

$f(x) = a_{0} + a_{1}x+a_{2}x^2+a_{3}x^3+\cdots+a_{N-1}t^{N-1},$

where $$\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]$$

By John Jasper

• 271