Day 38:
Convolution operators
Fourier Transform of a Cyclic Convolution
Example. If
\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]
\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]
then we already computed
Now, we take the Fourier Transform of \(x\circledast y\) and we get
\[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 & 1 & 1\\ 1 & \omega & \omega^2\\ 1 & \omega^2 & \omega^4\end{bmatrix} \begin{bmatrix} 11\\ 11\\ 5\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 11+ 11+ 5\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega^4\end{bmatrix}\]
\[=\frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]
Remember:
- \(\omega^3=1\)
- \(1+\omega+\omega^2=0\)
Example continued. Now, we take the Fourier Transform of \(x\circledast y\) and we get \[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]
\[F_{3}x = \frac{1}{\sqrt{3}} \begin{bmatrix} 3\\ 1+2\omega^2\\ 1+2\omega \end{bmatrix}\ \text{and}\ F_{3}y = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 1+5\omega+3\omega^2\\ 1+5\omega^2+3\omega \end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 4\omega+2\omega^2\\ 4\omega^2+2\omega \end{bmatrix}\]
\((1+2\omega^2)(4\omega+2\omega^2) = 4\omega+2\omega^2+8\omega^3+4\omega^4 = 8+8\omega+2\omega^2 = -6\omega^2\)
\((1+2\omega)(4\omega^2+2\omega) = 4\omega^2+2\omega+8\omega^3+4\omega^2 = 8+2\omega+8\omega^2 = -6\omega\)
\[F_{3}x\odot F_{3}y \]
Fourier Transform of a Cyclic Convolution
This is the entrywise product of two matrices of the same size. This is sometimes called the Hadamard product.
\[= \frac{1}{3}\begin{bmatrix} 3\cdot 9\\ (1+2\omega^2)(4+2\omega^2)\\ (1+2\omega)(4+2\omega)\end{bmatrix} = \frac{1}{3}\begin{bmatrix} 27\\ -6\omega^2\\ -6\omega\end{bmatrix} = \frac{1}{\sqrt{3}}F_{3}(x\circledast y)\]
Lemma. Let
\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]
\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]
Define the polynomials
\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]
\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}\]
Then,
\[f(t)g(t) = c_{0} + c_{1}t+c_{2}t^2+\cdots + c_{2N-2}t^{2N-2}\]
where
\[[c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2N-2}]^{\top} = x\ast y.\]
If \(\omega = e^{-2\pi i/N}\), then
\[f(\omega^{k})g(\omega^{k}) = d_{0} + d_{1}\omega^{k} + d_{2}\omega^{2k} + \cdots + d_{N-1}\omega^{(N-1)k}\]
where
\[[d_{0}\ \ d_{1}\ \ d_{2}\ \ \cdots\ \ d_{N-1}]^{\top} = x\circledast y.\]
Theorem. If \(x,y\in\mathbb{C}^{N}\), then
\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]
Fourier Transform of a Cyclic Convolution
Proof. Let
\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]
\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]
\[x\circledast y = \begin{bmatrix} d_{0} & d_{1} & d_{2} & \cdots & d_{N-1}\end{bmatrix}^{\top}\]
Define the polynomials
\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]
\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}.\]
Notice that
\[F_{N}x = \frac{1}{\sqrt{N}}\begin{bmatrix} f(1) & f(\omega) & f(\omega^{2}) & \cdots & f(\omega)^{N-1}\end{bmatrix}^{\top}\]
\[F_{N}y = \frac{1}{\sqrt{N}}\begin{bmatrix} g(1) & g(\omega) & g(\omega^{2}) & \cdots & g(\omega)^{N-1}\end{bmatrix}^{\top}\]
Fourier Transform of a Cyclic Convolution
Proof continued. Hence, if we define
\[h(t) = d_{0} + d_{1}t + d_{2}t^2 + \cdots + d_{N-1}t^{N-1},\]
then by the lemma we have
\[F_{N}x\odot F_{N}y= \]
\[= \frac{1}{N}\begin{bmatrix} f(1)g(1) & f(\omega)g(\omega) & f(\omega^{2})g(\omega^{2}) & \cdots & f(\omega^{N-1})g(\omega^{N-1})\end{bmatrix}^{\top}\]
\[= \frac{1}{\sqrt{N}}\left(\frac{1}{\sqrt{N}}\begin{bmatrix} h(1) & h(\omega) & h(\omega)^{2} & \cdots & h(\omega^{N-1})\end{bmatrix}^{\top}\right)\]
\[= \frac{1}{\sqrt{N}}\left(F_{N}(x\circledast y)\right).\ \Box\]
Convolution vs. Cyclic Convolution
Example. If
\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]
\[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]
\[\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}\]
\[\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}\]
Convolution:
\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]
\[\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}\]
\[\begin{array}{ccc} 0 & 6 & 3\\ 10 & 5 & 0\\ 1 & 0 & 2\\\hline 11 & 11 & 5\end{array}\]
Cyclic convolution:
Convolution vs. Cyclic Convolution
Example. If
\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]
Convolution:
Cyclic convolution: Let \(P\) be the \(3\times 3\) upward shift matrix:
\[(1+0x+2x^2)(1+5x+3x^2) = 1+5x+5x^2+10x^3+6x^4\]
\[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]
\[(1I+0P+2P^2)(I+5P+3P^2) = 1I+5P+5P^2+10P^3+6P^4\]
\[ = 1I+5P+5P^2+10I + 6P\]
\[ = 11I+11P+5P^2\]
\[x\circledast y = \begin{bmatrix} 11 & 11 & 5 \end{bmatrix}^{\top}\]
Example continued. Set
\[\tilde{x}=\begin{bmatrix} 1\\ 0\\ 2\\ 0\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} 1\\ 5\\ 3\\ 0\\ 0\end{bmatrix}\]
Let \(\tilde{P}\) be the \(5\times 5\) upward shift matrix:
\((1I+0\tilde{P}+2\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4)(1I+5\tilde{P}+3\tilde{P}^2+0\tilde{P}^3+0\tilde{P}^4) \)
\( = (1I+0\tilde{P}+2\tilde{P}^2)(1I+5\tilde{P}+3\tilde{P}^{2})\)
\( = 1I + 5\tilde{P} + 5\tilde{P}^2 + 10\tilde{P}^3+6\tilde{P}^4\)
\[\tilde{x}\circledast \tilde{y} = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top} = x\ast y\]
Convolution vs. Cyclic Convolution
Let \(\tilde{x},\tilde{y}\in \mathbb{C}^{2N-1}\) be the "zero padded" versions of \(x\) and \(y\), that is,
Given \(x,y\in\mathbb{C}^{N}\), where
\[x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\end{bmatrix}\]
\[\tilde{x}=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}\quad\text{and}\quad \tilde{y} = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\\ 0\\ \vdots\\ 0\end{bmatrix}\]
Then, \(\tilde{x}\circledast\tilde{y} = x\ast y\).
Using the notation from the previous slide allows us to compute the convolution:
\[x\ast y = \tilde{x}\circledast\tilde{y} = F_{2N-1}^{\ast}F_{2N-1}(\tilde{x}\circledast\tilde{y}) = \sqrt{2N-1}F_{2N-1}^{\ast}\big(F_{2N-1}\tilde{x}\odot F_{2N-1}\tilde{y}\big)\]
Computing convolutions with the Fourier Transform
Theorem (The Convolution Theorem). If \(x,y\in\mathbb{C}^{N}\), then
\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]
Hence, we can compute the cyclic convolution using:
\[x\circledast y = \sqrt{N}F_{N}^{\ast}\big(F_{N}x\odot F_{N}y\big).\]
Convolutions in Pictures
Given a vector \(x\in\R^{N}\) we sometimes think of \(x\) as a function with domain \(\{0,1,\ldots,N-1\}\). That is, if
\[x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}\]
then we write \(x[i] = a_{i}\). Thus, we can graph \(x\).
Convolutions in Pictures
Example. Let \(x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\).
Convolutions in Pictures
Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}\]
Convolutions in Pictures
Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]\]
\(0\cdot \)
\(0\cdot \)
\(1\cdot \)
\(1\cdot \)
\(2\cdot \)
\(1\cdot \)
\(0\cdot\)
To compute \(x\ast y\) we add up the following vectors:
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0\end{array}\right]^{\top}\]
\({}^{\top}\)
Convolutions in Pictures
Example. Let \[x=[ 0\ \ \ 1\ \ \ 2\ \ \ 2\ \ \ 3\ \ \ 1\ \ \ 0]^{\top}\quad\text{ and }\quad y=[0\ \ \ 0\ \ \ 1\ \ \ 1\ \ \ 2\ \ \ 1\ \ \ 0]^{\top}\]
\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 2 & 4 & 4 & 6 & 2 & 0 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 1 & 2 & 2 & 3 & 1 & 0 & 0\end{array}\right]^{\top}\]\[\left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{array}\right]^{\top}\]
To compute \(x\ast y\) we add up the following vectors:
\[x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}\]
\(+\)
Convolutions in Pictures
\[x =[0\ \ \ \ 1\ \ \ \ 2\ \ \ \ 2\ \ \ \ 3\ \ \ \ 1\ \ \ \ 0]^{\top}\]
\[y =[0\ \ \ \ 0\ \ \ \ 1\ \ \ \ 1\ \ \ \ 2\ \ \ \ 1\ \ \ \ 0]^{\top}\]
\[x\ast y = \left[\begin{array}{rrrrrrrrrrrrr} 0 & 0 & 0 & 1 & 3 & 6 & 10 & 10 & 9 & 5 & 1 & 0 & 0\end{array}\right]^{\top}\]
Walls are Convolutions
So, the sound on the other side of the wall is a sum of time shifted and muffled (scaled) versions of the same sound.
It's a convolution!
Walls are Convolutions
If \(x\) is a sound on one side of the wall, then there is some \(y\) such that \(x\ast y\) is the sound on the other side.
Recall that \(x\ast y = \tilde{x}\circledast \tilde{y}\)
Given \[\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\]
Recall that
\[C(\tilde{y}) = \begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\\ a_{N-1} & a_{0} & a_{1} & \cdots & a_{N-2}\\ a_{N-2} & a_{N-1} & a_{0} & \cdots & a_{N-3}\\ \vdots & \vdots & \vdots & \ddots & \vdots\\ a_{1} & a_{2} & a_{3} & \cdots & a_{0} \end{bmatrix}\]
and hence
\[C(\tilde{y})^{\top}\tilde{x} = \tilde{x}\circledast \tilde{y} = x\ast y\]
Walls are Convolutions
If \(x\) is a sound on one side of the wall, then there is some \(y\) such that \[x\ast y = \tilde{x}\circledast \tilde{y} = C(\tilde{y})^{\top}\tilde{x}\] is the sound on the other side.
\[C(\tilde{y}) = F_{N}\begin{bmatrix} f(\omega^{0}) & 0 & 0 & 0 & \cdots & 0\\ 0 & f(\omega) & 0 & 0 & \cdots & 0\\ 0 & 0 & f(\omega^2) & 0 & \cdots & 0\\ 0 & 0 & 0 & f(\omega^3) & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & f(\omega^{N-1})\end{bmatrix}F_{N}^{\ast}\]
If we define the polynomial
\[f(x) = a_{0} + a_{1}x+a_{2}x^2+a_{3}x^3+\cdots+a_{N-1}t^{N-1},\]
where \(\tilde{y} = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]\)
Linear Algebra Day 38
By John Jasper
