Day 37:
More Properties of the Fourier Transform
and convolutions
Circulant Matrices
Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. A matrix \(C\) is called circulant if there are numbers \(c_{0},c_{1},c_{2},\ldots,c_{N-1}\) such that
\[C = c_{0}I + c_{1}P + c_{2}P^{2} + \cdots + c_{N-1}P^{N-1}.\]
\[ = \begin{bmatrix} c_{0} & c_{1} & c_{2} & c_{3} & \cdots & c_{N-1}\\ c_{N-1} & c_{0} & c_{1} & c_{2} & \cdots & c_{N-2}\\ c_{N-2} & c_{N-1} & c_{0} & c_{1} & \cdots & c_{N-3}\\ c_{N-3} & c_{N-2} & c_{N-1} & c_{0} & \cdots & c_{N-4}\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ c_{1} & c_{2} & c_{3} & c_{4} & \cdots & c_{0}\end{bmatrix}\]
Circulant Matrices
Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\), define the circulant matrix
\[C(x) = a_{0} + a_{1}P + a_{2}P^{2} + \cdots + a_{N-1}P^{N-1}.\]
Lemma. Given \(x,y\in\mathbb{C}^{n}\) we have
\[C(x)C(y) = C(x\circledast y)\]
Eigenvalues/vectors of the upward shift
Lemma. If \(P\) is the \(N\times N\) upward shift matrix, and \(f_{k}\) is the \(k\)th column of the Fourier Transform matrix \(F_{N}\), then
\[Pf_{k} = e^{-2k\pi i/N}f_{k} = \omega^{k}f_{k}.\]
Proof. \[f_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\end{bmatrix}\]
\[Pf_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\\ 1\end{bmatrix} = \frac{\omega^{k}}{\sqrt{N}}\begin{bmatrix} \omega^0\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{-k}\end{bmatrix} = \omega^{k}\frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{Nk}\omega^{-k}\end{bmatrix} = \omega^{k}f_{k}\]
Eigenvalues/vectors of the upward shift
By the lemma, the columns of \(F_{N}\) are an orthonormal basis of eigenvectors of the \(N\times N\) upward shift matrix \(P\), thus
\[P = F_{N}\begin{bmatrix} \omega^{0} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^2 & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^3 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{N-1}\end{bmatrix}F_{N}^{\ast}\]
Set \[W_{N} := \begin{bmatrix} \omega^{0} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^2 & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^3 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{N-1}\end{bmatrix}\]
then we have \(P = F_{N}W_{N}F_{N}^{\ast}.\)
Powers of the upward shift
Note that \[W_{N}^{k} = \begin{bmatrix} \omega^{0k} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega^{k} & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^{2k} & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^{3k} & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{(N-1)k}\end{bmatrix},\]
and hence
\[P^{k} = \underbrace{(F_{N}W_{N}F_{N}^{\ast})(F_{N}W_{N}F_{N}^{\ast})(F_{N}W_{N}F_{N}^{\ast})\cdots (F_{N}W_{N}F_{N}^{\ast})}_{k\text{ factors}}\]
\(= F_{N}W_{N}(F_{N}^{\ast}F_{N})W_{N}(F_{N}^{\ast}F_{N})W_{N}(F_{N}^{\ast}F_{N})\cdots W_{N}F_{N}^{\ast}\)
\(= F_{N}W_{N}(I)W_{N}(I)W_{N}(I)\cdots W_{N}F_{N}^{\ast} = F_{N}W_{N}^{k}F_{N}^\ast\)
(Note that \(W_{N}^{N}\) is the identity, and hence \(P^{N} = F_{N}I F_{N}^{\ast} = I\) as we have already observed.)
Eigenvalues of a Circulant Matrix
\(= F_{N}\big(a_{0}W_{N}^{0} + a_{1}W_{N}^{1} + a_{2}W_{N}^2 + \cdots + a_{N-1}W_{N}^{N-1}\big)F_{N}^{\ast}\)
\(= a_{0}F_{N}W_{N}^{0}F_{N}^{\ast} + a_{1}F_{N}W_{N}^{1}F_{N}^{\ast} + a_{2}F_{N}W_{N}^{2}F_{N}^{\ast} + \cdots + a_{N-1}F_{N}W_{N}^{N-1}F_{N}^{\ast}\)
Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\). consider the circulant matrix:
\[C(x) = a_{0} + a_{1}P + a_{2}P^{2} + \cdots + a_{N-1}P^{N-1}\]
If we define the polynomial
\[f(x) = a_{0} + a_{1}x+a_{2}x^2+a_{3}x^3+\cdots+a_{N-1}x^{N-1},\]
then \[C(x) = F_{N}\begin{bmatrix} f(\omega^{0}) & 0 & 0 & 0 & \cdots & 0\\ 0 & f(\omega) & 0 & 0 & \cdots & 0\\ 0 & 0 & f(\omega^2) & 0 & \cdots & 0\\ 0 & 0 & 0 & f(\omega^3) & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & f(\omega^{N-1})\end{bmatrix}F_{N}^{\ast}\]
Eigenvalues of a Circulant Matrix
Example. Consider the vector \([1\ \ 3\ \ 2]^{\top}\), then
\[=F_{3}\left(\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix} + 3\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega & 0\\ 0 & 0 & \omega^2\end{bmatrix} + 2\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega^2 & 0\\ 0 & 0 & \omega^4\end{bmatrix}\right)F_{3}^{\ast}\]
\[=F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix}F_{3}^{\ast} + 3F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega & 0\\ 0 & 0 & \omega^2\end{bmatrix}F_{3}^{\ast} + 2F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega^2 & 0\\ 0 & 0 & \omega^4\end{bmatrix}F_{3}^{\ast}\]
\[=F_{3}\begin{bmatrix} 1+3(1)+2(1)^2 & 0 & 0\\ 0 & 1+3\omega+2\omega^2 & 0\\ 0 & 0 & 1+3(\omega^{2}) + 2(\omega^{2})^{2} \end{bmatrix} F_{3}^{\ast}\]
\[C(\begin{bmatrix}1\\ 3\\ 2\end{bmatrix}) = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix} + 3\begin{bmatrix} 0 & 1 & 0\\ 0 & 0 & 1\\ 1 & 0 & 0\end{bmatrix} + 2\begin{bmatrix} 0 & 0 & 1\\ 1 & 0 & 0\\ 0 & 1 & 0\end{bmatrix} = \begin{bmatrix} 1 & 3 & 2\\ 2 & 1 & 3\\ 3 & 2 & 1\end{bmatrix}\]
Eigenvalues of a Circulant Matrix
Theorem. Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\), define the polynomial
\[g(t) = a_{0} + a_{1}t + a_{2}t^{2} + a_{3}t^{3} + \cdots + a_{N-1}t^{N-1}.\]
If \(f_{k}\) is the \(k\)th column of the Fourier Transform matrix \(F_{N}\), then
\[C(x)f_{k} = g(\omega^{k})f_{k}.\]
That is, \(\{f_{0},f_{1},f_{2},\ldots,f_{N-1}\}\) is an orthonormal basis of eigenvectors of \(C(x)\) with associated eigenvalues \(\{g(1),g(\omega),g(\omega^{2}),\ldots,g(\omega^{N-1})\}\).
Example.
\[C(\begin{bmatrix}1\\ 3\\ 2\end{bmatrix}) = \begin{bmatrix} 1 & 3 & 2\\ 2 & 1 & 3\\ 3 & 2 & 1\end{bmatrix}\]
\[f_{1} = \frac{1}{\sqrt{3}}\begin{bmatrix} 1\\ \omega\\ \omega^2\end{bmatrix} = \frac{1}{\sqrt{3}}\begin{bmatrix} 1\\ -\frac{1}{2}-\frac{\sqrt{3}}{2}i\\ -\frac{1}{2}+\frac{\sqrt{3}}{2}i\end{bmatrix}\]
Set \(g(x) = 1+3x+2x^2\).
is an eigenvector with eigenvalue
\(g(\omega) = 1+3\omega+2\omega^2\)
\(=1+3(-\frac{1}{2}-\frac{\sqrt{3}}{2}i) + 2(-\frac{1}{2}+\frac{\sqrt{3}}{2}i)\)
\(=-\frac{3}{2} -\frac{\sqrt{3}}{2}i\)
Fourier Transform of a Cyclic Convolution
Example. If
\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]
\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]
then we already computed
Now, we take the Fourier Transform of \(x\circledast y\) and we get
\[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 & 1 & 1\\ 1 & \omega & \omega^2\\ 1 & \omega^2 & \omega^4\end{bmatrix} \begin{bmatrix} 11\\ 11\\ 5\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 11+ 11+ 5\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega^4\end{bmatrix}\]
\[=\frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]
Remember:
- \(\omega^3=1\)
- \(1+\omega+\omega^2=0\)
Example continued. Now, we take the Fourier Transform of \(x\circledast y\) and we get \[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]
\[F_{3}x = \frac{1}{\sqrt{3}} \begin{bmatrix} 3\\ 1+2\omega^2\\ 1+2\omega \end{bmatrix}\ \text{and}\ F_{3}y = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 1+5\omega+3\omega^2\\ 1+5\omega^2+3\omega \end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 4\omega+2\omega^2\\ 4\omega^2+2\omega \end{bmatrix}\]
\((1+2\omega^2)(4\omega+2\omega^2) = 4\omega+2\omega^2+8\omega^3+4\omega^4 = 8+8\omega+2\omega^2 = -6\omega^2\)
\((1+2\omega)(4\omega^2+2\omega) = 4\omega^2+2\omega+8\omega^3+4\omega^2 = 8+2\omega+8\omega^2 = -6\omega\)
\[F_{3}x\odot F_{3}y \]
Fourier Transform of a Cyclic Convolution
This is the entrywise product of two matrices of the same size. This is sometimes called the Hadamard product.
\[= \frac{1}{3}\begin{bmatrix} 3\cdot 9\\ (1+2\omega^2)(4+2\omega^2)\\ (1+2\omega)(4+2\omega)\end{bmatrix} = \frac{1}{3}\begin{bmatrix} 27\\ -6\omega^2\\ -6\omega\end{bmatrix} = \frac{1}{\sqrt{3}}F_{3}(x\circledast y)\]
Lemma. Let
\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]
\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]
Define the polynomials
\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]
\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}\]
Then,
\[f(t)g(t) = c_{0} + c_{1}t+c_{2}t^2+\cdots + c_{2N-2}t^{2N-2}\]
where
\[[c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2N-2}]^{\top} = x\ast y.\]
If \(\omega = e^{-2\pi i/N}\), then
\[f(\omega^{k})g(\omega^{k}) = d_{0} + d_{1}\omega^{k} + d_{2}\omega^{2k} + \cdots + d_{N-1}\omega^{(N-1)k}\]
where
\[[d_{0}\ \ d_{1}\ \ d_{2}\ \ \cdots\ \ d_{N-1}]^{\top} = x\circledast y.\]
Theorem. If \(x,y\in\mathbb{C}^{N}\), then
\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]
Fourier Transform of a Cyclic Convolution
Proof. Let
\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]
\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]
\[x\circledast y = \begin{bmatrix} d_{0} & d_{1} & d_{2} & \cdots & d_{N-1}\end{bmatrix}^{\top}\]
Define the polynomials
\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]
\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}.\]
Notice that
\[F_{N}x = \frac{1}{\sqrt{N}}\begin{bmatrix} f(1) & f(\omega) & f(\omega^{2}) & \cdots & f(\omega)^{N-1}\end{bmatrix}^{\top}\]
\[F_{N}y = \frac{1}{\sqrt{N}}\begin{bmatrix} g(1) & g(\omega) & g(\omega^{2}) & \cdots & g(\omega)^{N-1}\end{bmatrix}^{\top}\]
Fourier Transform of a Cyclic Convolution
Proof continued. Hence, if we define
\[h(t) = d_{0} + d_{1}t + d_{2}t^2 + \cdots + d_{N-1}t^{N-1},\]
then by the lemma we have
\[F_{N}x\odot F_{N}y= \]
\[= \frac{1}{N}\begin{bmatrix} f(1)g(1) & f(\omega)g(\omega) & f(\omega^{2})g(\omega^{2}) & \cdots & f(\omega^{N-1})g(\omega^{N-1})\end{bmatrix}^{\top}\]
\[= \frac{1}{\sqrt{N}}\left(\frac{1}{\sqrt{N}}\begin{bmatrix} h(1) & h(\omega) & h(\omega)^{2} & \cdots & h(\omega^{N-1})\end{bmatrix}^{\top}\right)\]
\[= \frac{1}{\sqrt{N}}\left(F_{N}(x\circledast y)\right).\ \Box\]
Linear Algebra Day 37
By John Jasper
Linear Algebra Day 37
- 342