Day 37:

More Properties of the Fourier Transform

and convolutions

Circulant Matrices

Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. A matrix \(C\) is called circulant if there are numbers \(c_{0},c_{1},c_{2},\ldots,c_{N-1}\) such that

\[C = c_{0}I + c_{1}P + c_{2}P^{2} + \cdots + c_{N-1}P^{N-1}.\]

\[ = \begin{bmatrix} c_{0} & c_{1} & c_{2} & c_{3} & \cdots & c_{N-1}\\ c_{N-1} & c_{0} & c_{1} & c_{2} &  \cdots & c_{N-2}\\ c_{N-2} & c_{N-1} & c_{0} & c_{1} & \cdots & c_{N-3}\\ c_{N-3} & c_{N-2} & c_{N-1} & c_{0} & \cdots & c_{N-4}\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ c_{1} & c_{2} & c_{3} & c_{4} & \cdots & c_{0}\end{bmatrix}\]

Circulant Matrices

Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\), define the circulant matrix

\[C(x) = a_{0} + a_{1}P + a_{2}P^{2} + \cdots + a_{N-1}P^{N-1}.\]

Lemma. Given \(x,y\in\mathbb{C}^{n}\) we have

\[C(x)C(y) = C(x\circledast y)\]

Eigenvalues/vectors of the upward shift

Lemma. If \(P\) is the \(N\times N\) upward shift matrix, and \(f_{k}\) is the \(k\)th column of the Fourier Transform matrix \(F_{N}\), then

\[Pf_{k} = e^{-2k\pi i/N}f_{k} = \omega^{k}f_{k}.\]

Proof.  \[f_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\end{bmatrix}\]

\[Pf_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\\ 1\end{bmatrix} = \frac{\omega^{k}}{\sqrt{N}}\begin{bmatrix} \omega^0\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{-k}\end{bmatrix} = \omega^{k}\frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{Nk}\omega^{-k}\end{bmatrix} = \omega^{k}f_{k}\]

Eigenvalues/vectors of the upward shift

By the lemma, the columns of \(F_{N}\) are an orthonormal basis of eigenvectors of the \(N\times N\) upward shift matrix \(P\), thus

\[P = F_{N}\begin{bmatrix} \omega^{0} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^2 & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^3 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{N-1}\end{bmatrix}F_{N}^{\ast}\]

Set \[W_{N} := \begin{bmatrix} \omega^{0} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^2 & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^3 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{N-1}\end{bmatrix}\]

then we have \(P = F_{N}W_{N}F_{N}^{\ast}.\)

Powers of the upward shift

Note that \[W_{N}^{k} = \begin{bmatrix} \omega^{0k} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega^{k} & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^{2k} & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^{3k} & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{(N-1)k}\end{bmatrix},\]

and hence

\[P^{k} = \underbrace{(F_{N}W_{N}F_{N}^{\ast})(F_{N}W_{N}F_{N}^{\ast})(F_{N}W_{N}F_{N}^{\ast})\cdots (F_{N}W_{N}F_{N}^{\ast})}_{k\text{ factors}}\]

\(= F_{N}W_{N}(F_{N}^{\ast}F_{N})W_{N}(F_{N}^{\ast}F_{N})W_{N}(F_{N}^{\ast}F_{N})\cdots W_{N}F_{N}^{\ast}\)

\(= F_{N}W_{N}(I)W_{N}(I)W_{N}(I)\cdots W_{N}F_{N}^{\ast} = F_{N}W_{N}^{k}F_{N}^\ast\)

(Note that \(W_{N}^{N}\) is the identity, and hence \(P^{N} = F_{N}I F_{N}^{\ast} = I\) as we have already observed.)

Eigenvalues of a Circulant Matrix

\(= F_{N}\big(a_{0}W_{N}^{0} + a_{1}W_{N}^{1} + a_{2}W_{N}^2 + \cdots + a_{N-1}W_{N}^{N-1}\big)F_{N}^{\ast}\)

\(= a_{0}F_{N}W_{N}^{0}F_{N}^{\ast} + a_{1}F_{N}W_{N}^{1}F_{N}^{\ast} + a_{2}F_{N}W_{N}^{2}F_{N}^{\ast} + \cdots + a_{N-1}F_{N}W_{N}^{N-1}F_{N}^{\ast}\)

Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\). consider the circulant matrix:

\[C(x) = a_{0} + a_{1}P + a_{2}P^{2} + \cdots + a_{N-1}P^{N-1}\]

If we define the polynomial

\[f(x) = a_{0} + a_{1}x+a_{2}x^2+a_{3}x^3+\cdots+a_{N-1}x^{N-1},\]

 

 

then \[C(x) = F_{N}\begin{bmatrix} f(\omega^{0}) & 0 & 0 & 0 & \cdots & 0\\ 0 & f(\omega) & 0 & 0 & \cdots & 0\\ 0 & 0 & f(\omega^2) & 0 & \cdots & 0\\ 0 & 0 & 0 & f(\omega^3) & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & f(\omega^{N-1})\end{bmatrix}F_{N}^{\ast}\]

Eigenvalues of a Circulant Matrix

Example. Consider the vector \([1\ \ 3\ \ 2]^{\top}\), then

\[=F_{3}\left(\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix} + 3\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega & 0\\ 0 & 0 & \omega^2\end{bmatrix} + 2\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega^2 & 0\\ 0 & 0 & \omega^4\end{bmatrix}\right)F_{3}^{\ast}\]

\[=F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix}F_{3}^{\ast} + 3F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega & 0\\ 0 & 0 & \omega^2\end{bmatrix}F_{3}^{\ast} + 2F_{3}\begin{bmatrix} 1 & 0 & 0\\ 0 & \omega^2 & 0\\ 0 & 0 & \omega^4\end{bmatrix}F_{3}^{\ast}\]

\[=F_{3}\begin{bmatrix} 1+3(1)+2(1)^2 & 0 & 0\\ 0 & 1+3\omega+2\omega^2 & 0\\ 0 & 0 & 1+3(\omega^{2}) + 2(\omega^{2})^{2} \end{bmatrix} F_{3}^{\ast}\]

\[C(\begin{bmatrix}1\\ 3\\ 2\end{bmatrix}) = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1\end{bmatrix} + 3\begin{bmatrix} 0 & 1 & 0\\ 0 & 0 & 1\\ 1 & 0 & 0\end{bmatrix} + 2\begin{bmatrix} 0 & 0 & 1\\ 1 & 0 & 0\\ 0 & 1 & 0\end{bmatrix} = \begin{bmatrix} 1 & 3 & 2\\ 2 & 1 & 3\\ 3 & 2 & 1\end{bmatrix}\]

Eigenvalues of a Circulant Matrix

Theorem. Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\), define the polynomial

\[g(t) = a_{0} + a_{1}t + a_{2}t^{2} + a_{3}t^{3} + \cdots + a_{N-1}t^{N-1}.\]

If \(f_{k}\) is the \(k\)th column of the Fourier Transform matrix \(F_{N}\), then

\[C(x)f_{k} = g(\omega^{k})f_{k}.\]

That is, \(\{f_{0},f_{1},f_{2},\ldots,f_{N-1}\}\) is an orthonormal basis of eigenvectors of \(C(x)\) with associated eigenvalues \(\{g(1),g(\omega),g(\omega^{2}),\ldots,g(\omega^{N-1})\}\).

Fourier Transform of a Cyclic Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]

then we already computed

Now, we take the Fourier Transform of \(x\circledast y\) and we get

\[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 & 1 & 1\\ 1 & \omega & \omega^2\\ 1 & \omega^2 & \omega^4\end{bmatrix} \begin{bmatrix} 11\\ 11\\ 5\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 11+ 11+ 5\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega^4\end{bmatrix}\]

\[=\frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ 11+ 11\omega+ 5\omega^2\\ 11+ 11\omega^2+ 5\omega\end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]

Example continued. Now, we take the Fourier Transform of \(x\circledast y\) and we get \[F_{3}(x\circledast y) = \frac{1}{\sqrt{3}} \begin{bmatrix} 27\\ -6\omega^2\\ -6\omega \end{bmatrix}\]

\[F_{3}x = \frac{1}{\sqrt{3}} \begin{bmatrix} 3\\ 1+2\omega^2\\ 1+2\omega \end{bmatrix}\ \text{and}\ F_{3}y = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 1+5\omega+3\omega^2\\ 1+5\omega^2+3\omega \end{bmatrix} = \frac{1}{\sqrt{3}} \begin{bmatrix} 9\\ 4\omega+2\omega^2\\ 4\omega^2+2\omega \end{bmatrix}\]

\((1+2\omega^2)(4\omega+2\omega^2) = 4\omega+2\omega^2+8\omega^3+4\omega^4 = 8+8\omega+2\omega^2 = -6\omega^2\)

\((1+2\omega)(4\omega^2+2\omega) = 4\omega^2+2\omega+8\omega^3+4\omega^2 = 8+2\omega+8\omega^2 = -6\omega\)

\[F_{3}x\odot F_{3}y \]

Fourier Transform of a Cyclic Convolution

Lemma. Let 

\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]

\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]

Define the polynomials

\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]

\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}\]

Then,

\[f(t)g(t) = c_{0} + c_{1}t+c_{2}t^2+\cdots + c_{2N-2}t^{2N-2}\]

where

\[[c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2N-2}]^{\top} = x\ast y.\]

If \(\omega = e^{-2\pi i/N}\), then

\[f(\omega^{k})g(\omega^{k}) = d_{0} + d_{1}\omega^{k} + d_{2}\omega^{2k} + \cdots + d_{N-1}\omega^{(N-1)k}\]

where

\[[d_{0}\ \ d_{1}\ \ d_{2}\ \ \cdots\ \ d_{N-1}]^{\top} = x\circledast y.\]

Theorem. If \(x,y\in\mathbb{C}^{N}\), then

\[F_{N}(x\circledast y) = \sqrt{N}\big(F_{N}x\odot F_{N}y\big).\]

Fourier Transform of a Cyclic Convolution

Proof. Let 

\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]

\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]

\[x\circledast y = \begin{bmatrix} d_{0} & d_{1} & d_{2} & \cdots & d_{N-1}\end{bmatrix}^{\top}\]

Define the polynomials

\[f(t) = a_{0} + a_{1}t + a_{2}t^2+\cdots + a_{N-1}t^{N-1}\]

\[g(t) = b_{0} + b_{1}t + b_{2}t^2+\cdots + b_{N-1}t^{N-1}.\]

Notice that 

\[F_{N}x = \frac{1}{\sqrt{N}}\begin{bmatrix} f(1) & f(\omega) & f(\omega^{2}) & \cdots & f(\omega)^{N-1}\end{bmatrix}^{\top}\]

\[F_{N}y = \frac{1}{\sqrt{N}}\begin{bmatrix} g(1) & g(\omega) & g(\omega^{2}) & \cdots & g(\omega)^{N-1}\end{bmatrix}^{\top}\]

Fourier Transform of a Cyclic Convolution

Proof continued. Hence, if we define

\[h(t) = d_{0} + d_{1}t + d_{2}t^2 + \cdots + d_{N-1}t^{N-1},\]

then by the lemma we have

\[F_{N}x\odot F_{N}y= \]

\[= \frac{1}{N}\begin{bmatrix} f(1)g(1) & f(\omega)g(\omega) & f(\omega^{2})g(\omega^{2}) & \cdots & f(\omega^{N-1})g(\omega^{N-1})\end{bmatrix}^{\top}\]

\[= \frac{1}{\sqrt{N}}\left(\frac{1}{\sqrt{N}}\begin{bmatrix} h(1) & h(\omega) & h(\omega)^{2} & \cdots & h(\omega^{N-1})\end{bmatrix}^{\top}\right)\]

\[= \frac{1}{\sqrt{N}}\left(F_{N}(x\circledast y)\right).\ \Box\]