Day 36:

Properties of the Fourier Transform

The Fourier Transform

Definition. Given a vector \(v\in\mathbb{C}^{N}\), the Fourier transform of \(v\) is the vector \(F_{N}v\).

(Important note: In some places, including Matlab, the Fourier transform of \(v\) is defined to be \(\sqrt{N}F_{N}v\).)

Example. If \(v = [1\ \ 1\ \ 1\ \ 1]^{\top}\), then the Fourier transform of \(v\) is the vector

\[F_{4}v = \frac{1}{2}\left[\begin{array}{rrrr} 1 & 1 & 1 & 1\\ 1 & -i & -1 & i\\ 1 & -1 & 1 & -1\\ 1 & i & -1 & -i\end{array}\right]\begin{bmatrix}1\\ 1\\ 1\\ 1\end{bmatrix} = \begin{bmatrix}2\\ 0\\ 0\\ 0\end{bmatrix}.\]

Note that \(\|F_{4}v\| = \|v\|.\) This always happens!

Convolution

Motivation. Given two vectors

\[x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}\]

\[y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}\]

Define the polynomials

\[f(x) = a_{0} + a_{1}x + a_{2}x^2+\cdots + a_{N-1}x^{N-1}\]

\[g(x) = b_{0} + b_{1}x + b_{2}x^2+\cdots + b_{N-1}x^{N-1}\]

Then,

\[f(x)g(x) = c_{0} + c_{1}x+c_{2}x^2+\cdots + c_{2N-2}x^{2N-2}\]

where

\[c_{0} = a_{0}b_{0},\quad c_{1} = a_{0}b_{1} + a_{1}b_{0},\quad c_{2} = a_{0}b_{2} + a_{1}b_{1} + a_{2}b_{0},\ldots\]

Following this pattern:

\[c_{N} = a_{0}b_{N} + a_{1}b_{N-1} + a_{2}b_{N-2} + \cdots + a_{N-1}b_{1} + a_{N}b_{0}\]

If we take \(a_{i}=b_{i}=0\) for \(i\geq N\), then we have \[c_{k} = \sum_{i=0}^{k}a_{i}b_{k-i}\]

Convolution

Definition. Given two vectors

\[x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\end{bmatrix}\]

The convolution, denoted \(x\ast y\), is the vector

\[x\ast y = \begin{bmatrix} c_{0}\\ c_{1}\\ c_{2}\\ \vdots\\ c_{2N-2}\end{bmatrix}\quad\text{where }c_{k} = \sum_{i=0}^{k}a_{i}b_{k-i}\]

Note that if \(x,y\in\R^{N}\), then \(x\ast y\in\R^{2N-1}\).

Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

Since

\[(1+2x^2)(1+5x+3x^2) = (1+2x^2)(1) + (1+2x^2)(5x)+(1+2x^2)(3x^2)\]

 

 

we see that \[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]

 

\[\begin{array}{ccc} 1 &  0 &  2\\ 1 &  5 &  3\\ \hline\end{array}\]

\[\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}\]

\(1+0x+2x^2\)

\(5x+0x^2+10x^3\)

\(3x^2+0x^3+6x^4\)

\( = (1+2x^2) + (5x+10x^3) + (3x^2+6x^4)\)

\[= 1+5x+5x^2+10x^3+6x^4\]

To quickly compute \(x\ast y\):

Cyclic Convolution

Example. If

\[x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}\]

\[x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}\]

 

\[\begin{array}{ccc} 1 &  0 &  2\\ 1 &  5 &  3\\ \hline\end{array}\]

\[\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}\]

Convolution:

\[x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}\]

 

\[\begin{array}{ccc} 1 &  0 &   2\\ 1 &  5 &   3\\ \hline\end{array}\]

\[\begin{array}{ccc} 0 & 6 & 3\\ 10 & 5 & 0\\ 1 & 0 & 2\\\hline 11 & 11 & 5\end{array}\]

Cyclic convolution:

Cyclic Convolution

Definition. Given two vectors \(x,y\in\mathbb{R}^{n}\), let

\[x\ast y = \begin{bmatrix} c_{0}\\ c_{1}\\ c_{2}\\ \vdots\\ c_{2N-2}\end{bmatrix}\]

Note that if \(x,y\in\R^{N}\), then \(x\circledast y\in\R^{N}\).

The cyclic convolution, denoted \(x\circledast y\), is the vector

\[x\circledast y = \begin{bmatrix} c_{0}+c_{N}\\ c_{1}+c_{N+1}\\ c_{2}+c_{N+2}\\ \vdots\\ c_{N-2}+c_{2N-2}\\ c_{N-1}\end{bmatrix}.\]

Upward Shift Matrix

Definition.  Let \(N\in\N\). For each \(i=1,2,\ldots,N\) let \(e_{i}\) be the \(i\)th column of the \(N\times N\) identity matrix.

The upward shift matrix is the matrix

\[P = \begin{bmatrix} \vert & \vert & \vert & \vert & & \vert\\ e_{N} & e_{1} & e_{2} & e_{3} & \cdots & e_{N-1}\\ \vert & \vert & \vert & \vert & & \vert\end{bmatrix}\]

Example. If \(N=4\), then the upward shift matrix is

\[\begin{bmatrix} 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1\\ 1 & 0 & 0 & 0\end{bmatrix}\]

Upward Shift Matrix

Proposition. If \(P\) is the \(N\times N\) upward shift matrix, then \(P^{N} = I\).

Proof. Let \(x = [x_{1}\ \ x_{2}\ \ x_{3}\ \ \cdots\ \ x_{N}]^{\top}\), then

\[Px = \begin{bmatrix} x_{2}\\ x_{3}\\ x_{4}\\ \vdots\\ x_{N}\\ x_{1}\end{bmatrix},\ P^{2}x = \begin{bmatrix} x_{3}\\ x_{4}\\ \vdots\\ x_{N}\\ x_{1}\\ x_{2}\end{bmatrix},\cdots, P^{N-1}x = \begin{bmatrix} x_{N}\\ x_{1}\\ x_{2}\\ x_{3}\\ \vdots \\ x_{N-1}\end{bmatrix},\ P^{N}x = \begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \vdots \\ x_{N-1}\\ x_{N}\end{bmatrix}.\Box\]

Upward Shift Matrix

Exercise. Let \(P\) be the \(4\times 4\) upward shift matrix.

1. Compute \(P^{2}, P^{3},\) and \(P^{4}\).
2. Find the matrix \(3I + 2P+5P^2+7P^3\).

Circulant Matrices

Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. A matrix \(C\) is called circulant if there are numbers \(c_{0},c_{1},c_{2},\ldots,c_{N-1}\) such that

\[C = c_{0}I + c_{1}P + c_{2}P^{2} + \cdots + c_{N-1}P^{N-1}.\]

\[ = \begin{bmatrix} c_{0} & c_{1} & c_{2} & c_{3} & \cdots & c_{N-1}\\ c_{N-1} & c_{0} & c_{1} & c_{2} &  \cdots & c_{N-2}\\ c_{N-2} & c_{N-1} & c_{0} & c_{1} & \cdots & c_{N-3}\\ c_{N-3} & c_{N-2} & c_{N-1} & c_{0} & \cdots & c_{N-4}\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ c_{1} & c_{2} & c_{3} & c_{4} & \cdots & c_{0}\end{bmatrix}\]

A circulant matrix is a matrix with all of its diagonals constant.

Circulant Matrices

Example. Consider the \(3\times 3\) circulant matrices

\[C = I + 2P^{2} = \begin{bmatrix} 1 & 0 & 2\\ 2 & 1 & 0\\ 0 & 2 & 1\end{bmatrix}\]

and

\[D = I +5P + 3P^{2} = \begin{bmatrix} 1 & 5 & 3\\ 3 & 1 & 5\\ 5 & 3 & 1\end{bmatrix}\]

Find \(CD\) and \(DC\). How do these matrices relate to \[[1\ \ 0\ \ 2]^{\top}\circledast [1\ \ 5\ \ 3]^{\top}?\]

Circulant Matrices

Definition. Let \(N\in\N\), and let \(P\) be the \(N\times N\) upward shift matrix. Given a vector \(x=[a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{N-1}]^{\top}\), define the circulant matrix

\[C(x) = a_{0}I + a_{1}P + a_{2}P^{2} + \cdots + a_{N-1}P^{N-1}.\]

Lemma. Given \(x,y\in\mathbb{C}^{n}\) we have

\[C(x)C(y) = C(x\circledast y)\]

Proof. 

Circulant Matrices

Lemma. Given \(x,y\in\mathbb{C}^{n}\) we have

\[C(x)C(y) = C(x\circledast y)\]

Proof. Let \(x = [a_{0}\ \ a_{1}\ \ a_{2}\ \ \cdots\ \ a_{n-1}]^{\top}\), \(y=[b_{0}\ \ b_{1}\ \ b_{2}\ \ \cdots\ \ b_{n-1}]^{\top}\), and \(x\ast y = [c_{0}\ \ c_{1}\ \ c_{2}\ \ \cdots\ \ c_{2n-2}]^{\top}\), then 

\(= (a_{0}I + a_{1}P + a_{2}P^{2} + \cdots + a_{n-1}P^{n-1})(b_{0}I + b_{1}P + b_{2}P^{2} + \cdots + b_{n-1}P^{n-1})\)

\( = c_{0}I+c_{1}P+c_{2}P^{2} + \cdots + c_{n-1}P^{n-1}\)

\(\quad + c_{n}I + c_{n+1}P + c_{n+2}P^{2} +\cdots + c_{2n-2}P^{n-2}\)

\( = c_{0}I+c_{1}P+c_{2}P^{2} + \cdots + c_{n-1}P^{n-1}\)

\(\quad + c_{n}P^{n} + c_{n+1}P^{n+1} + c_{n+2}P^{n+2} +\cdots + c_{2n-2}P^{2n-2}\)

\(C(x)C(y)\)

\( = (c_{0}+c_{n})I+(c_{1}+c_{n+1})P+(c_{2}+c_{n+2})P^{2} + \cdots +(c_{n-2}+c_{2n-2})P^{n-2}\)

\(\quad+c_{n-1}P^{n-1}.\ \Box\)

Eigenvalues/vectors of the upward shift

Lemma. If \(P\) is the \(N\times N\) upward shift matrix, and \(f_{k}\) is the \(k\)th column of the Fourier Transform matrix \(F_{N}\), then

\[Pf_{k} = e^{-2k\pi i/N}f_{k} = \omega^{k}f_{k}.\]

Proof.  \[f_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\end{bmatrix}\]

\[Pf_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\\ 1\end{bmatrix} = \frac{\omega^{k}}{\sqrt{N}}\begin{bmatrix} \omega^0\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{-k}\end{bmatrix} = \omega^{k}\frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^{k}\\ \omega^{2k}\\ \vdots\\ \omega^{(N-2)k}\\ \omega^{Nk}\omega^{-k}\end{bmatrix} = \omega^{k}f_{k}\]

Eigenvalues/vectors of the upward shift

By the lemma, the columns of \(F_{N}\) are an orthonormal basis of eigenvectors of the \(N\times N\) upward shift matrix \(P\), thus

\[P = F_{N}\begin{bmatrix} \omega^{0} & 0 & 0 & 0 & \cdots & 0\\ 0 & \omega & 0 & 0 & \cdots & 0\\ 0 & 0 & \omega^2 & 0 & \cdots & 0\\ 0 & 0 & 0 & \omega^3 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & \cdots & \omega^{N-1}\end{bmatrix}F_{N}^{\ast}\]