Day 35:

Complex Matrices and the Fourier Transform

Complex Vectors

$$\mathbb{C}^{2} = \left\{\left[\begin{matrix}a\\ b\end{matrix}\right] : a\in\mathbb{C} \text{ and } b\in\mathbb{C}\right\}$$

$$\mathbb{C}^{3} = \left\{\left[\begin{matrix}a\\ b\\c\end{matrix}\right] : a\in\mathbb{C} \text{ and } b\in\mathbb{C} \text{ and } c\in\mathbb{C}\right\}$$

$$\mathbb{C}^{n} = \left\{\left[\begin{matrix}a_{1}\\ a_{2}\\\vdots\\a_{n}\end{matrix}\right] : a_{1}\in\mathbb{C} \text{ and } a_{2}\in\mathbb{C}\text{ and }\ldots \text{ and } a_{n}\in\mathbb{C}\right\}$$

(Note that the textbook uses $$\mathbf{C}$$ for the set of complex numbers instead of $$\mathbb{C}$$.)

Assume we have a matrix $$A$$ with complex entries:

$A=\left[\begin{matrix} a_{11} & a_{12} & \cdots & a_{1n}\\ a_{21} & a_{22} & & a_{2n}\\ \vdots & & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn}\end{matrix}\right]$

The adjoint of $$A$$, denoted $$A^{\ast}$$, is the matrix

$A^{\ast}=\left[\begin{matrix} \overline{a_{11}} & \overline{a_{21}} & \cdots & \overline{a_{n1}}\\ \overline{a_{12}} & \overline{a_{22}} & & \overline{a_{n2}}\\ \vdots & & \ddots & \vdots \\ \overline{a_{1m}} & \overline{a_{2m}} & \cdots & \overline{a_{nm}}\end{matrix}\right]$

Note that we obtain the adjoint by taking the transpose of $$A$$ and conjugating each entry.

In the homework we will show that our favorite property of the transpose also holds for the adjoint, namely, $(AB)^{\ast} = B^{\ast}A^{\ast}.$

Dot Product and Norm

Given two vectors

$u = \begin{bmatrix}u_{1}\\ u_{2}\\ \vdots\\ u_{d}\end{bmatrix}\quad\text{and}\quad v = \begin{bmatrix}v_{1}\\ v_{2}\\ \vdots\\ v_{d}\end{bmatrix}$

in  $$\mathbb{C}^{d}$$ (or $$\R^{d}$$), their dot product is given by

$u\cdot v=u^{\ast}v= \begin{bmatrix}\overline{u}_{1} & \overline{u}_{2} & \cdots & \overline{u}_{d}\end{bmatrix}\begin{bmatrix}v_{1}\\ v_{2}\\ \vdots\\ v_{d}\end{bmatrix} = \sum_{i=1}^{d}\overline{u}_{i}v_{i}.$

The norm of $$u$$ is given by

$\|u\| = \sqrt{u\cdot u} = \sqrt{u^{\ast}u} = \sqrt{\sum_{i=1}^{d}\overline{u}_{i}u_{i}} = \sqrt{\sum_{i=1}^{d}|u_{i}|^{2}}.$

Dot Product and Norm

Example. Consider the vectors

$v_{1} = \frac{1}{2}\left[\begin{array}{r}1\\ 1\\ 1\\ 1\end{array}\right],\ v_{2}=\frac{1}{2}\left[\begin{array}{r}1\\ -i\\ -1\\ i\end{array}\right],\ v_{3}=\frac{1}{2}\left[\begin{array}{r}1\\ -1\\ 1\\ -1\end{array}\right],\ v_{4}=\frac{1}{2}\left[\begin{array}{r}1\\ i\\ -1\\ -i\end{array}\right]$

$v_{2}\cdot v_{3} = \left(\frac{1}{2}[1\ \ i\ -1\ \ -i]\right)\left(\frac{1}{2}\left[\begin{array}{r}1\\ -1\\ 1\\ -1\end{array}\right]\right)$

$= \frac{1}{4}\big((1)(1)+(i)(-1)+(-1)(1)+(-i)(-1)\big) = \frac{1}{4}(1-i-1+i)=0$

Hence, $$v_{2}$$ and $$v_{3}$$ are orthogonal. One can similarly check that each pair of these vectors is orthogonal, and each as norm $$1$$, thus $$\{v_{1},v_{2},v_{3},v_{4}\}$$ is an orthonormal basis for $$\mathbb{C}^{4}$$.

Unitary matrices

Definition. Let $$U$$ be an $$n\times n$$ matrix with complex entries. The matrix $$U$$ is called unitary if  $$U^{\ast}U = UU^{\ast} = I$$.

Theorem. An $$n\times n$$ matrix $$U$$ is unitary if and only if the columns of $$U$$ form an orthonormal basis for $$\mathbb{C}^{n}$$.

Example. Consider the matrix

$U = \frac{1}{2}\left[\begin{array}{rrrr} 1 & 1 & 1 & 1\\ 1 & -i & -1 & i\\ 1 & -1 & 1 & -1\\ 1 & i & -1 & -i\end{array}\right]$

Using the notation from the previous example we have

$U^{\ast}U = \begin{bmatrix} v_{1}\cdot v_{1} & v_{1}\cdot v_{2} & v_{1}\cdot v_{3} & v_{1}\cdot v_{4}\\ v_{2}\cdot v_{1} & v_{2}\cdot v_{2} & v_{2}\cdot v_{3} & v_{2}\cdot v_{4}\\ v_{3}\cdot v_{1} & v_{3}\cdot v_{2} & v_{3}\cdot v_{3} & v_{3}\cdot v_{4}\\ v_{4}\cdot v_{4} & v_{4}\cdot v_{2} & v_{4}\cdot v_{3} & v_{4}\cdot v_{4}\\ \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & 0\\ 0 & 1 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1\end{bmatrix}$

The Fourier Transform Matrix

Definition. Given a positive whole number $$N$$ the Fourier transform matrix is the matrix

$F_{N} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1 & 1 & 1 & 1 & \cdots & 1\\ 1 & \omega & \omega^2 & \omega^3 & \cdots & \omega^{N-1}\\ 1 & \omega^2 & \omega^4 & \omega^6 & \cdots & \omega^{2(N-1)}\\ 1 & \omega^3 & \omega^6 & \omega^9 & \cdots & \omega^{3(N-1)}\\ \vdots & \vdots & \vdots & \vdots & \ddots & \vdots\\ 1 & \omega^{N-1} & \omega^{2(N-1)} & \omega^{3(N-1)} & \cdots & \omega^{(N-1)(N-1)}\end{bmatrix}$

where $$\omega=e^{-2\pi i/N}$$.

Example. For $$N=4$$ we have $$\omega=e^{-2\pi i/4} = e^{-\pi i/2} = -i$$, and hence

$F_{4} = \frac{1}{\sqrt{4}}\begin{bmatrix} 1 & 1 & 1 & 1\\ 1 & (-i) & (-i)^2 & (-i)^3\\ 1 & (-i)^2 & (-i)^4 & (-i)^6\\ 1 & (-i)^3 & (-i)^6 & (-i)^9\end{bmatrix} = \frac{1}{2}\left[\begin{array}{rrrr} 1 & 1 & 1 & 1\\ 1 & -i & -1 & i\\ 1 & -1 & 1 & -1\\ 1 & i & -1 & -i\end{array}\right]$

Lemma (The geometric sum formula). If $$r\in\mathbb{C}\setminus\{0,1\}$$ and $$N\in\N$$, then

$\sum_{m=0}^{N-1}r^{m} = \frac{1-r^{N}}{1-r}.$

Proof. Set

$S = \sum_{m=0}^{N-1}r^{m} = 1+r+r^2+\cdots + r^{N-1}.$

Then,

$rS = r+r^2+r^3+\cdots +r^{N}.$

Subtracting, we have

$(1-r)S = S-rS = \big(1+r+r^2+\cdots + r^{N-1}\big) - \big(r+r^2+r^3+\cdots +r^{N}\big)$

$$=1-r^N$$

Since $$r\neq 1$$ we see that $$1-r\neq 0$$. Thus we can divide both sides by $$1-r$$ and we obtain the desired equality. $$\Box$$

Theorem. For each $$N\in\N$$ the Fourier Transform Matrix $$F_{N}$$ is a unitary matrix.

Proof. For $$k=0,1,2,\ldots,N-1$$ set

$f_{k} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1\\ \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\end{bmatrix}.$

Since $$f_{0},f_{1},f_{2},\ldots,f_{N-1}$$ are the columns of $$F_{N}$$, it is enough to show that they form an orthonormal basis for $$\mathbb{C}^{N}$$.

Next note that if $$\theta\in\R$$, then
$\overline{e^{i\theta}} = \overline{\cos(\theta)+i\sin(\theta)} = \cos(\theta)-i\sin(\theta) = \cos(-\theta) + i\sin(-\theta) = e^{-i\theta}.$

Since $$\omega = e^{-2\pi i/N}$$ we see that

$\overline{\omega^{n}} = \overline{\left(e^{-2n\pi i/N}\right)} = \left(e^{2n\pi i/N}\right) = \omega^{-n}.$

Proof continued. Note that

$f_{k}^{\ast} = \frac{1}{\sqrt{N}}\begin{bmatrix} 1 & \overline{\omega^k} & \overline{\omega^{2k}} & \overline{\omega^{3k}} & \cdots & \overline{\omega^{(N-1)k}}\end{bmatrix}$

$= \frac{1}{\sqrt{N}}\begin{bmatrix} 1 & \omega^{-k} & \omega^{-2k} & \omega^{-3k} & \cdots & \omega^{-(N-1)k}\end{bmatrix}$

Now if $$j\neq k$$, we use the geometric sum formula to compute

$f_{j}\cdot f_{k} = f_{j}^{\ast}f_{k} = \frac{1}{N}\begin{bmatrix} 1 & \omega^{-j} & \omega^{-2j} & \omega^{-3j} & \cdots & \omega^{-(N-1)j}\end{bmatrix}\begin{bmatrix} 1\\ \omega^k\\ \omega^{2k}\\ \omega^{3k}\\ \vdots\\ \omega^{(N-1)k}\end{bmatrix}$

$= \frac{1}{N}\sum_{m=0}^{N-1}\omega^{-jm}\omega^{km} = \frac{1}{N}\sum_{m=0}^{N-1}(\omega^{k-j})^{m} = \frac{1}{N}\frac{1-(\omega^{(k-j)})^N}{1-\omega^{j-k}}.$

Note that  $$(\omega^{j-k})^{N} = (\omega^{N})^{(j-k)} = (1)^{(j-k)}=1$$, and hence

$f_{j}\cdot f_{k} = 0.$

Proof continued. On the other hand

$f_{j}\cdot f_{j}= \frac{1}{N}\sum_{m=0}^{N-1}\omega^{-jm}\omega^{jm} = \sum_{m=0}^{N-1}\omega^{0} = \frac{1}{N}\sum_{m=0}^{N-1}1 = \frac{1}{N}\cdot N = 1.$

In summary,

$f_{j}\cdot f_{k} = \begin{cases} 1 & \text{if } i=j,\\ 0 & \text{if }i\neq j,\end{cases}$

that is $$\{f_{0},f_{1},\ldots,f_{N-1}\}$$ is orthonormal. Since this is $$N$$ orthonormal vectors in $$\mathbb{C}^{N}$$, it is an orthonormal basis. $$\Box$$

The Fourier Transform

Definition. Given a vector $$v\in\mathbb{C}^{N}$$, the Fourier transform of $$v$$ is the vector $$F_{N}v$$.

(Important note: In some places, including Matlab, the Fourier transform of $$v$$ is defined to be $$\sqrt{N}F_{N}v$$.)

Example. If $$v = [1\ \ 1\ \ 1\ \ 1]^{\top}$$, then the Fourier transform of $$v$$ is the vector

$F_{4}v = \frac{1}{2}\left[\begin{array}{rrrr} 1 & 1 & 1 & 1\\ 1 & -i & -1 & i\\ 1 & -1 & 1 & -1\\ 1 & i & -1 & -i\end{array}\right]\begin{bmatrix}1\\ 1\\ 1\\ 1\end{bmatrix} = \begin{bmatrix}2\\ 0\\ 0\\ 0\end{bmatrix}.$

Note that $$\|F_{4}v\| = \|v\|.$$ This always happens!

The Fourier Transform

Theorem. If $$v\in\mathbb{C}^{N}$$, and $$U$$ is an $$N\times N$$ unitary matrix, then

$\|v\| = \|Uv\|.$

Proof. $$\|Uv\|^{2} = (Uv)^{\ast}(Uv) = v^{\ast}U^{\ast}Uv = v^{\ast}Iv = \|v\|^{2}.\ \Box$$

Corollary. If $$v\in\mathbb{C}^{N}$$, then

$\|v\| = \|F_{N}v\|.$

Convolution

Motivation. Given two vectors

$x=\begin{bmatrix} a_{0} & a_{1} & a_{2} & \cdots & a_{N-1}\end{bmatrix}^{\top}$

$y = \begin{bmatrix} b_{0} & b_{1} & b_{2} & \cdots & b_{N-1}\end{bmatrix}^{\top}$

Define the polynomials

$f(x) = a_{0} + a_{1}x + a_{2}x^2+\cdots + a_{N-1}x^{N-1}$

$g(x) = b_{0} + b_{1}x + b_{2}x^2+\cdots + b_{N-1}x^{N-1}$

Then,

$f(x)g(x) = c_{0} + c_{1}x+c_{2}x^2+\cdots + c_{2N-2}x^{2N-2}$

where

$c_{0} = a_{0}b_{0},\quad c_{1} = a_{0}b_{1} + a_{1}b_{0},\quad c_{2} = a_{0}b_{2} + a_{1}b_{1} + a_{2}b_{0},\ldots$

Following this pattern:

$c_{N} = a_{0}b_{N} + a_{1}b_{N-1} + a_{2}b_{N-2} + \cdots + a_{N-1}b_{1} + a_{N}b_{0}$

If we take $$a_{i}=b_{i}=0$$ for $$i\geq N$$, then we have $c_{k} = \sum_{i=0}^{k}a_{i}b_{k-i}$

Convolution

Definition. Given two vectors

$x=\begin{bmatrix} a_{0}\\ a_{1}\\ a_{2}\\ \vdots\\ a_{N-1}\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} b_{0}\\ b_{1}\\ b_{2}\\ \vdots\\ b_{N-1}\end{bmatrix}$

The convolution, denoted $$x\ast y$$, is the vector

$x\ast y = \begin{bmatrix} c_{0}\\ c_{1}\\ c_{2}\\ \vdots\\ c_{2N-2}\end{bmatrix}\quad\text{where }c_{k} = \sum_{i=0}^{k}a_{i}b_{k-i}$

Note that if $$x,y\in\R^{N}$$, then $$x\ast y\in\R^{2N-1}$$.

Convolution

Example. If

$x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}$

Since

$(1+2x^2)(1+5x+3x^2) = (1+2x^2)(1) + (1+2x^2)(5x)+(1+2x^2)(3x^2)$

we see that $x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}$

$\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}$

$\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}$

$$1+0x+2x^2$$

$$5x+0x^2+10x^3$$

$$3x^2+0x^3+6x^4$$

$$= (1+2x^2) + (5x+10x^3) + (3x^2+6x^4)$$

$= 1+5x+5x^2+10x^3+6x^4$

To quickly compute $$x\ast y$$:

Cyclic Convolution

Example. If

$x=\begin{bmatrix} 1\\ 0\\ 2\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 5\\ 3\end{bmatrix}$

$x\ast y = \begin{bmatrix} 1 & 5 & 5 & 10 & 6\end{bmatrix}^{\top}$

$\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}$

$\begin{array}{ccccc} & & 3 & 0 & 6\\ & 5 & 0 & 10 & \\ 1 & 0 & 2 & & \\ \hline 1 & 5 & 5 & 10 & 6\end{array}$

Convolution:

$x\circledast y = \begin{bmatrix} 11 & 11 & 5\end{bmatrix}^{\top}$

$\begin{array}{ccc} 1 & 0 & 2\\ 1 & 5 & 3\\ \hline\end{array}$

$\begin{array}{ccc} 0 & 6 & 3\\ 10 & 5 & 0\\ 1 & 0 & 2\\\hline 11 & 11 & 5\end{array}$

Cyclic convolution:

Cyclic Convolution

Example. If

$x=\begin{bmatrix} \phantom{-}1\\ \phantom{-}0\\ \phantom{-}2\\ -1\end{bmatrix}\quad\text{and}\quad y = \begin{bmatrix} 1\\ 2\\ 3\\ 0\end{bmatrix}$

Compute $$x\ast y$$ and $$x\circledast y$$

Answer: $x\ast y = \left[\begin{array}{r} 1\\ 2\\ 5\\ 3\\ 4\\ -3\\ 0\end{array}\right]\quad\text{and}\quad x\circledast y = \left[\begin{array}{r} 5\\ -1\\ 5\\ 3\end{array}\right]$

By John Jasper

• 376