CS6015: Linear Algebra and Random Processes

Lecture 18:  Diagonalisation (Eigenvalue Decomposition) of a matrix, Computing powers of A

Learning Objectives

What is eigenvalue decomposition?

What do we mean by diagonalisation of a matrix? 

or

Why do we care about diagonalisation?

The Eigenstory

real

imaginary

distinct

repeating

\(A^\top\)

\(A^{-1}\)

\(AB\)

\(A^\top A\)

(basis)

powers of A

PCA

optimisation 

diagonalisation

\(A+B\)

\(U\)

\(R\)

\(A^2\)

\(A + kI\)

How to compute eigenvalues?

What are the possible values?

What are the eigenvalues of some special matrices ?

What is the relation between the eigenvalues of related matrices?

What do eigen values reveal about a matrix?

What are some applications in which eigenvalues play an important role?

Identity

Projection

Reflection

Markov

Rotation

Singular

Orthogonal

Rank one

Symmetric

Permutation

det(A - \lambda I) = 0

trace

determinant

invertibility 

rank

nullspace

columnspace

(positive semidefinite matrices)

positive pivots

(independent eigenvectors)

(orthogonal eigenvectors)

... ...

(symmetric)

(where are we?)

(characteristic equation)
(desirable)

HW5

distinct values

independent eigenvectors

\(\implies\)

steady state

(Markov matrices)

eigenvalues

The bigger picture

\(n\) distinct values

\(\implies\)

\(n\) independent eigenvectors

(follows from a theorem)

repeating values*

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

\(n\) independent eigenvectors

\(<n\) independent eigenvectors

(e.g. Identity matrix, Projection matrix)

(diagonalizable)

(diagonalizable)

(not diagonalizable)

\begin{bmatrix} 3&1\\ 0&3 \end{bmatrix}
(e.g.      ) 

The bigger picture

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

(questions)

What do we mean by saying that a matrix is diagonalizable?

Why do we care about diagonalizability?

What is the condition under which we will have \(n\) independent vectors even with repeating eigenvalues?

The bigger picture

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

As usual we will focus on the good case first!

The good case: n independent eigenvectors

Theorem (HW5): If eigenvectors \(s_1, \dots, s_n\) correspond to different eigenvalues \(\lambda_1, \dots, \lambda_n\) , then those eigenvectors are linearly independent

Construct a matrix \(S\) whose columns are these \(n\) independent eigenvectors

\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ s_1&s_2&\dots&s_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
S
A
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ As_1&As_2&\dots&As_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ \lambda_1s_1&\lambda_2s_2&\dots&\lambda_ns_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ s_1&s_2&\dots&s_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix} \begin{bmatrix} \lambda_1&0&\dots&0\\ 0&\lambda_2&\dots&0\\ \dots&\dots&\dots&\lambda_n\\ \end{bmatrix}
AS = S\Lambda

The good case: n independent eigenvectors

AS = S\Lambda
A = S\Lambda S^{-1}
S^{-1}AS = \Lambda

 \(S\) is invertible as it is a square matrix with independent columns

(diagonalisation)

(eigenvalue decomposition)

Why do we care about diagonalisation?

Let's look at some applications

Powers of \(A\)

A\mathbf{x} = \lambda \mathbf{x}
A^2\mathbf{x} = AA\mathbf{x} = A\lambda \mathbf{x} = \lambda A\mathbf{x} = \lambda^2 \mathbf{x}
A^3\mathbf{x} = AA^2\mathbf{x} = A\lambda^2 \mathbf{x} = \lambda^2 A\mathbf{x} = \lambda^3 \mathbf{x}
A^n\mathbf{x} = \lambda^n \mathbf{x}

Powers of \(A\) have the same eigenvectors as \(A\) with the eigenvalues raised to the appropriate power 

Powers of \(A\)

Powers of \(A\) have the same eigenvectors as \(A\) with the eigenvalues raised to the appropriate power 

(the diagonalized view)

A = S\Lambda S^{-1}
A^2 = S\Lambda S^{-1}S\Lambda S^{-1}
=S\Lambda\Lambda S^{-1}
=S\Lambda^2 S^{-1}
A^3 = AA^2 = S\Lambda S^{-1}S\Lambda^2 S^{-1}
=S\Lambda^3 S^{-1}
A^n=S\Lambda^n S^{-1}

Powers of \(A\)

Imagine a particle in 2d space

(why do we care?)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

Suppose the particle moves to a new location every second 

The displacement is given by a matrix

\begin{bmatrix} 1&0.5\\ 0.5&1\\ \end{bmatrix}

Where would the particle be after 24 hours (86400 seconds)?

Powers of \(A\)

1 sec:

(why do we care?)

A\mathbf{v}_0

Where would the particle be after 24 hours (86400 seconds)?

2 sec:

AA\mathbf{v}_0=A^2\mathbf{v}_0

3 sec:

A^3\mathbf{v}_0

... ...

... ...

86400 sec:

A^{86400}\mathbf{v}_0
(very expensive computation)

But what if we use the EVD of A?

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

Powers of \(A\)

(use EVD of A \(\equiv\) change of basis)

k seconds:

A^{k}\mathbf{v}_0
S\Lambda^{k}S^{-1}\mathbf{v}_0
\overbrace{~~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~}

Eigenbasis:

Translating from std. basis to eigenbasis

The basis formed by the eigenvectors of \(A\)

The transformation becomes very simple in this basis

Translate back to the standard basis

(A becomes diagonal in this basis)
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

Powers of \(A\)

(computational efficiency)

[\mathbf{v}]_S
S^{-1}
[A^k\mathbf{v}]_S
\Lambda^k
O(n^2)
A^{k}\mathbf{v} = S\Lambda^{k}S^{-1}\mathbf{v}
S
O(n^2)
O(nk)
O(kn^3)
O(n^2 + nk + n^2)

+ the cost of computing EVs

EVD/Diagonalization/Eigenbasis is useful when the same matrix \(A\) operates on many vectors repeatedly (i.e., if we want to apply \(A^n\) to many vectors)

(this one time cost is then justified in the long run)
\mathbf{v}
A^k\mathbf{v}
A^k
O(kn^3)

The Eigenstory

real

imaginary

distinct

repeating

\(A^\top\)

\(A^{-1}\)

\(AB\)

\(A^\top A\)

(basis)

powers of A

steady state

PCA

optimisation 

diagonalisation

\(A+B\)

\(U\)

\(R\)

\(A^2\)

\(A + kI\)

How to compute eigenvalues?

What are the possible values?

What are the eigenvalues of some special matrices ?

What is the relation between the eigenvalues of related matrices?

What do eigen values reveal about a matrix?

What are some applications in which eigenvalues play an important role?

Identity

Projection

Reflection

Markov

Rotation

Singular

Orthogonal

Rank one

Symmetric

Permutation

det(A - \lambda I) = 0

trace

determinant

invertibility 

rank

nullspace

columnspace

(Markov matrices)

(positive semidefinite matrices)

positive pivots

(independent eigenvectors)

(orthogonal eigenvectors)

... ...

(symmetric)

(where are we?)

(characteristic equation)
(desirable)

HW5

distinct values

independent eigenvectors

\(\implies\)

Let's dig a bit deeper into our previous example

Where would the particle land?

A = \begin{bmatrix} 1&0.5\\ 0.5&1\\ \end{bmatrix}

(after 100 sec.)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
A = S\Lambda S^{-1}
\Lambda = \begin{bmatrix} 1.5&0\\ 0&0.5\\ \end{bmatrix}
S = \begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix}
S^{-1} = \frac{1}{2}\begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix}
(orthogonal)
A^{100}\mathbf{v}_0 = S\Lambda^{100} S^{-1}\mathbf{v}_0
=\frac{1}{2}\begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 1.5^{100}&0\\ 0&0.5^{100}\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

Where would the particle land?

(after 100 sec.)

\frac{1}{2}\begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 1.5^{100}&0\\ 0&0.5^{100}\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\approx 0
\frac{1}{2}\begin{bmatrix} 1*1.5^{100}&\approx0\\ 1*1.5^{100}&\approx0\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 1&0\\ 1&0\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 1&1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 3\\ 3\\ \end{bmatrix}
=\frac{3*1.5^{100}}{2}\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
\begin{bmatrix} -1\\1 \end{bmatrix}
=k\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1\\1 \end{bmatrix}
A^{100}\mathbf{v}_0 =
(some multiple of the dominant eigenvector)

Where would the particle land?

(after 100 sec.)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
\begin{bmatrix} -1\\1 \end{bmatrix}
k\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1\\1 \end{bmatrix}
A^{100}\mathbf{v}_0 =
(some multiple of the dominant eigenvector)

If \(\lambda_1, \lambda_2, \dots, \lambda_n\) are the eigenvalues of a matrix then \(\lambda_i\) is called the dominant eigenvalue if \(|\lambda_i| \geq |\lambda_j|~\forall j\neq i\)

The corresponding eigenvector is called the dominant eigenvector \(e_d\)

\lambda=1.5
\lambda=0.5

Observation:

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0}, \dots, ke_d

Was this a coincidence?

No!

(proof on next slide)

What does this sequence approach?

\mathbf{v}_0 = c_1\mathbf{s}_1 + c_2\mathbf{s}_2 + \cdots + c_n\mathbf{s}_n
\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d
(n independent eigenvectors form a basis. Hence any vector can be written as their linear combination)
A^{k}\mathbf{v}_0 = c_1A^{k}\mathbf{s}_1 + c_2A^{k}\mathbf{s}_2 + \cdots + c_nA^{k}\mathbf{s}_n
A^{k}\mathbf{v}_0 = c_1\lambda_1^{k}\mathbf{s}_1 + c_2\lambda_2^{k}\mathbf{s}_2 + \cdots + c_n\lambda_n^{k}\mathbf{s}_n
(without loss of generality let lambda1 be the dominant eigenvalue)
A^{k}\mathbf{v}_0 = \lambda_1^{k}(c_1\mathbf{s}_1 + c_2(\frac{\lambda_2}{\lambda_1})^{k}\mathbf{s}_2 + \cdots + c_n(\frac{\lambda_n}{\lambda_1})^{k}\mathbf{s}_n)
As~k\rightarrow\infty~~(\frac{\lambda_i}{\lambda_1})^k \rightarrow 0
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}
(hence these terms will disappear)
A^{k}\mathbf{v}_0 \rightarrow \lambda_1^{k}c_1\mathbf{s}_1
(some multiple of the dominant eigenvector)

Note that this result does holds true irrespective of the initial value \(\mathbf{v_0}\)

(as long as c1 is not 0)

What does this sequence approach?

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d

This method of finding the dominant eigenvector is called Power Iteration or the Power method

The sequence approaches the dominant eigenvector of \(A\)

What does the dominant \(\lambda\) tell us?

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d

Let \(p\) be the timestep at which the sequence approaches a multiple of the dominant eigenvector \(e_d\)

A^p\mathbf{v_0} = ke_d
A^{(p+1)}\mathbf{v_0} = AA^{(p)}\mathbf{v_0} = Ake_d = k \lambda_d e_d
A^{(p+2)}\mathbf{v_0} = k \lambda_d^2 e_d
A^{(p+n)}\mathbf{v_0} = k \lambda_d^n e_d
\dots \dots

What does the dominant \(\lambda\) tell us?

A^{(p+n)}\mathbf{v_0} = k \lambda_d^n e_d

What would happen to the sequence \(\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0}, \dots\) if

|\lambda_d| > 1
|\lambda_d| < 1
|\lambda_d| = 1
(will explode)
(will vanish)
(will reach a steady state)

Special case: Markov matrix

Theorem: The dominant eigenvalue of a Markov matrix is 1

Proof (part 1): 1 is an eigenvalue of a Markov matrix

Proof (part 2): all other eigenvalues are less than 1

HW5

HW5

A square matrix \(M\) is called a stochastic (or Markov) matrix if all its values are positive and the sum of the elements in each column (or row) is equal to 1

Special case: Markov matrix

Transition between two restaurants

Transition between two cities

\begin{bmatrix} p&1-q\\ 1-p&q \end{bmatrix} \begin{bmatrix} k_1\\ k_2 \end{bmatrix}
(initial state on Day 0)
M
(transition matrix)

Day 1:

M\mathbf{v}_0

Day 2:

MM\mathbf{v}_0=M^2\mathbf{v}_0

Day 3:

M^3\mathbf{v}_0

... ...

... ...

Day n:

M^{n}\mathbf{v}_0
\mathbf{v}_0
We know that this sequence will approach a multiple of the dominant eigenvector

\(\dots~~\dots\)

Special case: Markov matrix

On some day \(p\)

M^{p}\mathbf{v}_0 = ke_d
M^{(p+1)}\mathbf{v}_0 = Mke_d = k\lambda_d e_d = k e_d
\because \lambda_d = 1
M^{(p+2)}\mathbf{v}_0 = M^2ke_d = k\lambda_d^2 e_d = k e_d

The number of customers in the two restaurants stabilizes and the system reaches a steady state!

the sequence will approach a multiple of the dominant eigenvector

The Eigenstory

real

imaginary

distinct

repeating

\(A^\top\)

\(A^{-1}\)

\(AB\)

\(A^\top A\)

(basis)

powers of A

steady state

PCA

optimisation 

diagonalisation

\(A+B\)

\(U\)

\(R\)

\(A^2\)

\(A + kI\)

How to compute eigenvalues?

What are the possible values?

What are the eigenvalues of some special matrices ?

What is the relation between the eigenvalues of related matrices?

What do eigen values reveal about a matrix?

What are some applications in which eigenvalues play an important role?

Identity

Projection

Reflection

Markov

Rotation

Singular

Orthogonal

Rank one

Symmetric

Permutation

det(A - \lambda I) = 0

trace

determinant

invertibility 

rank

nullspace

columnspace

(Markov matrices)

(positive semidefinite matrices)

positive pivots

(independent eigenvectors)

(orthogonal eigenvectors)

... ...

(symmetric)

(where are we?)

(characteristic equation)
(desirable)

HW5

distinct values

independent eigenvectors

\(\implies\)

Learning Objectives

What is eigenvalue decomposition?

What do we mean by diagonalisation of a matrix? 

or

Why do we care about diagonalisation?