# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (where are we?)

(characteristic equation)
(desirable)

# The bigger picture

### $$n$$ independent eigenvectors

(follows from a theorem)

### repeating values*

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

### $$<n$$ independent eigenvectors

(e.g. Identity matrix, Projection matrix)

### (not diagonalizable)

\begin{bmatrix} 3&1\\ 0&3 \end{bmatrix}
(e.g.      )

# The bigger picture

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

# The bigger picture

* more than 1 value can repeat - e.g. in a projection matrix both the eigenvalues 1 and 0 may repeat

# The good case: n independent eigenvectors

### Construct a matrix $$S$$ whose columns are these $$n$$ independent eigenvectors

\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ s_1&s_2&\dots&s_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
S
A
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ As_1&As_2&\dots&As_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ \lambda_1s_1&\lambda_2s_2&\dots&\lambda_ns_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix}
=\begin{bmatrix} \uparrow&\uparrow&\dots&\uparrow\\ s_1&s_2&\dots&s_n\\ \downarrow&\downarrow&\dots&\downarrow\\ \end{bmatrix} \begin{bmatrix} \lambda_1&0&\dots&0\\ 0&\lambda_2&\dots&0\\ \dots&\dots&\dots&\lambda_n\\ \end{bmatrix}
AS = S\Lambda

# The good case: n independent eigenvectors

AS = S\Lambda
A = S\Lambda S^{-1}
S^{-1}AS = \Lambda

# Powers of $$A$$

A\mathbf{x} = \lambda \mathbf{x}
A^2\mathbf{x} = AA\mathbf{x} = A\lambda \mathbf{x} = \lambda A\mathbf{x} = \lambda^2 \mathbf{x}
A^3\mathbf{x} = AA^2\mathbf{x} = A\lambda^2 \mathbf{x} = \lambda^2 A\mathbf{x} = \lambda^3 \mathbf{x}
A^n\mathbf{x} = \lambda^n \mathbf{x}

# Powers of $$A$$

### (the diagonalized view)

A = S\Lambda S^{-1}
A^2 = S\Lambda S^{-1}S\Lambda S^{-1}
=S\Lambda\Lambda S^{-1}
=S\Lambda^2 S^{-1}
A^3 = AA^2 = S\Lambda S^{-1}S\Lambda^2 S^{-1}
=S\Lambda^3 S^{-1}
A^n=S\Lambda^n S^{-1}

# Powers of $$A$$

### (why do we care?)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

### The displacement is given by a matrix

\begin{bmatrix} 1&0.5\\ 0.5&1\\ \end{bmatrix}

# Powers of $$A$$

A\mathbf{v}_0

### 2 sec:

AA\mathbf{v}_0=A^2\mathbf{v}_0

A^3\mathbf{v}_0

### 86400 sec:

A^{86400}\mathbf{v}_0
(very expensive computation)

### But what if we use the EVD of A?

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

# Powers of $$A$$

### k seconds:

A^{k}\mathbf{v}_0
S\Lambda^{k}S^{-1}\mathbf{v}_0
\overbrace{~~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~}

### Translate back to the standard basis

(A becomes diagonal in this basis)
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

# Powers of $$A$$

### (computational efficiency)

[\mathbf{v}]_S
S^{-1}
[A^k\mathbf{v}]_S
\Lambda^k
O(n^2)
A^{k}\mathbf{v} = S\Lambda^{k}S^{-1}\mathbf{v}
S
O(n^2)
O(nk)
O(kn^3)
O(n^2 + nk + n^2)

### EVD/Diagonalization/Eigenbasis is useful when the same matrix $$A$$ operates on many vectors repeatedly (i.e., if we want to apply $$A^n$$ to many vectors)

(this one time cost is then justified in the long run)
\mathbf{v}
A^k\mathbf{v}
A^k
O(kn^3)

# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (where are we?)

(characteristic equation)
(desirable)

# Where would the particle land?

A = \begin{bmatrix} 1&0.5\\ 0.5&1\\ \end{bmatrix}

### (after 100 sec.)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
A = S\Lambda S^{-1}
\Lambda = \begin{bmatrix} 1.5&0\\ 0&0.5\\ \end{bmatrix}
S = \begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix}
S^{-1} = \frac{1}{2}\begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix}
(orthogonal)
A^{100}\mathbf{v}_0 = S\Lambda^{100} S^{-1}\mathbf{v}_0
=\frac{1}{2}\begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 1.5^{100}&0\\ 0&0.5^{100}\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}

# Where would the particle land?

### (after 100 sec.)

\frac{1}{2}\begin{bmatrix} 1&-1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 1.5^{100}&0\\ 0&0.5^{100}\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\approx 0
\frac{1}{2}\begin{bmatrix} 1*1.5^{100}&\approx0\\ 1*1.5^{100}&\approx0\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 1&0\\ 1&0\\ \end{bmatrix} \begin{bmatrix} 1&1\\ -1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 1&1\\ 1&1\\ \end{bmatrix} \begin{bmatrix} 2\\ 1\\ \end{bmatrix}
\frac{1.5^{100}}{2}\begin{bmatrix} 3\\ 3\\ \end{bmatrix}
=\frac{3*1.5^{100}}{2}\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
\begin{bmatrix} -1\\1 \end{bmatrix}
=k\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1\\1 \end{bmatrix}
A^{100}\mathbf{v}_0 =
(some multiple of the dominant eigenvector)

# Where would the particle land?

### (after 100 sec.)

\mathbf{v}_0=\begin{bmatrix} 2\\1 \end{bmatrix}
\begin{bmatrix} -1\\1 \end{bmatrix}
k\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1\\1 \end{bmatrix}
A^{100}\mathbf{v}_0 =
(some multiple of the dominant eigenvector)

\lambda=1.5
\lambda=0.5

### Observation:

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0}, \dots, ke_d

### No!

(proof on next slide)

# What does this sequence approach?

\mathbf{v}_0 = c_1\mathbf{s}_1 + c_2\mathbf{s}_2 + \cdots + c_n\mathbf{s}_n
\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d
(n independent eigenvectors form a basis. Hence any vector can be written as their linear combination)
A^{k}\mathbf{v}_0 = c_1A^{k}\mathbf{s}_1 + c_2A^{k}\mathbf{s}_2 + \cdots + c_nA^{k}\mathbf{s}_n
A^{k}\mathbf{v}_0 = c_1\lambda_1^{k}\mathbf{s}_1 + c_2\lambda_2^{k}\mathbf{s}_2 + \cdots + c_n\lambda_n^{k}\mathbf{s}_n
(without loss of generality let lambda1 be the dominant eigenvalue)
A^{k}\mathbf{v}_0 = \lambda_1^{k}(c_1\mathbf{s}_1 + c_2(\frac{\lambda_2}{\lambda_1})^{k}\mathbf{s}_2 + \cdots + c_n(\frac{\lambda_n}{\lambda_1})^{k}\mathbf{s}_n)
As~k\rightarrow\infty~~(\frac{\lambda_i}{\lambda_1})^k \rightarrow 0
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}
(hence these terms will disappear)
A^{k}\mathbf{v}_0 \rightarrow \lambda_1^{k}c_1\mathbf{s}_1
(some multiple of the dominant eigenvector)

### Note that this result does holds true irrespective of the initial value $$\mathbf{v_0}$$

(as long as c1 is not 0)

# What does this sequence approach?

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d

# What does the dominant $$\lambda$$ tell us?

\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0},~to~ke_d

### Let $$p$$ be the timestep at which the sequence approaches a multiple of the dominant eigenvector $$e_d$$

A^p\mathbf{v_0} = ke_d
A^{(p+1)}\mathbf{v_0} = AA^{(p)}\mathbf{v_0} = Ake_d = k \lambda_d e_d
A^{(p+2)}\mathbf{v_0} = k \lambda_d^2 e_d
A^{(p+n)}\mathbf{v_0} = k \lambda_d^n e_d
\dots \dots

# What does the dominant $$\lambda$$ tell us?

A^{(p+n)}\mathbf{v_0} = k \lambda_d^n e_d

### What would happen to the sequence $$\mathbf{v_0}, A\mathbf{v_0}, A^2\mathbf{v_0}, A^3\mathbf{v_0}, \dots$$ if

|\lambda_d| > 1
|\lambda_d| < 1
|\lambda_d| = 1
(will explode)
(will vanish)
(will reach a steady state)

# Special case: Markov matrix

### Transition between two cities

\begin{bmatrix} p&1-q\\ 1-p&q \end{bmatrix} \begin{bmatrix} k_1\\ k_2 \end{bmatrix}
(initial state on Day 0)
M
(transition matrix)

M\mathbf{v}_0

### Day 2:

MM\mathbf{v}_0=M^2\mathbf{v}_0

M^3\mathbf{v}_0

### Day n:

M^{n}\mathbf{v}_0
\mathbf{v}_0
We know that this sequence will approach a multiple of the dominant eigenvector

# Special case: Markov matrix

### On some day $$p$$

M^{p}\mathbf{v}_0 = ke_d
M^{(p+1)}\mathbf{v}_0 = Mke_d = k\lambda_d e_d = k e_d
\because \lambda_d = 1
M^{(p+2)}\mathbf{v}_0 = M^2ke_d = k\lambda_d^2 e_d = k e_d

### The number of customers in the two restaurants stabilizes and the system reaches a steady state!

the sequence will approach a multiple of the dominant eigenvector

# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (where are we?)

(characteristic equation)
(desirable)