# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (symmetric)

(A - \lambda I)\mathbf{x} = \mathbf{0}

# Intuition

\mathbf{x}

### What happens when a matrix hits a vector?

A\mathbf{x}
\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\begin{bmatrix} 1\\ 2\\ \end{bmatrix}
=\begin{bmatrix} 5\\ 4 \end{bmatrix}

### Most vectors get knocked off their span

(i.e., most vectors change their direction)

### But some special vectors stay on the span

\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
=\begin{bmatrix} 3\\ 3 \end{bmatrix}

### These are called eigenvectors and the scaling factor is called the eigenvalue

=3\begin{bmatrix} 1\\ 1 \end{bmatrix}
(i.e., they only get scaled or squished)

# Definition

\mathbf{x}

### The span of a vector is a line $$c\mathbf{u}~\forall c \in \mathbb{R}$$

\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\begin{bmatrix} 2\\ 2\\ \end{bmatrix}
=\begin{bmatrix} 6\\ 6 \end{bmatrix}

### Observation 1:

=3\begin{bmatrix} 2\\ 2 \end{bmatrix}

### If $$\mathbf{x}$$ is an eigenvector, then so is $$c\mathbf{x}$$

(i.e., any vector in the span of x)

### The vector $$\mathbf{x}$$ is an eigenvector of A if

A\mathbf{x} = \lambda\mathbf{x}

# Definition

\mathbf{x}

### The vector $$\mathbf{x}$$ is an eigenvector of A if

A\mathbf{x} = \lambda\mathbf{x}

### If $$\mathbf{x}$$ is an eigenvector, then it lies in the column space of A

(this is useful and we will return back to it later)

# Definition

\mathbf{x}

### The vector $$\mathbf{x}$$ is an eigenvector of A if

A\mathbf{x} = \lambda\mathbf{x}

### Eigenvectors only make sense for square matrices

(these two vectors can never be the same as they are in two different spaces)
A\mathbf{x} = \lambda\mathbf{x'}
m\times n
n\times1
m\times1

# How do we compute the eigenvalues?

A\mathbf{x} = \lambda\mathbf{x}
\implies A\mathbf{x} = \lambda I\mathbf{x}
\implies A\mathbf{x} - \lambda I\mathbf{x} = 0
(we now have a matrix on both sides)
\implies (A - \lambda I)\mathbf{x} = 0
(trivial solution: x = 0 -- not very interesting)
(we are looking for a non-zero eigenvector)

### $$\implies$$ $$det(A - \lambda I) = 0$$

\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\mathbf{x} = \lambda\mathbf{x}
\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\mathbf{x} = \lambda\begin{bmatrix} 1&0\\ 0&1\\ \end{bmatrix}\mathbf{x}
\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}\mathbf{x} - \lambda\begin{bmatrix} 1&0\\ 0&1\\ \end{bmatrix}\mathbf{x} = 0
\begin{bmatrix} 1-\lambda&2\\ 2&1-\lambda\\ \end{bmatrix}\mathbf{x}= 0

# How do we compute the eigenvalues?

### $$det(A - \lambda I) = 0$$

A=\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}
A-\lambda I=\begin{bmatrix} 1-\lambda&2\\ 2&1-\lambda\\ \end{bmatrix}
det(A-\lambda I)= (1-\lambda)\cdot(1-\lambda) - 2\cdot2 = 0
\lambda^2 - 2\lambda - 3 = 0
(\lambda - 3)(\lambda + 1) = 0
\lambda = 3, \lambda = -1
(this is called the characteristic equation)

# How do we compute the eigenvectors?

### $$det(A - \lambda I) = 0$$

A=\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}
(A-\lambda I)\mathbf{x} = 0
\lambda = 3, \lambda = -1
\begin{bmatrix} 1-3&2\\ 2&1-3\\ \end{bmatrix}\mathbf{x} = \mathbf{0}
(Ax = 0 - we know how to solve this)
\lambda = 3
\begin{bmatrix} -2&2\\ 2&-2\\ \end{bmatrix}\mathbf{x} = \mathbf{0}
\mathbf{x}=\begin{bmatrix} 1\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1-(-1)&2\\ 2&1-(-1)\\ \end{bmatrix}\mathbf{x} = \mathbf{0}
\lambda = -1
\begin{bmatrix} 2&2\\ 2&2\\ \end{bmatrix}\mathbf{x} = \mathbf{0}
\mathbf{x}=\begin{bmatrix} 1\\ -1\\ \end{bmatrix}

# The $$n \times n$$ case

### $$det(A - \lambda I) = 0$$

\begin{bmatrix} a_{11}-\lambda&a_{12}&a_{13}&\cdots&a_{1n}\\ a_{21}&a_{22-\lambda}&a_{23}&\cdots&a_{2n}\\ a_{31}&a_{32}&a_{33}-\lambda&\cdots&a_{3n}\\ \cdots&\cdots&\cdots&\cdots&\cdots\\ \cdots&\cdots&\cdots&\cdots&\cdots\\ a_{n1}&a_{n2}&a_{n3}&\cdots&a_{nn}-\lambda\\ \end{bmatrix}
Characteristic Equation:
\lambda^n + \cdots = 0

### We expect $$n$$ roots: $$n$$ eigenvalues, $$n$$ eigenvectors

(but sometimes things could go wrong)

# What could go wrong?

### The not-so-good case

A=\begin{bmatrix} 1&2\\ 2&1\\ \end{bmatrix}
\lambda = 3, \lambda = -1

### $$n$$ independent eigenvectors

(I see a basis there - coming soon)
A=\begin{bmatrix} 0&-1\\ 1&0\\ \end{bmatrix}
\begin{vmatrix} 1-\lambda&2\\ 2&1-\lambda\\ \end{vmatrix} = 0
\lambda^2 - 2\lambda - 3 = 0
\begin{vmatrix} 0-\lambda&-1\\ 1&0-\lambda\\ \end{vmatrix} = 0
\lambda^2 + 1 = 0
\lambda = i, \lambda = -i

### imaginary eigenvectors

(In many real world applications, imaginary values are not good)
A=\begin{bmatrix} 3&1\\ 0&3\\ \end{bmatrix}
\begin{vmatrix} 3-\lambda&1\\ 0&3-\lambda\\ \end{vmatrix} = 0
(3 - \lambda)(3 - \lambda) = 0
\lambda = 3, \lambda = 3

### $$< n$$ independent eigenvectors

(I see an incomplete basis there - coming soon)

# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (where are we?)

(characteristic equation)
(desirable)
(A - \lambda I)\mathbf{x} = \mathbf{0}

# EVs of some special matrices: Identity

I=\begin{bmatrix} 1&0&0\\ 0&1&0\\ 0&0&1\\ \end{bmatrix}
I=\begin{bmatrix} 1-\lambda&0&0\\ 0&1-\lambda&0\\ 0&0&1-\lambda\\ \end{bmatrix}
(1-\lambda)^3= 0
\lambda_1=1,\lambda_2=1,\lambda_3=1

### but no shortage of eigenvectors

(every n dimensional vector is an eigenvector of I)

# EVs of some special matrices:Projection

### $$\mathbf{x}$$ in column space of $$A$$

column space of A
\mathbf{p}
\mathbf{a}_1 \in \mathbb{R}^n
\mathbf{a}_2
P = A(A^\top A)^{-1}A^\top

### If $$\mathbf{x} \in \mathbb{R}^n$$ there are three possibilities

(Px = 1.x)

### $$\mathbf{x}$$ is orthogonal to column space of $$A$$

(Px = 0.x)

### $$\mathbf{x}$$ is at some angle  to the column space of $$A$$

(Px != c.x) - direction will change

### Corresponding eigenvectors: nullspace of $$A^\top$$, column space of $$A$$

(angle = 0)
(angle = 90)
(angle != 0,90)

# EVs of some special matrices:Rotation

A=\begin{bmatrix} cos\theta&-sin\theta\\ sin\theta&cos\theta\\ \end{bmatrix}
A=\begin{bmatrix} 0&-1\\ 1&0\\ \end{bmatrix}
\theta=90\degree

### What is the question that we are asking?

(Which vectors will not move?)

### Will we get a good answer?

(Obviously not!)
\lambda_1=i
u_1=\begin{bmatrix} i\\1 \end{bmatrix}
\lambda_2=-i
u_2=\begin{bmatrix} -i\\1 \end{bmatrix}

# EVs of some special matrices:Singular

### (non-zero)

(special solution(s) of Ax = 0)

### Hence, 0 is always an eigenvalue of any singular (non-invertible) matrix

A\mathbf{x}=\mathbf{0}
A\mathbf{x}=\mathbf{0}\cdot\mathbf{x}

### What are the corresponding eigenvectors?

I=\begin{bmatrix} 1&2&3\\ 1&2&3\\ 1&2&3\\ \end{bmatrix}
\lambda_1 = 0, \lambda_2 = 0, \lambda_3 = 6
\mathbf{x_1} = \begin{bmatrix} -3\\0\\1 \end{bmatrix}
\mathbf{x_2} = \begin{bmatrix} -2\\1\\0 \end{bmatrix}
\mathbf{x_3} = \begin{bmatrix} 1\\1\\1 \end{bmatrix}

# Do EVs tell us anything about the rank?

### $$\implies r = n~-$$ number of 0 eigenvalues

(Case1: no 0 eigenvalue)
(Case2: one or more 0 eigenvalues)
(Counter Example)
I=\begin{bmatrix} 0&3&0\\ 0&0&3\\ 0&0&0\\ \end{bmatrix}
r=2\\ \lambda_1=\lambda_2=\lambda_3 = 0

# The Eigenstory

### Permutation

det(A - \lambda I) = 0

### (where are we?)

(desirable)

### $$\implies$$

(characteristic equation)
(A - \lambda I)\mathbf{x} = \mathbf{0}