Day 20:

Orthogonal Compliments, Eigenvalues, and Eigenvectors

Orthogonal complements

Definition. Given a set of vectors $$V\subset\R^{n}$$ we can define a set of vectors $$V^{\bot}$$ called the orthogonal complement of $$V$$, as follows

$V^{\bot} = \{x\in\R^{n} : x\cdot v = 0\text{ for all }v\in V\}$

Proposition 1. If $$A$$ is a matrix, then $$C(A^{\top})^{\bot} = N(A)$$.

Proof. Suppose $$x\in N(A)$$. Then $$Ax=0$$. Let $$\{b_{1},\ldots,b_{m}\}$$ be the columns of $$A^{\top}$$. The fact that $$Ax=0$$ implies that $$b_{i}\cdot x=0$$ for $$i=1,\ldots,m$$. If $$y\in C(A^{\top})$$, then

$y = \alpha_{1}b_{1} + \alpha_{2}b_{2} + \cdots + \alpha_{m}b_{m}$

By the linearity of the dot product, we see that $$x\cdot y = 0$$. That is, $$x\in C(A^{\top})^{\bot}$$. The other direction is left as an exercise. $$\Box$$

Orthogonal complements

Proposition 2. Assume $$V\subset\R^{n}$$ is a subspace. For each $$x\in\R^{n}$$ there are unique vectors $$v\in V$$ and $$w\in V^{\bot}$$ such that

$x=v+w.$

Proof. Let $$x\in\R^{n}$$. Let $$P$$ be the orthogonal projection onto $$V$$. Note that $$v:=Px\in V$$ and $$w:=x-Px\in V^{\bot}$$. Clearly

$x=Px+(x-Px).$

Assume there are other vectors $$v'\in V$$ and $$w'\in V^{\bot}$$ such that $$x=v'+w'$$. Multiplying by $$P$$ on the left we have

$Px=Pv'+Pw' = v'+Pw'.$

Since $$w'\in V^{\bot}$$ we know that

$0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').$

This shows that $$Pw'=0$$, that is $$Px=v'$$. $$\Box$$

Finding a basis for $$V^{\perp}$$

Let $$V\subset\R^{n}$$ be a subspace, and assume $$\{v_{1},\ldots,v_{k}\}$$ is a basis for $$V$$. Using Gram-Schmidt, we can find an onb $$\{e_{1},\ldots,e_{k}\}$$ for $$V$$.

Set

$A = \begin{bmatrix} - & e_{1}^{\top} & -\\ - & e_{2}^{\top} & -\\ - & e_{3}^{\top} & -\\ & \vdots & \\ - & e_{k}^{\top} & -\end{bmatrix}$

Then $$V = C(A^{\top})$$, and hence $$V^{\bot}=N(A)$$.

We already know how to find a basis for $$N(A)$$.

By the rank-nullity theorem $$N(A)$$ has dimension $$n-k$$.

So, there is a basis $$\{w_{k+1},\ldots,w_{n}\}$$ for $$N(A)$$.

Using Gram-Schmidt, we obtain an onb $$\{f_{k+1},\ldots,f_{n}\}$$ for $$V^{\bot}$$.

Finding a basis for $$C(A)^{\bot}$$

Example.  Consider the matrix

$A = \left[\begin{array}{rrrr}1 & 1 & 1 & 1\\ -1 & -1 & -1 & -1\\ -1 & -1 & 0 & -2\\ 1 & 1 & 0 & 2 \end{array}\right]$

We want to find a basis for $$C(A)^{\bot}$$.

By the previous theorem $$C(A)^{\bot} = N(A^{\top})$$. We already know how to find a basis for $$N(A^{\top}) = N(\text{rref}(A^{\top}))$$.

$$\text{rref}(A^{\top}) = \begin{bmatrix} 1 & -1 & 0 & 0\\ 0 & 0 & 1 & -1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}$$

$$\left\{\begin{bmatrix} 1\\ 1\\ 0\\ 0\end{bmatrix},\begin{bmatrix} 0\\ 0\\ 1\\ 1\end{bmatrix}\right\}$$ is a basis for $$C(A)^{\bot}$$

We can find a basis for $$N(A)^{\bot}$$ by observing that

$N(A)^{\bot} = (C(A^{\top})^{\bot})^{\bot} = C(A^{\top}).$

Theorem. If $$V\subset \R^{n}$$ is a subspace, then $$(V^{\bot})^{\bot} = V$$.

Prove it!

Theorem. If $$V\subset \R^{n}$$ is a subspace, then $$(V^{\bot})^{\bot} = V$$.

Proof. Let $$x\in (V^{\bot})^{\bot}$$.  Let $$P$$ be the orthogonal projection onto $$V$$. We have already shown that $$x-Px\in V^{\bot}$$. Let $$y\in V^{\bot}$$, then $y\cdot(x-Px) = y\cdot x - y\cdot(Px).$ By assumption $$y\cdot x=0$$, and since $$Px\in V$$ we also conclude that

$$y\cdot(Px)=0$$, thus $$y\cdot(x-Px)=0$$. Since $$y\in V^{\bot}$$ was arbitrary, this shows that $$x-Px\in (V^{\bot})^{\bot}$$.

Thus $$x-Px$$ is in both $$V^{\bot}$$ and $$(V^{\bot})^{\bot}$$. This implies $(x-Px)\cdot (x-Px)=0,$ and hence $$x-Px=0$$. Therefore $$x=Px\in V$$. Thus, we have shown that $$(V^{\bot})^{\bot} \subset V$$.

Let $$x\in V$$, and let $$y\in V^{\bot}$$ be arbitrary. It is immediate that $$x\cdot y=0$$, and since $$y\in V^{\bot}$$ was arbitrary, we conclude that $$x\in(V^{\bot})^{\bot}$$. Therefore, $$V\subset(V^{\bot})^{\bot}$$. $$\Box$$

Orthogonal complements

Proposition. Assume $$V\subset\R^{n}$$ is a subspace. For each $$x\in\R^{n}$$ there are unique vectors $$v\in V$$ and $$w\in V^{\bot}$$ such that

$x=v+w.$

Proof. Let $$x\in\R^{n}$$. Let $$P$$ be the orthogonal projection onto $$V$$. Note that $$v:=Px\in V$$ and $$w:=x-Px\in V^{\bot}$$. Clearly

$x=Px+(x-Px).$

Assume there are other vectors $$v'\in V$$ and $$w'\in V^{\bot}$$ such that $$x=v'+w'$$. Multiplying by $$P$$ on the left we have

$Px=Pv'+Pw' = v'+Pw'.$

Since $$w'\in V^{\bot}$$ we know that

$0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').$

This shows that $$Pw'=0$$, that is $$Px=v'$$. $$\Box$$

"Eigen"-things

Definition. Let $$A$$ be an $$n\times n$$ matrix. A nonzero vector $$v\in\R^{n}$$ is called an eigenvector of $$A$$ if there is a scalar $$\lambda$$ such that

$Av=\lambda v$

The number $$\lambda$$ is called an eigenvalue of $$A$$ associated with $$v.$$

Example 1. $$\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}\begin{bmatrix} 1\\ 0\end{bmatrix} = \begin{bmatrix}3\\ 0\end{bmatrix} = 3\begin{bmatrix}1\\ 0\end{bmatrix}$$

Thus,

• $$\begin{bmatrix}1\\ 0\end{bmatrix}$$ is an eigenvector of $$\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}$$ and
• $$3$$ is an eigenvalue of $$\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}$$ associated with $$\begin{bmatrix} 1 \\ 0\end{bmatrix}$$

"Eigen"-things

Example 2. $$\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix} = \begin{bmatrix} -9\\ 6\\ 3\end{bmatrix} = 3\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}$$

Thus,

• $$\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}$$ is an eigenvector of $$\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}$$
• $$3$$ is an eigenvalue of $$\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}$$ with associated eigenvector $$\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}$$

Two new problems

Now, given a matrix $$A$$ we have two new problems:

1. Find all of the eigenvalues of $$A$$.
2. Given an eigenvalue $$\lambda$$, find all eigenvectors of $$A$$ with associated eigenvalue $$\lambda$$.

We're going to focus on this one (because it's simpler)

Finding eigenvectors

Given an $$n\times n$$ matrix $$A$$, it is difficult to find the eigenvalues of $$A$$ (if it even has any!) However, if we are told that $$\lambda$$ is an eigenvalue of $$A$$, then we simply need to solve

$Ax=\lambda x.$

Example. Solve

$\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}-2x_{1}\\ -2x_{2}\\ -2x_{3}\end{bmatrix}$

Finding eigenvectors

Given an $$n\times n$$ matrix $$A$$, it is difficult to find the eigenvalues of $$A$$ (if it even has any!) However, if we are told that $$\lambda$$ is an eigenvalue of $$A$$, then we simply need to solve

$Ax=\lambda I x.$

Example. Solve

$\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix}$

Finding eigenvectors

Given an $$n\times n$$ matrix $$A$$, it is difficult to find the eigenvalues of $$A$$ (if it even has any!) However, if we are told that $$\lambda$$ is an eigenvalue of $$A$$, then we simply need to solve

$Ax-\lambda I x=0.$

Example. Solve

$\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} - \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}$

Finding eigenvectors

Given an $$n\times n$$ matrix $$A$$, it is difficult to find the eigenvalues of $$A$$ (if it even has any!) However, if we are told that $$\lambda$$ is an eigenvalue of $$A$$, then we simply need to solve

$(A-\lambda I )x=0.$

Example. Solve

$\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}$

The eigenspace of $$A$$ associated with the eigenvalue $$\lambda$$ is $$N(A-\lambda I)$$

Finding eigenvectors

Example. Solve

$\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}$

Note that

$\text{rref}\left(\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\right) = \left[\begin{array}{rrr} 1 & 0 & -\frac{3}{2}\\[1ex] 0 & 1 & \frac{1}{2}\\[1ex] 0 & 0 & 0\end{array}\right]$

Hence, a basis for the eigenspace associated with the eigenvalue $$-2$$ is

$\left\{\begin{bmatrix}\frac{3}{2}\\[1ex] -\frac{1}{2}\\[1ex] 1\end{bmatrix}\right\}$

Proposition. If $$A$$ is an $$n\times n$$ matrix, and $$\lambda$$ is an eigenvalue of $$A$$, then the set

$W=\{x\in \R^{n} : Ax=\lambda x\} = N(A-\lambda I)$

is a subspace. This subspace is called the eigenspace of $$A$$ associated with $$\lambda$$.

Proof. Let $$x,y$$ be two vectors in $$W$$ and let $$\alpha\in \R$$, then we have

$A(x+y) = Ax+Ay = \lambda x + \lambda y = \lambda(x+y)$

which shows that $$x+y\in W$$. We also have

$A(\alpha x) = \alpha Ax = \alpha\lambda x = \lambda(\alpha x)$

which shows that $$\alpha x\in W$$. $$\Box$$

Note: The set of eigenvectors associated with $$\lambda$$ is not the same as the eigenspace associated with $$\lambda$$, since $$0$$ is in the latter.

Two new problems

Now, given a matrix $$A$$ we have two new problems:

1. Find all of the eigenvalues of $$A$$.
2. Given an eigenvalue $$\lambda$$, find all eigenvectors of $$A$$ with associated eigenvalue $$\lambda$$.

Now we have to work on the harder problem

Finding eigenvalues

Theorem. Let $$A$$ be a square matrix. The following are equivalent:

1. $$\lambda$$ is an eigenvalue of $$A$$
2. $$(A-\lambda I)x=0$$ has a nontrivial solution
3. $$N(A-\lambda I)\neq \{0\}$$
4. The columns of $$A-\lambda I$$ are dependent.
5. $$A-\lambda I$$ is not invertible.

Example. Assume there is a number $$\lambda\in\R$$ such that

$\begin{bmatrix}1 & -1\\ 1 & 1\end{bmatrix} - \lambda\begin{bmatrix}1 & 0\\ 0 & 1\end{bmatrix} = \begin{bmatrix} 1-\lambda & -1\\ 1 & 1-\lambda\end{bmatrix}$

has dependent columns. Then there is a constant $$c$$ such that

$c\begin{bmatrix} 1-\lambda\\1\end{bmatrix} = \begin{bmatrix} -1\\ 1-\lambda \end{bmatrix}$

This means $$c=1-\lambda$$, and hence $$(1-\lambda)^2=-1$$, but this is false.

This matrix has no eigenvalues!

Similar matrices

Definition. Two matrices $$A$$ and $$B$$ are called similar if there is an invertible matrix $$X$$ such that

$A=XBX^{-1}.$

(Note that this definition implies that $$A$$ and $$B$$ are both square and the same size.)

Theorem. If $$A$$ and $$B$$ are similar, then they have the exact same set of eigenvalues.

Proof. Assume $$A=XBX^{-1}$$ and $$\lambda$$ is an eigenvalue of $$B$$. This means that $$Bx=\lambda x$$ for some nonzero $$x$$ and some scalar $$\lambda$$. Set $$y=Xx$$. Since $$X$$ in invertible, $$y$$ is not the zero vector, and

$Ay = XBX^{-1}Xx = XBx=X\lambda x = \lambda Xx = \lambda y$

hence $$\lambda$$ is an eigenvalue of $$A$$. Since $$B=X^{-1}AX$$, the other direction is similar. $$\Box$$

Similar matrices

Example. Consider the diagonal matrix

$A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},$

and the invertible matrix

$X = \begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\quad\text{with inverse }\quad X^{-1} = \begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix}.$

The matrix $XAX^{-1}=\begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix}\begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},$

is similar to $$A$$.

Similar matrices

Example continued.  The eigenvalues and eigenvectors of $$A$$ are obvious. The eigenvectors of $$XAX^{-1}$$ are not.

$XAX^{-1} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},$

$A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},$

Eigenvalue $$\lambda$$

Eigenspace $$N(A-\lambda I)$$

$$\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}$$

$$\text{span}\left\{\begin{bmatrix} 0\\ 1\\ 0\end{bmatrix}\right\}$$

$$\text{span}\left\{\begin{bmatrix} 0\\ 0\\ 1\end{bmatrix}\right\}$$

$$1$$

$$-2$$

$$3$$

Eigenvalue $$\lambda$$

Eigenspace $$N(XAX^{-1}-\lambda I)$$

$$\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}$$

$$\text{span}\left\{\begin{bmatrix} 2\\ 1\\ 0\end{bmatrix}\right\}$$

$$\text{span}\left\{\begin{bmatrix} 0\\ -1\\ 1\end{bmatrix}\right\}$$

$$1$$

$$-2$$

$$3$$

Diagonalizable matrices

Definition. A matrix $$A$$ is called diagonalizable if it is similar to a diagonal matrix.

Example. Take the diagonal matrix

$A = \begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}$

Set

$B = \begin{bmatrix}-3 & -2 & 0 & 4\\ 1 & 2 & -2 & -3\\ -1 & -2 & 3 & 3\\ -3 & -3 & -1 & 5\end{bmatrix}\begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\begin{bmatrix} 1 & -8 & -6 & -2\\ 4 & -14 & -11 & -5\\ 0 & 1 & 1 & 0\\ 3 & -13 & -10 & -4\end{bmatrix}$

$= \left[\begin{array}{rrrr}-30 & 132 & 102 & 42\\ 26 & -98 & -76 & -34\\ -26 & 97 & 75 & 34\\ -42 & 175 & 136 & 57\end{array}\right]$

So, $$B$$ is diagonalizable

By John Jasper

• 564