Day 20:
Orthogonal Compliments, Eigenvalues, and Eigenvectors
Orthogonal complements
Definition. Given a set of vectors \(V\subset\R^{n}\) we can define a set of vectors \(V^{\bot}\) called the orthogonal complement of \(V\), as follows
\[V^{\bot} = \{x\in\R^{n} : x\cdot v = 0\text{ for all }v\in V\}\]
Proposition 1. If \(A\) is a matrix, then \(C(A^{\top})^{\bot} = N(A)\).
Proof. Suppose \(x\in N(A)\). Then \(Ax=0\). Let \(\{b_{1},\ldots,b_{m}\}\) be the columns of \(A^{\top}\). The fact that \(Ax=0\) implies that \(b_{i}\cdot x=0\) for \(i=1,\ldots,m\). If \(y\in C(A^{\top})\), then
\[y = \alpha_{1}b_{1} + \alpha_{2}b_{2} + \cdots + \alpha_{m}b_{m}\]
By the linearity of the dot product, we see that \(x\cdot y = 0\). That is, \(x\in C(A^{\top})^{\bot}\). The other direction is left as an exercise. \(\Box\)
Orthogonal complements
Proposition 2. Assume \(V\subset\R^{n}\) is a subspace. For each \(x\in\R^{n}\) there are unique vectors \(v\in V\) and \(w\in V^{\bot}\) such that
\[x=v+w.\]
Proof. Let \(x\in\R^{n}\). Let \(P\) be the orthogonal projection onto \(V\). Note that \(v:=Px\in V\) and \(w:=x-Px\in V^{\bot}\). Clearly
\[x=Px+(x-Px).\]
Assume there are other vectors \(v'\in V\) and \(w'\in V^{\bot}\) such that \(x=v'+w'\). Multiplying by \(P\) on the left we have
\[Px=Pv'+Pw' = v'+Pw'.\]
Since \(w'\in V^{\bot}\) we know that
\[0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').\]
This shows that \(Pw'=0\), that is \(Px=v'\). \(\Box\)
Finding a basis for \(V^{\perp}\)
Let \(V\subset\R^{n}\) be a subspace, and assume \(\{v_{1},\ldots,v_{k}\}\) is a basis for \(V\). Using Gram-Schmidt, we can find an onb \(\{e_{1},\ldots,e_{k}\}\) for \(V\).
Set
\[A = \begin{bmatrix} - & e_{1}^{\top} & -\\ - & e_{2}^{\top} & -\\ - & e_{3}^{\top} & -\\ & \vdots & \\ - & e_{k}^{\top} & -\end{bmatrix}\]
Then \(V = C(A^{\top})\), and hence \(V^{\bot}=N(A)\).
We already know how to find a basis for \(N(A)\).
By the rank-nullity theorem \(N(A)\) has dimension \(n-k\).
So, there is a basis \(\{w_{k+1},\ldots,w_{n}\}\) for \(N(A)\).
Using Gram-Schmidt, we obtain an onb \(\{f_{k+1},\ldots,f_{n}\}\) for \(V^{\bot}\).
Finding a basis for \(C(A)^{\bot}\)
Example. Consider the matrix
\[A = \left[\begin{array}{rrrr}1 & 1 & 1 & 1\\ -1 & -1 & -1 & -1\\ -1 & -1 & 0 & -2\\ 1 & 1 & 0 & 2 \end{array}\right]\]
We want to find a basis for \(C(A)^{\bot}\).
By the previous theorem \(C(A)^{\bot} = N(A^{\top})\). We already know how to find a basis for \(N(A^{\top}) = N(\text{rref}(A^{\top}))\).
\(\text{rref}(A^{\top}) = \begin{bmatrix} 1 & -1 & 0 & 0\\ 0 & 0 & 1 & -1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\)
\(\left\{\begin{bmatrix} 1\\ 1\\ 0\\ 0\end{bmatrix},\begin{bmatrix} 0\\ 0\\ 1\\ 1\end{bmatrix}\right\}\) is a basis for \(C(A)^{\bot}\)
We can find a basis for \(N(A)^{\bot}\) by observing that
\[N(A)^{\bot} = (C(A^{\top})^{\bot})^{\bot} = C(A^{\top}).\]
Theorem. If \(V\subset \R^{n}\) is a subspace, then \((V^{\bot})^{\bot} = V\).
Prove it!
Theorem. If \(V\subset \R^{n}\) is a subspace, then \((V^{\bot})^{\bot} = V\).
Proof. Let \(x\in (V^{\bot})^{\bot}\). Let \(P\) be the orthogonal projection onto \(V\). We have already shown that \(x-Px\in V^{\bot}\). Let \(y\in V^{\bot}\), then \[y\cdot(x-Px) = y\cdot x - y\cdot(Px).\] By assumption \(y\cdot x=0\), and since \(Px\in V\) we also conclude that
\(y\cdot(Px)=0\), thus \(y\cdot(x-Px)=0\). Since \(y\in V^{\bot}\) was arbitrary, this shows that \(x-Px\in (V^{\bot})^{\bot}\).
Thus \(x-Px\) is in both \(V^{\bot}\) and \((V^{\bot})^{\bot}\). This implies \[(x-Px)\cdot (x-Px)=0,\] and hence \(x-Px=0\). Therefore \(x=Px\in V\). Thus, we have shown that \((V^{\bot})^{\bot} \subset V\).
Let \(x\in V\), and let \(y\in V^{\bot}\) be arbitrary. It is immediate that \(x\cdot y=0\), and since \(y\in V^{\bot}\) was arbitrary, we conclude that \(x\in(V^{\bot})^{\bot}\). Therefore, \(V\subset(V^{\bot})^{\bot}\). \(\Box\)
Orthogonal complements
Proposition. Assume \(V\subset\R^{n}\) is a subspace. For each \(x\in\R^{n}\) there are unique vectors \(v\in V\) and \(w\in V^{\bot}\) such that
\[x=v+w.\]
Proof. Let \(x\in\R^{n}\). Let \(P\) be the orthogonal projection onto \(V\). Note that \(v:=Px\in V\) and \(w:=x-Px\in V^{\bot}\). Clearly
\[x=Px+(x-Px).\]
Assume there are other vectors \(v'\in V\) and \(w'\in V^{\bot}\) such that \(x=v'+w'\). Multiplying by \(P\) on the left we have
\[Px=Pv'+Pw' = v'+Pw'.\]
Since \(w'\in V^{\bot}\) we know that
\[0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').\]
This shows that \(Pw'=0\), that is \(Px=v'\). \(\Box\)
"Eigen"-things
Definition. Let \(A\) be an \(n\times n\) matrix. A nonzero vector \(v\in\R^{n}\) is called an eigenvector of \(A\) if there is a scalar \(\lambda\) such that
\[Av=\lambda v\]
The number \(\lambda\) is called an eigenvalue of \(A\) associated with \(v.\)
Example 1. \(\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}\begin{bmatrix} 1\\ 0\end{bmatrix} = \begin{bmatrix}3\\ 0\end{bmatrix} = 3\begin{bmatrix}1\\ 0\end{bmatrix}\)
Thus,
- \(\begin{bmatrix}1\\ 0\end{bmatrix}\) is an eigenvector of \(\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}\) and
- \(3\) is an eigenvalue of \(\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}\) associated with \(\begin{bmatrix} 1 \\ 0\end{bmatrix}\)
"Eigen"-things
Example 2. \(\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix} = \begin{bmatrix} -9\\ 6\\ 3\end{bmatrix} = 3\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}\)
Thus,
- \(\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}\) is an eigenvector of \(\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}\)
- \(3\) is an eigenvalue of \(\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}\) with associated eigenvector \(\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}\)
Two new problems
Now, given a matrix \(A\) we have two new problems:
- Find all of the eigenvalues of \(A\).
- Given an eigenvalue \(\lambda\), find all eigenvectors of \(A\) with associated eigenvalue \(\lambda\).
We're going to focus on this one (because it's simpler)
Finding eigenvectors
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax=\lambda x.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}-2x_{1}\\ -2x_{2}\\ -2x_{3}\end{bmatrix} \]
Finding eigenvectors
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax=\lambda I x.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} \]
Finding eigenvectors
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax-\lambda I x=0.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} - \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
Finding eigenvectors
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[(A-\lambda I )x=0.\]
Example. Solve
\[\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
The eigenspace of \(A\) associated with the eigenvalue \(\lambda\) is \(N(A-\lambda I)\)
Finding eigenvectors
Example. Solve
\[\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
Note that
\[\text{rref}\left(\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\right) = \left[\begin{array}{rrr} 1 & 0 & -\frac{3}{2}\\[1ex] 0 & 1 & \frac{1}{2}\\[1ex] 0 & 0 & 0\end{array}\right]\]
Hence, a basis for the eigenspace associated with the eigenvalue \(-2\) is
\[\left\{\begin{bmatrix}\frac{3}{2}\\[1ex] -\frac{1}{2}\\[1ex] 1\end{bmatrix}\right\}\]
Proposition. If \(A\) is an \(n\times n\) matrix, and \(\lambda\) is an eigenvalue of \(A\), then the set
\[W=\{x\in \R^{n} : Ax=\lambda x\} = N(A-\lambda I)\]
is a subspace. This subspace is called the eigenspace of \(A\) associated with \(\lambda\).
Proof. Let \(x,y\) be two vectors in \(W\) and let \(\alpha\in \R\), then we have
\[A(x+y) = Ax+Ay = \lambda x + \lambda y = \lambda(x+y)\]
which shows that \(x+y\in W\). We also have
\[A(\alpha x) = \alpha Ax = \alpha\lambda x = \lambda(\alpha x) \]
which shows that \(\alpha x\in W\). \(\Box\)
Note: The set of eigenvectors associated with \(\lambda\) is not the same as the eigenspace associated with \(\lambda\), since \(0\) is in the latter.
Two new problems
Now, given a matrix \(A\) we have two new problems:
- Find all of the eigenvalues of \(A\).
- Given an eigenvalue \(\lambda\), find all eigenvectors of \(A\) with associated eigenvalue \(\lambda\).
Now we have to work on the harder problem
Finding eigenvalues
Theorem. Let \(A\) be a square matrix. The following are equivalent:
- \(\lambda\) is an eigenvalue of \(A\)
- \((A-\lambda I)x=0\) has a nontrivial solution
- \(N(A-\lambda I)\neq \{0\}\)
- The columns of \(A-\lambda I\) are dependent.
- \(A-\lambda I\) is not invertible.
Example. Assume there is a number \(\lambda\in\R\) such that
\[\begin{bmatrix}1 & -1\\ 1 & 1\end{bmatrix} - \lambda\begin{bmatrix}1 & 0\\ 0 & 1\end{bmatrix} = \begin{bmatrix} 1-\lambda & -1\\ 1 & 1-\lambda\end{bmatrix}\]
has dependent columns. Then there is a constant \(c\) such that
\[c\begin{bmatrix} 1-\lambda\\1\end{bmatrix} = \begin{bmatrix} -1\\ 1-\lambda \end{bmatrix}\]
This means \(c=1-\lambda\), and hence \((1-\lambda)^2=-1\), but this is false.
This matrix has no eigenvalues!
Similar matrices
Definition. Two matrices \(A\) and \(B\) are called similar if there is an invertible matrix \(X\) such that
\[A=XBX^{-1}.\]
(Note that this definition implies that \(A\) and \(B\) are both square and the same size.)
Theorem. If \(A\) and \(B\) are similar, then they have the exact same set of eigenvalues.
Proof. Assume \(A=XBX^{-1}\) and \(\lambda\) is an eigenvalue of \(B\). This means that \(Bx=\lambda x\) for some nonzero \(x\) and some scalar \(\lambda\). Set \(y=Xx\). Since \(X\) in invertible, \(y\) is not the zero vector, and
\[Ay = XBX^{-1}Xx = XBx=X\lambda x = \lambda Xx = \lambda y\]
hence \(\lambda\) is an eigenvalue of \(A\). Since \(B=X^{-1}AX\), the other direction is similar. \(\Box\)
Similar matrices
Example. Consider the diagonal matrix
\[A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},\]
and the invertible matrix
\[X = \begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\quad\text{with inverse }\quad X^{-1} = \begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix}.\]
The matrix \[XAX^{-1}=\begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix}\begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},\]
is similar to \(A\).
Similar matrices
Example continued. The eigenvalues and eigenvectors of \(A\) are obvious. The eigenvectors of \(XAX^{-1}\) are not.
\[XAX^{-1} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},\]
\[A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},\]
Eigenvalue \(\lambda\)
Eigenspace \(N(A-\lambda I)\)
\(\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ 1\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ 0\\ 1\end{bmatrix}\right\}\)
\(1\)
\(-2\)
\(3\)
Eigenvalue \(\lambda\)
Eigenspace \(N(XAX^{-1}-\lambda I)\)
\(\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 2\\ 1\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ -1\\ 1\end{bmatrix}\right\}\)
\(1\)
\(-2\)
\(3\)
Diagonalizable matrices
Definition. A matrix \(A\) is called diagonalizable if it is similar to a diagonal matrix.
Example. Take the diagonal matrix
\[A = \begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\]
Set
\[B = \begin{bmatrix}-3 & -2 & 0 & 4\\ 1 & 2 & -2 & -3\\ -1 & -2 & 3 & 3\\ -3 & -3 & -1 & 5\end{bmatrix}\begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\begin{bmatrix} 1 & -8 & -6 & -2\\ 4 & -14 & -11 & -5\\ 0 & 1 & 1 & 0\\ 3 & -13 & -10 & -4\end{bmatrix}\]
\[= \left[\begin{array}{rrrr}-30 & 132 & 102 & 42\\ 26 & -98 & -76 & -34\\ -26 & 97 & 75 & 34\\ -42 & 175 & 136 & 57\end{array}\right]\]
So, \(B\) is diagonalizable
Linear Algebra Day 20
By John Jasper
Linear Algebra Day 20
- 542
