Orthogonal Compliments, Eigenvalues, and Eigenvectors
Definition. Given a set of vectors \(V\subset\R^{n}\) we can define a set of vectors \(V^{\bot}\) called the orthogonal complement of \(V\), as follows
\[V^{\bot} = \{x\in\R^{n} : x\cdot v = 0\text{ for all }v\in V\}\]
Proposition 1. If \(A\) is a matrix, then \(C(A^{\top})^{\bot} = N(A)\).
Proof. Suppose \(x\in N(A)\). Then \(Ax=0\). Let \(\{b_{1},\ldots,b_{m}\}\) be the columns of \(A^{\top}\). The fact that \(Ax=0\) implies that \(b_{i}\cdot x=0\) for \(i=1,\ldots,m\). If \(y\in C(A^{\top})\), then
\[y = \alpha_{1}b_{1} + \alpha_{2}b_{2} + \cdots + \alpha_{m}b_{m}\]
By the linearity of the dot product, we see that \(x\cdot y = 0\). That is, \(x\in C(A^{\top})^{\bot}\). The other direction is left as an exercise. \(\Box\)
Proposition 2. Assume \(V\subset\R^{n}\) is a subspace. For each \(x\in\R^{n}\) there are unique vectors \(v\in V\) and \(w\in V^{\bot}\) such that
\[x=v+w.\]
Proof. Let \(x\in\R^{n}\). Let \(P\) be the orthogonal projection onto \(V\). Note that \(v:=Px\in V\) and \(w:=x-Px\in V^{\bot}\). Clearly
\[x=Px+(x-Px).\]
Assume there are other vectors \(v'\in V\) and \(w'\in V^{\bot}\) such that \(x=v'+w'\). Multiplying by \(P\) on the left we have
\[Px=Pv'+Pw' = v'+Pw'.\]
Since \(w'\in V^{\bot}\) we know that
\[0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').\]
This shows that \(Pw'=0\), that is \(Px=v'\). \(\Box\)
Let \(V\subset\R^{n}\) be a subspace, and assume \(\{v_{1},\ldots,v_{k}\}\) is a basis for \(V\). Using Gram-Schmidt, we can find an onb \(\{e_{1},\ldots,e_{k}\}\) for \(V\).
Set
\[A = \begin{bmatrix} - & e_{1}^{\top} & -\\ - & e_{2}^{\top} & -\\ - & e_{3}^{\top} & -\\ & \vdots & \\ - & e_{k}^{\top} & -\end{bmatrix}\]
Then \(V = C(A^{\top})\), and hence \(V^{\bot}=N(A)\).
We already know how to find a basis for \(N(A)\).
By the rank-nullity theorem \(N(A)\) has dimension \(n-k\).
So, there is a basis \(\{w_{k+1},\ldots,w_{n}\}\) for \(N(A)\).
Using Gram-Schmidt, we obtain an onb \(\{f_{k+1},\ldots,f_{n}\}\) for \(V^{\bot}\).
Example. Consider the matrix
\[A = \left[\begin{array}{rrrr}1 & 1 & 1 & 1\\ -1 & -1 & -1 & -1\\ -1 & -1 & 0 & -2\\ 1 & 1 & 0 & 2 \end{array}\right]\]
We want to find a basis for \(C(A)^{\bot}\).
By the previous theorem \(C(A)^{\bot} = N(A^{\top})\). We already know how to find a basis for \(N(A^{\top}) = N(\text{rref}(A^{\top}))\).
\(\text{rref}(A^{\top}) = \begin{bmatrix} 1 & -1 & 0 & 0\\ 0 & 0 & 1 & -1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\)
\(\left\{\begin{bmatrix} 1\\ 1\\ 0\\ 0\end{bmatrix},\begin{bmatrix} 0\\ 0\\ 1\\ 1\end{bmatrix}\right\}\) is a basis for \(C(A)^{\bot}\)
We can find a basis for \(N(A)^{\bot}\) by observing that
\[N(A)^{\bot} = (C(A^{\top})^{\bot})^{\bot} = C(A^{\top}).\]
Theorem. If \(V\subset \R^{n}\) is a subspace, then \((V^{\bot})^{\bot} = V\).
Theorem. If \(V\subset \R^{n}\) is a subspace, then \((V^{\bot})^{\bot} = V\).
Proof. Let \(x\in (V^{\bot})^{\bot}\). Let \(P\) be the orthogonal projection onto \(V\). We have already shown that \(x-Px\in V^{\bot}\). Let \(y\in V^{\bot}\), then \[y\cdot(x-Px) = y\cdot x - y\cdot(Px).\] By assumption \(y\cdot x=0\), and since \(Px\in V\) we also conclude that
\(y\cdot(Px)=0\), thus \(y\cdot(x-Px)=0\). Since \(y\in V^{\bot}\) was arbitrary, this shows that \(x-Px\in (V^{\bot})^{\bot}\).
Thus \(x-Px\) is in both \(V^{\bot}\) and \((V^{\bot})^{\bot}\). This implies \[(x-Px)\cdot (x-Px)=0,\] and hence \(x-Px=0\). Therefore \(x=Px\in V\). Thus, we have shown that \((V^{\bot})^{\bot} \subset V\).
Let \(x\in V\), and let \(y\in V^{\bot}\) be arbitrary. It is immediate that \(x\cdot y=0\), and since \(y\in V^{\bot}\) was arbitrary, we conclude that \(x\in(V^{\bot})^{\bot}\). Therefore, \(V\subset(V^{\bot})^{\bot}\). \(\Box\)
Proposition. Assume \(V\subset\R^{n}\) is a subspace. For each \(x\in\R^{n}\) there are unique vectors \(v\in V\) and \(w\in V^{\bot}\) such that
\[x=v+w.\]
Proof. Let \(x\in\R^{n}\). Let \(P\) be the orthogonal projection onto \(V\). Note that \(v:=Px\in V\) and \(w:=x-Px\in V^{\bot}\). Clearly
\[x=Px+(x-Px).\]
Assume there are other vectors \(v'\in V\) and \(w'\in V^{\bot}\) such that \(x=v'+w'\). Multiplying by \(P\) on the left we have
\[Px=Pv'+Pw' = v'+Pw'.\]
Since \(w'\in V^{\bot}\) we know that
\[0 = (w')^{\top}(Pw')=w'^{\top}Pw' = w'^{\top}PPw'=w'^{\top}P^{\top}Pw' = (Pw')^{\top}(Pw').\]
This shows that \(Pw'=0\), that is \(Px=v'\). \(\Box\)
Definition. Let \(A\) be an \(n\times n\) matrix. A nonzero vector \(v\in\R^{n}\) is called an eigenvector of \(A\) if there is a scalar \(\lambda\) such that
\[Av=\lambda v\]
The number \(\lambda\) is called an eigenvalue of \(A\) associated with \(v.\)
Example 1. \(\begin{bmatrix} 3 & 2\\ 0 & 1\end{bmatrix}\begin{bmatrix} 1\\ 0\end{bmatrix} = \begin{bmatrix}3\\ 0\end{bmatrix} = 3\begin{bmatrix}1\\ 0\end{bmatrix}\)
Thus,
Example 2. \(\begin{bmatrix} 2 & -1 & -1\\ -1 & 2 & -1\\ -1 & -1 & 2\end{bmatrix}\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix} = \begin{bmatrix} -9\\ 6\\ 3\end{bmatrix} = 3\begin{bmatrix} -3\\ 2\\ 1\end{bmatrix}\)
Thus,
Now, given a matrix \(A\) we have two new problems:
We're going to focus on this one (because it's simpler)
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax=\lambda x.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}-2x_{1}\\ -2x_{2}\\ -2x_{3}\end{bmatrix} \]
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax=\lambda I x.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} \]
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[Ax-\lambda I x=0.\]
Example. Solve
\[\left[\begin{array}{rrr} -2 & 2 & 1\\ 2 & -2 & -3\\ 1 & 3 & -2\end{array}\right]\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} - \begin{bmatrix} -2 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & -2\end{bmatrix}\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
Given an \(n\times n\) matrix \(A\), it is difficult to find the eigenvalues of \(A\) (if it even has any!) However, if we are told that \(\lambda\) is an eigenvalue of \(A\), then we simply need to solve
\[(A-\lambda I )x=0.\]
Example. Solve
\[\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
The eigenspace of \(A\) associated with the eigenvalue \(\lambda\) is \(N(A-\lambda I)\)
Example. Solve
\[\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\begin{bmatrix}x_{1}\\ x_{2}\\ x_{3}\end{bmatrix} = \begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]
Note that
\[\text{rref}\left(\left[\begin{array}{rrr} 0 & 2 & 1\\ 2 & 0 & -3\\ 1 & 3 & 0\end{array}\right]\right) = \left[\begin{array}{rrr} 1 & 0 & -\frac{3}{2}\\[1ex] 0 & 1 & \frac{1}{2}\\[1ex] 0 & 0 & 0\end{array}\right]\]
Hence, a basis for the eigenspace associated with the eigenvalue \(-2\) is
\[\left\{\begin{bmatrix}\frac{3}{2}\\[1ex] -\frac{1}{2}\\[1ex] 1\end{bmatrix}\right\}\]
Proposition. If \(A\) is an \(n\times n\) matrix, and \(\lambda\) is an eigenvalue of \(A\), then the set
\[W=\{x\in \R^{n} : Ax=\lambda x\} = N(A-\lambda I)\]
is a subspace. This subspace is called the eigenspace of \(A\) associated with \(\lambda\).
Proof. Let \(x,y\) be two vectors in \(W\) and let \(\alpha\in \R\), then we have
\[A(x+y) = Ax+Ay = \lambda x + \lambda y = \lambda(x+y)\]
which shows that \(x+y\in W\). We also have
\[A(\alpha x) = \alpha Ax = \alpha\lambda x = \lambda(\alpha x) \]
which shows that \(\alpha x\in W\). \(\Box\)
Note: The set of eigenvectors associated with \(\lambda\) is not the same as the eigenspace associated with \(\lambda\), since \(0\) is in the latter.
Now, given a matrix \(A\) we have two new problems:
Now we have to work on the harder problem
Theorem. Let \(A\) be a square matrix. The following are equivalent:
Example. Assume there is a number \(\lambda\in\R\) such that
\[\begin{bmatrix}1 & -1\\ 1 & 1\end{bmatrix} - \lambda\begin{bmatrix}1 & 0\\ 0 & 1\end{bmatrix} = \begin{bmatrix} 1-\lambda & -1\\ 1 & 1-\lambda\end{bmatrix}\]
has dependent columns. Then there is a constant \(c\) such that
\[c\begin{bmatrix} 1-\lambda\\1\end{bmatrix} = \begin{bmatrix} -1\\ 1-\lambda \end{bmatrix}\]
This means \(c=1-\lambda\), and hence \((1-\lambda)^2=-1\), but this is false.
This matrix has no eigenvalues!
Definition. Two matrices \(A\) and \(B\) are called similar if there is an invertible matrix \(X\) such that
\[A=XBX^{-1}.\]
(Note that this definition implies that \(A\) and \(B\) are both square and the same size.)
Theorem. If \(A\) and \(B\) are similar, then they have the exact same set of eigenvalues.
Proof. Assume \(A=XBX^{-1}\) and \(\lambda\) is an eigenvalue of \(B\). This means that \(Bx=\lambda x\) for some nonzero \(x\) and some scalar \(\lambda\). Set \(y=Xx\). Since \(X\) in invertible, \(y\) is not the zero vector, and
\[Ay = XBX^{-1}Xx = XBx=X\lambda x = \lambda Xx = \lambda y\]
hence \(\lambda\) is an eigenvalue of \(A\). Since \(B=X^{-1}AX\), the other direction is similar. \(\Box\)
Example. Consider the diagonal matrix
\[A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},\]
and the invertible matrix
\[X = \begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\quad\text{with inverse }\quad X^{-1} = \begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix}.\]
The matrix \[XAX^{-1}=\begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & -1\\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix}\begin{bmatrix} 1 & -2 & -2\\ 0 & 1 & 1\\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},\]
is similar to \(A\).
Example continued. The eigenvalues and eigenvectors of \(A\) are obvious. The eigenvectors of \(XAX^{-1}\) are not.
\[XAX^{-1} = \begin{bmatrix} 1 & -6 & -6\\ 0 & -2 & -5\\ 0 & 0 & 3 \end{bmatrix},\]
\[A = \begin{bmatrix} 1 & 0 & 0\\ 0 & -2 & 0\\ 0 & 0 & 3\end{bmatrix},\]
Eigenvalue \(\lambda\)
Eigenspace \(N(A-\lambda I)\)
\(\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ 1\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ 0\\ 1\end{bmatrix}\right\}\)
\(1\)
\(-2\)
\(3\)
Eigenvalue \(\lambda\)
Eigenspace \(N(XAX^{-1}-\lambda I)\)
\(\text{span}\left\{\begin{bmatrix} 1\\ 0\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 2\\ 1\\ 0\end{bmatrix}\right\}\)
\(\text{span}\left\{\begin{bmatrix} 0\\ -1\\ 1\end{bmatrix}\right\}\)
\(1\)
\(-2\)
\(3\)
Definition. A matrix \(A\) is called diagonalizable if it is similar to a diagonal matrix.
Example. Take the diagonal matrix
\[A = \begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\]
Set
\[B = \begin{bmatrix}-3 & -2 & 0 & 4\\ 1 & 2 & -2 & -3\\ -1 & -2 & 3 & 3\\ -3 & -3 & -1 & 5\end{bmatrix}\begin{bmatrix} 2 & 0 & 0 & 0\\ 0 & 3 & 0 & 0\\ 0 & 0 & -1 & 0\\ 0 & 0 & 0 & 0\end{bmatrix}\begin{bmatrix} 1 & -8 & -6 & -2\\ 4 & -14 & -11 & -5\\ 0 & 1 & 1 & 0\\ 3 & -13 & -10 & -4\end{bmatrix}\]
\[= \left[\begin{array}{rrrr}-30 & 132 & 102 & 42\\ 26 & -98 & -76 & -34\\ -26 & 97 & 75 & 34\\ -42 & 175 & 136 & 57\end{array}\right]\]
So, \(B\) is diagonalizable