Day 25:
Diagonalizing in Matlab and
Positive definite matrices
Example. Let \(A = \left[\begin{array}{rrrr} 5 & 3 & 1 & -1\\ 3 & 5 & -1 & 1\\ 1 & -1 & 5 & 3\\ -1 & 1 & 3 & 5\end{array}\right]\)
- Start with a random vector \(v = \text{randn}(4,1)\). For example, Matlab just gave me \[v = [0.5377\ \ \ 1.8339\ -2.2588\ \ \ 0.8622]^\top\]
- Normalize \(v\): Set \(v=v/\text{norm}(v)\). Then, \[v=[0.1745\ \ \ 0.5951\ -0.7329\ \ \ 0.2798]^{\top}\]
- Now, replace \(v\) with \(Av/\|Av\|\) over and over until \(v\) is very close to being an eigenvector. How do you check that \(v\) is close to being an eigenvector? Check that \(\|Av-\lambda v\|\) is small. What is \(\lambda\)? Check out the Power Method.
- Replace \(A\) with \(A-\lambda vv^{\top}\). If this is the zero matrix, then you're done! Otherwise, repeat these steps.
- There is one last problem...
Diagonalizing in Matlab
Example continued.
The first eigenvector, with eigenvalue \(8\), is \[v_{1} = [0.6093\ \ \ 0.6093\ -0.3588\ -0.3588]^{\top}\] The second eigenvector, with eigenvalue \(8\), is \[v_{2} = [-0.3588\ -0.3588\ -0.6093\ -0.6093]^{\top}\] The third eigenvector, with eigenvalue \(4\) is \[v_{3} = [-0.5\ \ \ 0.5\ -0.5\ \ \ 0.5]^{\top}\] Now, we see \[A-8v_{1}v_{1}^{\top} - 8v_{2}v_{2}^{\top} - 4v_{3}v_{3}^{\top}=\left[\begin{array}{rrrr} 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{array}\right]\]
Diagonalizing in Matlab
The vectors \(\{v_{1},v_{2},v_{3}\}\) form an orthonormal basis for \(C(A)\), but in order to build the orthogonal matrix \(Q\) from the spectral theorem, we need an orthonormal basis for \(\R^{4}\). Where do we find the last eigenvector? An ONB for \(N(A)\).
Definition. A square matrix \(A\) is called positive definite if it is symmetric and all of the eigenvalues of \(A\) are positive. A square matrix \(A\) is called positive semidefinite if it is symmetric and all of the eigenvalues of \(A\) are nonnegative.
Examples.
- \(\begin{bmatrix} 1 & 0\\ 0 & 3\end{bmatrix}\) is positive definite.
- \(\begin{bmatrix} 1 & -1\\ 1 & 1\end{bmatrix}\begin{bmatrix} 4 & 0\\ 0 & 1\end{bmatrix} \begin{bmatrix} 1 & 1\\ -1 & 1\end{bmatrix}\) is positive definite.
- Any diagonal matrix with positive diagonal entries is positive definite
- Any diagonal matrix with nonnegative diagonal entries is positive semidefinite.
- If \(A\) is positive definite, and \(Q\) is orthogonal, then \(QAQ^{\top}\) is positive definite.
- If \(A\) is positive semidefinite, and \(Q\) is orthogonal, then \(QAQ^{\top}\) is semidefinite.
Theorem. If \(A\) is an \(n\times n\) positive (semi)definite matrix and \(Q\) is an \(n\times n\) orthogonal matrix, then \(QAQ^{\top}\) is positive (semi)definite.
Proof. Since \(A\) is symmetric with all its eigenvalues greater than (or equal to) zero, by the Spectral Theorem there is an orthogonal matrix \(R\) and a diagonal matrix \(\Lambda\) with all of the eigenvalues of \(A\) on the diagonal such that \[A = R\Lambda R^{\top}.\] Hence \[QAQ^{\top} = QR\Lambda R^{\top}Q^{\top} = QR\Lambda (QR)^{\top}.\]
Note that \[(QR)(QR)^{\top} = QRR^{\top}Q^{\top} = QIQ^{\top} = QQ^{\top} = I.\]
This shows that \(QR\) is an orthogonal matrix, and hence \(QAQ^{\top}\) is orthogonally diagonalizable. Therefore, by the Spectral Theorem \(QAQ^{\top}\) is symmetric and all of the eigenvalues of \(QAQ^{\top}\) are the diagonal entries in \(\Lambda.\) \(\Box\)
Theorem (The energy test). A symmetric matrix \(A\) is positive definite if and only if \[x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Proof. Assume \(A\) is positive definite. By the Spectral Theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with all positive entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal, such that, \(A=Q\Lambda Q^{\top}.\) Let
\(x\in\R^{n}\setminus\{0\}\), then
\[x^{\top}Ax = x^{\top}Q\Lambda Q^{\top}x = (Q^{\top}x)^{\top}\Lambda (Q^{\top}x).\] Writing \(y=Q^{\top}x\) we have \(x^{\top}Ax = y^{\top}\Lambda y.\) If \(y=[y_{1}\ y_{2}\ \cdots y_{n}]^{\top}\) then we see that
\[y^{\top}\Lambda y = \sum_{i=1}^{n}\lambda_{i}y_{i}^{2}.\]
Since \(x\neq 0\) and \(Q^{\top}\) is invertible, \(y\neq 0\). Hence there is at least one \(i_{0}\in\{1,\ldots,n\}\) such that \(y_{i_{0}}\neq 0.\) From this we deduce that \[y^{\top}\Lambda y\geq \lambda_{i_{0}}y_{i_{0}}^{2}>0.\]
Proof continued. Assume \(x^{\top}Ax>0\) for all \(x\in\R^{n}\setminus\{0\}.\) By the Spectral Theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal, such that, \(A=Q\Lambda Q^{\top}.\) Let \(v_{i}\) denote the \(i\)th column of \(Q\), and let \(e_{i}\) denote the \(i\)th standard basis vector.
\[Av_{i} = Q\Lambda Q^{\top}v_{i} = Q\Lambda e_{i}=Q\lambda_{i}e_{i} = \lambda_{i}Qe_{i} = \lambda_{i}v_{i}.\]
Hence, for each \(i\) we have
\[0<v_{i}^{\top}Av_{i} = \lambda_{i}v_{i}^{\top}v_{i} = \lambda_{i}.\quad\Box\]
Theorem (The energy test). A symmetric matrix \(A\) is positive definite if and only if \[x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Example. Consider the symmetric matrix
\[A = \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix}.\]
The energy in the direction \(x = [x_{1}\ x_{2}]^{\top}\) is given by
\[x^{\top}Ax = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix} \begin{bmatrix}x_{1}\\ x_{2}\end{bmatrix} = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix}2x_{1}-x_{2}\\-x_{1}+ 4x_{2}\end{bmatrix}\]
\[= x_{1}(2x_{1} - x_{2}) + x_{2}(-x_{1}+4x^{2}) = 2x_{1}^{2} - 2x_{1}x_{2}+4x_{2}^{2}\]
\[= 2\left(x_{1}-\frac{1}{2}x_{2}\right)^2 + \frac{7}{2}x_{2}^{2}\]
We can see from this that \(x^{\top}Ax=0\) if and only if \[x_{2}=0\quad\text{ and }\quad x_{1} = \frac{1}{2}x_{2},\] that is \(x=0\). Hence \(A\) is positive definite by the previous theorem.
Theorem. If \(A_{1}\) and \(A_{2}\) are positive definite matrices, then \(A_{1}+A_{2}\) is positive definite.
Proof. Since \(A_{1}+A_{2}\) is obviously symmetric we can use the energy test. If \(x\neq 0,\) then
\[x^{\top}(A_{1}+A_{2})x = x^{\top}A_{1}x + x^{\top}A_{2}x>0.\quad\Box\]
Theorem (The energy test). A symmetric matrix \(A\) is positive semidefinite if and only if \[x^{\top}Ax\geq 0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Example. If \(A\) is positive definite, and \(D\) is a diagonal matrix with all positive entries, then \(A+D\) is positive definite.
Hence, increasing all of the entries on the diagonal of a matrix results in a matrix that is still positive definite.
Theorem. If \(A=B^{\top}B\) for some matrix \(B\), then \(A\) is positive semidefinite. If \(A=B^{\top}B\) for some matrix \(B\) with independent columns, then \(A\) is positive definite.
Proof. Note that for any vector \(x\) we have
\[x^{\top}Ax = x^{\top}B^{\top}Bx = (Bx)^{\top}(Bx) = \|Bx\|^{2}\geq 0.\]
The quantity \(\|Bx\|^{2}\) is only zero if \(Bx=0\). This only occurs if \(x=0\) or the columns of \(B\) are dependent. Hence, if the columns of \(B\) are independent and \(x\neq 0,\) then \(x^{\top}Ax>0.\) \(\Box\)
Theorem. A matrix \(A\) is positive semidefinite if and only if there is a symmetric matrix \(B\) so that \(A=B^{2}\).
Proof. Note that \(B^{2}\) is symmetric and all of its eigenvalues are nonnegative. Hence \(B^{2}\) is positive semidefinite.
Next, assume \(A\) is positive semidefinite. By the Spectral theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with all nonnegative entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal such that \(A = Q\Lambda Q^{\top}.\)
Let \(\sqrt{\Lambda}\) denote the diagonal matrix with \(\sqrt{\lambda_{1}},\ldots,\sqrt{\lambda_{n}}\) on the diagonal. Set
\[B =Q\sqrt{\Lambda}Q^{\top}\]
then \(B\) is symmetric (since it is orthogonally diagonalizable), and
\[B^{2} = Q\sqrt{\Lambda}Q^{\top}Q\sqrt{\Lambda}Q^{\top} = Q\sqrt{\Lambda}\sqrt{\Lambda}Q^{\top} = Q\Lambda Q^{\top} = A.\quad\Box\]
Linear Algebra Day 25
By John Jasper
Linear Algebra Day 25
- 549
