Diagonalizing in Matlab and
Positive definite matrices
Example. Let \(A = \left[\begin{array}{rrrr} 5 & 3 & 1 & -1\\ 3 & 5 & -1 & 1\\ 1 & -1 & 5 & 3\\ -1 & 1 & 3 & 5\end{array}\right]\)
Example continued.
The first eigenvector, with eigenvalue \(8\), is \[v_{1} = [0.6093\ \ \ 0.6093\ -0.3588\ -0.3588]^{\top}\] The second eigenvector, with eigenvalue \(8\), is \[v_{2} = [-0.3588\ -0.3588\ -0.6093\ -0.6093]^{\top}\] The third eigenvector, with eigenvalue \(4\) is \[v_{3} = [-0.5\ \ \ 0.5\ -0.5\ \ \ 0.5]^{\top}\] Now, we see \[A-8v_{1}v_{1}^{\top} - 8v_{2}v_{2}^{\top} - 4v_{3}v_{3}^{\top}=\left[\begin{array}{rrrr} 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{array}\right]\]
The vectors \(\{v_{1},v_{2},v_{3}\}\) form an orthonormal basis for \(C(A)\), but in order to build the orthogonal matrix \(Q\) from the spectral theorem, we need an orthonormal basis for \(\R^{4}\). Where do we find the last eigenvector? An ONB for \(N(A)\).
Definition. A square matrix \(A\) is called positive definite if it is symmetric and all of the eigenvalues of \(A\) are positive. A square matrix \(A\) is called positive semidefinite if it is symmetric and all of the eigenvalues of \(A\) are nonnegative.
Examples.
Theorem. If \(A\) is an \(n\times n\) positive (semi)definite matrix and \(Q\) is an \(n\times n\) orthogonal matrix, then \(QAQ^{\top}\) is positive (semi)definite.
Proof. Since \(A\) is symmetric with all its eigenvalues greater than (or equal to) zero, by the Spectral Theorem there is an orthogonal matrix \(R\) and a diagonal matrix \(\Lambda\) with all of the eigenvalues of \(A\) on the diagonal such that \[A = R\Lambda R^{\top}.\] Hence \[QAQ^{\top} = QR\Lambda R^{\top}Q^{\top} = QR\Lambda (QR)^{\top}.\]
Note that \[(QR)(QR)^{\top} = QRR^{\top}Q^{\top} = QIQ^{\top} = QQ^{\top} = I.\]
This shows that \(QR\) is an orthogonal matrix, and hence \(QAQ^{\top}\) is orthogonally diagonalizable. Therefore, by the Spectral Theorem \(QAQ^{\top}\) is symmetric and all of the eigenvalues of \(QAQ^{\top}\) are the diagonal entries in \(\Lambda.\) \(\Box\)
Theorem (The energy test). A symmetric matrix \(A\) is positive definite if and only if \[x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Proof. Assume \(A\) is positive definite. By the Spectral Theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with all positive entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal, such that, \(A=Q\Lambda Q^{\top}.\) Let
\(x\in\R^{n}\setminus\{0\}\), then
\[x^{\top}Ax = x^{\top}Q\Lambda Q^{\top}x = (Q^{\top}x)^{\top}\Lambda (Q^{\top}x).\] Writing \(y=Q^{\top}x\) we have \(x^{\top}Ax = y^{\top}\Lambda y.\) If \(y=[y_{1}\ y_{2}\ \cdots y_{n}]^{\top}\) then we see that
\[y^{\top}\Lambda y = \sum_{i=1}^{n}\lambda_{i}y_{i}^{2}.\]
Since \(x\neq 0\) and \(Q^{\top}\) is invertible, \(y\neq 0\). Hence there is at least one \(i_{0}\in\{1,\ldots,n\}\) such that \(y_{i_{0}}\neq 0.\) From this we deduce that \[y^{\top}\Lambda y\geq \lambda_{i_{0}}y_{i_{0}}^{2}>0.\]
Proof continued. Assume \(x^{\top}Ax>0\) for all \(x\in\R^{n}\setminus\{0\}.\) By the Spectral Theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal, such that, \(A=Q\Lambda Q^{\top}.\) Let \(v_{i}\) denote the \(i\)th column of \(Q\), and let \(e_{i}\) denote the \(i\)th standard basis vector.
\[Av_{i} = Q\Lambda Q^{\top}v_{i} = Q\Lambda e_{i}=Q\lambda_{i}e_{i} = \lambda_{i}Qe_{i} = \lambda_{i}v_{i}.\]
Hence, for each \(i\) we have
\[0<v_{i}^{\top}Av_{i} = \lambda_{i}v_{i}^{\top}v_{i} = \lambda_{i}.\quad\Box\]
Theorem (The energy test). A symmetric matrix \(A\) is positive definite if and only if \[x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Example. Consider the symmetric matrix
\[A = \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix}.\]
The energy in the direction \(x = [x_{1}\ x_{2}]^{\top}\) is given by
\[x^{\top}Ax = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix} \begin{bmatrix}x_{1}\\ x_{2}\end{bmatrix} = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix}2x_{1}-x_{2}\\-x_{1}+ 4x_{2}\end{bmatrix}\]
\[= x_{1}(2x_{1} - x_{2}) + x_{2}(-x_{1}+4x^{2}) = 2x_{1}^{2} - 2x_{1}x_{2}+4x_{2}^{2}\]
\[= 2\left(x_{1}-\frac{1}{2}x_{2}\right)^2 + \frac{7}{2}x_{2}^{2}\]
We can see from this that \(x^{\top}Ax=0\) if and only if \[x_{2}=0\quad\text{ and }\quad x_{1} = \frac{1}{2}x_{2},\] that is \(x=0\). Hence \(A\) is positive definite by the previous theorem.
Theorem. If \(A_{1}\) and \(A_{2}\) are positive definite matrices, then \(A_{1}+A_{2}\) is positive definite.
Proof. Since \(A_{1}+A_{2}\) is obviously symmetric we can use the energy test. If \(x\neq 0,\) then
\[x^{\top}(A_{1}+A_{2})x = x^{\top}A_{1}x + x^{\top}A_{2}x>0.\quad\Box\]
Theorem (The energy test). A symmetric matrix \(A\) is positive semidefinite if and only if \[x^{\top}Ax\geq 0\quad\text{for all }x\in\R^{n}\setminus\{0\}.\]
Example. If \(A\) is positive definite, and \(D\) is a diagonal matrix with all positive entries, then \(A+D\) is positive definite.
Hence, increasing all of the entries on the diagonal of a matrix results in a matrix that is still positive definite.
Theorem. If \(A=B^{\top}B\) for some matrix \(B\), then \(A\) is positive semidefinite. If \(A=B^{\top}B\) for some matrix \(B\) with independent columns, then \(A\) is positive definite.
Proof. Note that for any vector \(x\) we have
\[x^{\top}Ax = x^{\top}B^{\top}Bx = (Bx)^{\top}(Bx) = \|Bx\|^{2}\geq 0.\]
The quantity \(\|Bx\|^{2}\) is only zero if \(Bx=0\). This only occurs if \(x=0\) or the columns of \(B\) are dependent. Hence, if the columns of \(B\) are independent and \(x\neq 0,\) then \(x^{\top}Ax>0.\) \(\Box\)
Theorem. A matrix \(A\) is positive semidefinite if and only if there is a symmetric matrix \(B\) so that \(A=B^{2}\).
Proof. Note that \(B^{2}\) is symmetric and all of its eigenvalues are nonnegative. Hence \(B^{2}\) is positive semidefinite.
Next, assume \(A\) is positive semidefinite. By the Spectral theorem there is an orthogonal matrix \(Q\) and a diagonal matrix \(\Lambda\) with all nonnegative entries \(\lambda_{1},\ldots,\lambda_{n}\) on the diagonal such that \(A = Q\Lambda Q^{\top}.\)
Let \(\sqrt{\Lambda}\) denote the diagonal matrix with \(\sqrt{\lambda_{1}},\ldots,\sqrt{\lambda_{n}}\) on the diagonal. Set
\[B =Q\sqrt{\Lambda}Q^{\top}\]
then \(B\) is symmetric (since it is orthogonally diagonalizable), and
\[B^{2} = Q\sqrt{\Lambda}Q^{\top}Q\sqrt{\Lambda}Q^{\top} = Q\sqrt{\Lambda}\sqrt{\Lambda}Q^{\top} = Q\Lambda Q^{\top} = A.\quad\Box\]