# Day 25:

Diagonalizing in Matlab and

Positive definite matrices

Example. Let $$A = \left[\begin{array}{rrrr} 5 & 3 & 1 & -1\\ 3 & 5 & -1 & 1\\ 1 & -1 & 5 & 3\\ -1 & 1 & 3 & 5\end{array}\right]$$

• Start with a random vector $$v = \text{randn}(4,1)$$. For example, Matlab just gave me $v = [0.5377\ \ \ 1.8339\ -2.2588\ \ \ 0.8622]^\top$
• Normalize $$v$$: Set $$v=v/\text{norm}(v)$$. Then, $v=[0.1745\ \ \ 0.5951\ -0.7329\ \ \ 0.2798]^{\top}$
• Now, replace $$v$$ with $$Av/\|Av\|$$ over and over until $$v$$ is very close to being an eigenvector. How do you check that $$v$$ is close to being an eigenvector? Check that $$\|Av-\lambda v\|$$ is small. What is $$\lambda$$? Check out the Power Method.
• Replace $$A$$ with $$A-\lambda vv^{\top}$$. If this is the zero matrix, then you're done! Otherwise, repeat these steps.
• There is one last problem...

### Diagonalizing in Matlab

Example continued.

The first eigenvector, with eigenvalue $$8$$, is $v_{1} = [0.6093\ \ \ 0.6093\ -0.3588\ -0.3588]^{\top}$ The second eigenvector, with eigenvalue $$8$$, is $v_{2} = [-0.3588\ -0.3588\ -0.6093\ -0.6093]^{\top}$ The third eigenvector, with eigenvalue $$4$$ is $v_{3} = [-0.5\ \ \ 0.5\ -0.5\ \ \ 0.5]^{\top}$ Now, we see $A-8v_{1}v_{1}^{\top} - 8v_{2}v_{2}^{\top} - 4v_{3}v_{3}^{\top}=\left[\begin{array}{rrrr} 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{array}\right]$

### Diagonalizing in Matlab

The vectors $$\{v_{1},v_{2},v_{3}\}$$ form an orthonormal basis for $$C(A)$$, but in order to build the orthogonal matrix $$Q$$ from the spectral theorem, we need an orthonormal basis for $$\R^{4}$$. Where do we find the last eigenvector? An ONB for $$N(A)$$.

Definition. A square matrix $$A$$ is called positive definite if it is symmetric and all of the eigenvalues of $$A$$ are positive. A square matrix $$A$$ is called positive semidefinite if it is symmetric and all of the eigenvalues of $$A$$ are nonnegative.

Examples.

• $$\begin{bmatrix} 1 & 0\\ 0 & 3\end{bmatrix}$$ is positive definite.
• $$\begin{bmatrix} 1 & -1\\ 1 & 1\end{bmatrix}\begin{bmatrix} 4 & 0\\ 0 & 1\end{bmatrix} \begin{bmatrix} 1 & 1\\ -1 & 1\end{bmatrix}$$ is positive definite.
• Any diagonal matrix with positive diagonal entries is positive definite
• Any diagonal matrix with nonnegative diagonal entries is positive semidefinite.
• If $$A$$ is positive definite, and $$Q$$ is orthogonal, then $$QAQ^{\top}$$ is positive definite.
• If $$A$$ is positive semidefinite, and $$Q$$ is orthogonal, then $$QAQ^{\top}$$ is semidefinite.

Theorem. If $$A$$ is an $$n\times n$$ positive (semi)definite matrix and $$Q$$ is an $$n\times n$$ orthogonal matrix, then $$QAQ^{\top}$$ is positive (semi)definite.

Proof. Since $$A$$ is symmetric with all its eigenvalues greater than (or equal to) zero, by the Spectral Theorem there is an orthogonal matrix $$R$$ and a diagonal matrix $$\Lambda$$ with all of the eigenvalues of $$A$$ on the diagonal such that $A = R\Lambda R^{\top}.$ Hence $QAQ^{\top} = QR\Lambda R^{\top}Q^{\top} = QR\Lambda (QR)^{\top}.$

Note that $(QR)(QR)^{\top} = QRR^{\top}Q^{\top} = QIQ^{\top} = QQ^{\top} = I.$

This shows that $$QR$$ is an orthogonal matrix, and hence $$QAQ^{\top}$$ is orthogonally diagonalizable. Therefore, by the Spectral Theorem $$QAQ^{\top}$$ is symmetric and all of the eigenvalues of $$QAQ^{\top}$$ are the diagonal entries in $$\Lambda.$$  $$\Box$$

Theorem (The energy test). A symmetric matrix $$A$$ is positive definite if and only if $x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.$

Proof. Assume $$A$$ is positive definite. By the Spectral Theorem there is an orthogonal matrix $$Q$$ and a diagonal matrix $$\Lambda$$ with all positive entries $$\lambda_{1},\ldots,\lambda_{n}$$ on the diagonal, such that, $$A=Q\Lambda Q^{\top}.$$ Let

$$x\in\R^{n}\setminus\{0\}$$, then

$x^{\top}Ax = x^{\top}Q\Lambda Q^{\top}x = (Q^{\top}x)^{\top}\Lambda (Q^{\top}x).$ Writing $$y=Q^{\top}x$$ we have $$x^{\top}Ax = y^{\top}\Lambda y.$$ If $$y=[y_{1}\ y_{2}\ \cdots y_{n}]^{\top}$$ then we see that

$y^{\top}\Lambda y = \sum_{i=1}^{n}\lambda_{i}y_{i}^{2}.$

Since $$x\neq 0$$ and $$Q^{\top}$$ is invertible, $$y\neq 0$$. Hence there is at least one $$i_{0}\in\{1,\ldots,n\}$$ such that $$y_{i_{0}}\neq 0.$$ From this we deduce that $y^{\top}\Lambda y\geq \lambda_{i_{0}}y_{i_{0}}^{2}>0.$

Proof continued. Assume $$x^{\top}Ax>0$$ for all $$x\in\R^{n}\setminus\{0\}.$$ By the Spectral Theorem there is an orthogonal matrix $$Q$$ and a diagonal matrix $$\Lambda$$ with entries $$\lambda_{1},\ldots,\lambda_{n}$$ on the diagonal, such that, $$A=Q\Lambda Q^{\top}.$$ Let $$v_{i}$$ denote the $$i$$th column of $$Q$$, and let $$e_{i}$$ denote the $$i$$th standard basis vector.

$Av_{i} = Q\Lambda Q^{\top}v_{i} = Q\Lambda e_{i}=Q\lambda_{i}e_{i} = \lambda_{i}Qe_{i} = \lambda_{i}v_{i}.$

Hence, for each $$i$$ we have

$0<v_{i}^{\top}Av_{i} = \lambda_{i}v_{i}^{\top}v_{i} = \lambda_{i}.\quad\Box$

Theorem (The energy test). A symmetric matrix $$A$$ is positive definite if and only if $x^{\top}Ax>0\quad\text{for all }x\in\R^{n}\setminus\{0\}.$

Example. Consider the symmetric matrix

$A = \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix}.$

The energy in the direction $$x = [x_{1}\ x_{2}]^{\top}$$ is given by

$x^{\top}Ax = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix} 2 & -1\\ -1 & 4\end{bmatrix} \begin{bmatrix}x_{1}\\ x_{2}\end{bmatrix} = \begin{bmatrix}x_{1} & x_{2}\end{bmatrix} \begin{bmatrix}2x_{1}-x_{2}\\-x_{1}+ 4x_{2}\end{bmatrix}$

$= x_{1}(2x_{1} - x_{2}) + x_{2}(-x_{1}+4x^{2}) = 2x_{1}^{2} - 2x_{1}x_{2}+4x_{2}^{2}$

$= 2\left(x_{1}-\frac{1}{2}x_{2}\right)^2 + \frac{7}{2}x_{2}^{2}$

We can see from this that $$x^{\top}Ax=0$$ if and only if $x_{2}=0\quad\text{ and }\quad x_{1} = \frac{1}{2}x_{2},$ that is $$x=0$$. Hence $$A$$ is positive definite by the previous theorem.

Theorem. If $$A_{1}$$ and $$A_{2}$$ are positive definite matrices, then $$A_{1}+A_{2}$$ is positive definite.

Proof. Since $$A_{1}+A_{2}$$ is obviously symmetric we can use the energy test. If $$x\neq 0,$$ then

$x^{\top}(A_{1}+A_{2})x = x^{\top}A_{1}x + x^{\top}A_{2}x>0.\quad\Box$

Theorem (The energy test). A symmetric matrix $$A$$ is positive semidefinite if and only if $x^{\top}Ax\geq 0\quad\text{for all }x\in\R^{n}\setminus\{0\}.$

Example. If $$A$$ is positive definite, and $$D$$ is a diagonal matrix with all positive entries, then $$A+D$$ is positive definite.

Hence, increasing all of the entries on the diagonal of a matrix results in a matrix that is still positive definite.

Theorem. If $$A=B^{\top}B$$ for some matrix $$B$$, then $$A$$ is positive semidefinite. If $$A=B^{\top}B$$ for some matrix $$B$$ with independent columns, then $$A$$ is positive definite.

Proof. Note that for any vector $$x$$ we have

$x^{\top}Ax = x^{\top}B^{\top}Bx = (Bx)^{\top}(Bx) = \|Bx\|^{2}\geq 0.$

The quantity $$\|Bx\|^{2}$$ is only zero if $$Bx=0$$. This only occurs if $$x=0$$ or the columns of $$B$$ are dependent. Hence, if the columns of $$B$$ are independent and $$x\neq 0,$$ then $$x^{\top}Ax>0.$$ $$\Box$$

Theorem. A matrix $$A$$ is positive semidefinite if and only if there is a symmetric matrix $$B$$ so that $$A=B^{2}$$.

Proof. Note that $$B^{2}$$ is symmetric and all of its eigenvalues are nonnegative. Hence $$B^{2}$$ is positive semidefinite.

Next, assume $$A$$ is positive semidefinite. By the Spectral theorem there is an orthogonal matrix $$Q$$ and a diagonal matrix $$\Lambda$$ with all nonnegative entries $$\lambda_{1},\ldots,\lambda_{n}$$ on the diagonal such that $$A = Q\Lambda Q^{\top}.$$

Let $$\sqrt{\Lambda}$$ denote the diagonal matrix with $$\sqrt{\lambda_{1}},\ldots,\sqrt{\lambda_{n}}$$ on the diagonal. Set

$B =Q\sqrt{\Lambda}Q^{\top}$

then $$B$$ is symmetric (since it is orthogonally diagonalizable), and

$B^{2} = Q\sqrt{\Lambda}Q^{\top}Q\sqrt{\Lambda}Q^{\top} = Q\sqrt{\Lambda}\sqrt{\Lambda}Q^{\top} = Q\Lambda Q^{\top} = A.\quad\Box$

By John Jasper

• 472