# The bigger picture

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n

### a peek into the future

m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A
(we are focusing on this nice well-behaved case for now)
(These two are the more interesting cases that we will come to a bit later in the case)

# Recap

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}
x_3 = 1
4x_2 -4(1) = -4
x_2 = 0
x_1 + 2(0) - (1) = 1
x_1 = 2

### pivots

(along the diagonal)

# The good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}
m = n

n

1

# The not-so-good cases

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 2 + equation 1)
(equation 3 - 2*equation 1)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 3 + 3/4*equation 2)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~1~
~3~
~3~
~0~

# The not-so-good cases

\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~0~
0 = 1
4x_2 -4x_3 = -4
x_1 + 2x_2 - x_1 = 1

### (switch to geogebra)

Notes: plot these 3 equations

# The not-so-good cases

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 2 + equation 1)
(equation 3 - 2*equation 1)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 3 + 3/4*equation 2)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~1~
~3~
~3~
~0~
~5~
~3~
~3~
~0~

# The not-so-good cases

\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~0~
~0~
0 = 0
4x_2 -4x_3 = -4
x_1 + 2x_2 - x_3 = 1

### (switch to geogebra)

Notes: plot these 3 equations

# The not-so-good cases

### Which are the b's for which we will not have 0 solutions?

\begin{bmatrix} 1\\ -1\\ 2 \end{bmatrix}
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 2\\ 2\\ 1 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} -1\\ -3\\ 2 \end{bmatrix}
x_1
+ x_2
+ x_3
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}

# The not-so-good cases

### Which are the b's for which we will have infinite solutions?

\begin{bmatrix} 1\\ -1\\ 2 \end{bmatrix}
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} -1\\ 1\\ -2 \end{bmatrix}
\begin{bmatrix} 1&-1&-1\\ -1&1&-3\\ 2&-2&2 \end{bmatrix}
\begin{bmatrix} -1\\ -3\\ 2 \end{bmatrix}
x_1
+ x_2
+ x_3
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}

### (those b's which can be expressed as linear combinations of the columns of A in multiple ways)

\mathbf{b}=\begin{bmatrix} -2\\ -6\\ 4 \end{bmatrix}
\mathbf{x}=\begin{bmatrix} 0\\ 0\\ 2 \end{bmatrix}
,\begin{bmatrix} 1\\ 1\\ 2 \end{bmatrix}
\dots,\begin{bmatrix} c\\ c\\ 2 \end{bmatrix}

## Our current focus is on the good case

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n
m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A
n~non\_zero~pivots

# Back to the good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}

### pivots

(along the diagonal)

# Back to the good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}

### How do we represent the above steps as matrix operations?

B =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix}

# Recap

### (Fun with matrix multiplication)

E =\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -2 & 1\\ \end{bmatrix}
EB =
\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ -4 & 0 & -3 & 0\\ \end{bmatrix}
(subtracting 2 times row 2 from row 3)

# Gaussian Elimination as matrix operations

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
A
U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
E_{31}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{32}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)

E_{21}
=

# Gaussian Elimination as matrix operations

\begin{bmatrix} 1&2&-1&~~~\\ -1&2&-3&~~~\\ 2&1&2&~~~ \end{bmatrix}
\begin{bmatrix} 1&2&-1&~~~\\ 0&4&-4&~~~\\ 0&0&1&~~~ \end{bmatrix}
\underbrace{A~\mathbf{b}}_{Augmented~Matrix}
U~\mathbf{c}
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
E_{31}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{32}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)

### Step 3,2: get 0 in position 3,2

E_{21}
=
\begin{matrix} 1\\-5\\6 \end{matrix}
\begin{matrix} 1\\-4\\1 \end{matrix}

# Gaussian Elimination as matrix operations

E_{32}E_{31}E_{21}A = U
(E_{32}E_{31}E_{21})A = U
associativity law
EA = U
(E = E_{32}E_{31}E_{21})

# Gaussian Elimination as matrix operations

\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
= \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
E
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ \frac{-5}{4} & \frac{3}{4} & 1\\ \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
A
U

# LU factorisation

\begin{bmatrix} 1&1\\ 1&2 \end{bmatrix}
A
\begin{bmatrix} 1&0\\ -1&1 \end{bmatrix}
=\begin{bmatrix} 1&1\\ 0&1 \end{bmatrix}
E
U

### We know that an elementary matrix is invertible

(in fact we even know how to compute the inverse!)
EA = U
A = E^{-1}U
\begin{bmatrix} 1&1\\ 1&2 \end{bmatrix} =
\begin{bmatrix} 1&1\\ 0&1 \end{bmatrix}
\begin{bmatrix} 1&0\\ 1&1 \end{bmatrix}

# LU factorisation

E_{32}E_{31}E_{21}A = U
A = (E_{32}E_{31}E_{21})^{-1}U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
\therefore A = E_{21}^{-1}E_{31}^{-1}E_{32}^{-1}U
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}=

### (Proof in HW1)

\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -\frac{3}{4} & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ -1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}

# LU factorisation

E_{32}E_{31}E_{21}A = U
A = (E_{32}E_{31}E_{21})^{-1}U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
\therefore A = E_{21}^{-1}E_{31}^{-1}E_{32}^{-1}U
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}=

### (Proof in HW1)

\begin{bmatrix} 1 & 0 & 0\\ -1 & 1 & 0\\ 2 & -\frac{3}{4} & 1\\ \end{bmatrix}
A
L
U
(Unlike E, the multipliers sit nicely in the right positions in L)
E
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ \frac{-5}{4} & \frac{3}{4} & 1\\ \end{bmatrix}

# LU factorisation

EA = U
A = LU
lower

upper

triangular