CS6015: Linear Algebra and Random Processes

Lecture 4: Gaussian Elimination, LU factorisation

Learning Objectives

Why did elimination work so well in the last lecture?

What is LU factorisation?

How to represent Gauss Elimination as matrix multiplication?

What is the intuition behind 0 and infinite solutions?

(for today's lecture)

The bigger picture

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n

a peek into the future

m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A
(we are focusing on this nice well-behaved case for now)
(These two are the more interesting cases that we will come to a bit later in the course)

Recap

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}
x_3 = 1
4x_2 -4(1) = -4
x_2 = 0
x_1 + 2(0) - (1) = 1
x_1 = 2

back-substitution

Gaussian Elimination

pivots

(along the diagonal)

The good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}
m = n

Gaussian Elimination

non-zero pivots

n

unique solution

1

The not-so-good cases

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 2 + equation 1)
(equation 3 - 2*equation 1)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 3 + 3/4*equation 2)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~1~
~3~
~3~
~0~

The not-so-good cases

\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~0~
0 = 1
4x_2 -4x_3 = -4
x_1 + 2x_2 - x_1 = 1

We have a zero pivot

The last equation cannot be satisfied

0 solutions

(switch to geogebra)

Notes: plot these 3 equations

The not-so-good cases

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 2 + equation 1)
(equation 3 - 2*equation 1)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&-3&4 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 4 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(equation 3 + 3/4*equation 2)
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~1~
~3~
~3~
~0~
~5~
~3~
~3~
~0~

The not-so-good cases

\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
~0~
~0~
0 = 0
4x_2 -4x_3 = -4
x_1 + 2x_2 - x_3 = 1

We have a zero pivot

only two equations need to be satisfied

infinite solutions (intersection of two planes)

(switch to geogebra)

Notes: plot these 3 equations

The not-so-good cases

Will we have a solution for every b?

(some more intuition)

or

Which are the b's for which we will not have 0 solutions?

\begin{bmatrix} 1\\ -1\\ 2 \end{bmatrix}
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 2\\ 2\\ 1 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} -1\\ -3\\ 2 \end{bmatrix}
x_1
+ x_2
+ x_3
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}

(those b's which are linear combinations of the columns of A)

The not-so-good cases

(some more intuition)

Which are the b's for which we will have infinite solutions?

\begin{bmatrix} 1\\ -1\\ 2 \end{bmatrix}
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} -1\\ 1\\ -2 \end{bmatrix}
\begin{bmatrix} 1&-1&-1\\ -1&1&-3\\ 2&-2&2 \end{bmatrix}
\begin{bmatrix} -1\\ -3\\ 2 \end{bmatrix}
x_1
+ x_2
+ x_3
=\begin{bmatrix} b_1\\ b_2\\ b_3 \end{bmatrix}

(those b's which can be expressed as linear combinations of the columns of A in multiple ways)

\mathbf{b}=\begin{bmatrix} -2\\ -6\\ 4 \end{bmatrix}
\mathbf{x}=\begin{bmatrix} 0\\ 0\\ 2 \end{bmatrix}
,\begin{bmatrix} 1\\ 1\\ 2 \end{bmatrix}
\dots,\begin{bmatrix} c\\ c\\ 2 \end{bmatrix}

Our current focus is on the good case

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n
m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A
n~non\_zero~pivots

Back to the good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}

Gaussian Elimination

pivots

(along the diagonal)

Step 2,1: get 0 in position 2,1

Step 3,1: get 0 in position 3,1

Step 3,2: get 0 in position 3,2

Back to the good case

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1\\ -5\\ 6 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
=\begin{bmatrix} 1\\ -4\\ 1 \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)
A
\mathbf{x}
\mathbf{b}
U
\mathbf{x}
\mathbf{c}

Gaussian Elimination

Step 2,1: get 0 in position 2,1

Step 3,1: get 0 in position 3,1

Step 3,2: get 0 in position 3,2

How do we represent the above steps as matrix operations?

B =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix}

Recap

(Fun with matrix multiplication)

E =\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -2 & 1\\ \end{bmatrix}
EB =
\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ -4 & 0 & -3 & 0\\ \end{bmatrix} 
(subtracting 2 times row 2 from row 3)

Gaussian Elimination as matrix operations

\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
A
U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
E_{31}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{32}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)

Step 2,1: get 0 in position 2,1

Step 3,1: get 0 in position 3,1

Step 3,2: get 0 in position 3,2

E_{21}
=

Gaussian Elimination as matrix operations

\begin{bmatrix} 1&2&-1&~~~\\ -1&2&-3&~~~\\ 2&1&2&~~~ \end{bmatrix}
\begin{bmatrix} 1&2&-1&~~~\\ 0&4&-4&~~~\\ 0&0&1&~~~ \end{bmatrix}
\underbrace{A~\mathbf{b}}_{Augmented~Matrix}
U~\mathbf{c}
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
E_{31}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{32}
(row 2 + row 1)
(row 3 - 2*row 1)
(row 3 + 3/4*row 2)

Step 2,1: get 0 in position 2,1

Step 3,1: get 0 in position 3,1

Step 3,2: get 0 in position 3,2

E_{21}
=
\begin{matrix} 1\\-5\\6 \end{matrix}
\begin{matrix} 1\\-4\\1 \end{matrix}

Gaussian Elimination as matrix operations

E_{32}E_{31}E_{21}A = U
(E_{32}E_{31}E_{21})A = U
associativity law
EA = U
(E = E_{32}E_{31}E_{21})

Gaussian Elimination as matrix operations

\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
= \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
E
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ \frac{-5}{4} & \frac{3}{4} & 1\\ \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
=\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
A
U

LU factorisation

\begin{bmatrix} 1&1\\ 1&2 \end{bmatrix}
A
\begin{bmatrix} 1&0\\ -1&1 \end{bmatrix}
=\begin{bmatrix} 1&1\\ 0&1 \end{bmatrix}
E
U

We know that an elementary matrix is invertible

(in fact we even know how to compute the inverse!)
EA = U
A = E^{-1}U
\begin{bmatrix} 1&1\\ 1&2 \end{bmatrix} =
\begin{bmatrix} 1&1\\ 0&1 \end{bmatrix}
\begin{bmatrix} 1&0\\ 1&1 \end{bmatrix}

LU factorisation

E_{32}E_{31}E_{21}A = U
A = (E_{32}E_{31}E_{21})^{-1}U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
\therefore A = E_{21}^{-1}E_{31}^{-1}E_{32}^{-1}U
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}=

(Proof in HW1)

\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -\frac{3}{4} & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ -1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}

LU factorisation

E_{32}E_{31}E_{21}A = U
A = (E_{32}E_{31}E_{21})^{-1}U
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ 0 & 0 & 1\\ \end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}_{Elementary~Matrices}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ -2 & 0 & 1\\ \end{bmatrix}
\begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & \frac{3}{4} & 1\\ \end{bmatrix}
E_{31}
E_{32}
E_{21}
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}
\begin{bmatrix} 1&2&-1\\ 0&4&-4\\ 0&0&1 \end{bmatrix}
\therefore A = E_{21}^{-1}E_{31}^{-1}E_{32}^{-1}U
\begin{bmatrix} 1&2&-1\\ -1&2&-3\\ 2&1&2 \end{bmatrix}=

(Proof in HW1)

\begin{bmatrix} 1 & 0 & 0\\ -1 & 1 & 0\\ 2 & -\frac{3}{4} & 1\\ \end{bmatrix}
A
L
U
(Unlike E, the multipliers sit nicely in the right positions in L)
E
\begin{bmatrix} 1 & 0 & 0\\ 1 & 1 & 0\\ \frac{-5}{4} & \frac{3}{4} & 1\\ \end{bmatrix}

LU factorisation

EA = U
A = LU
lower 

upper 

triangular

L and U contain all the information about:

the matrix A and

the Gaussian Elimination of A

Learning Objectives Achieved

Why did elimination work so well in the last lecture?

What is LU factorisation?

How to represent Gauss Elimination as matrix multiplication?

What is the intuition behind 0 and infinite solutions?

(for today's lecture)

Made with Slides.com