CS6015: Linear Algebra and Random Processes
Lecture 5: PA = LU, cost of Gaussian Elimination, the practical utility of LU factorisation, computing inverse using Gaussian Elimination
Learning Objectives
What if we get a 0 in the pivot position (in the good case) ?
To compute inverse or not, that is the question?
How do you compute the inverse of a matrix using Gaussian Elimination?
Why do we care about LU factorisation?
What is the cost of Gaussian Elimination?
(for today's lecture)
The bigger picture
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m < n
a peek into the future
m=n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
m > n
rank =
A
A
A
(we are still focusing on this nice well-behaved case)
(These two are the more interesting cases that we will come to a bit later in the course)
Things could still go wrong
(even in the good case)
\begin{bmatrix}
1&2&-1\\
-1&-2&-3\\
2&1&2
\end{bmatrix}
=\begin{bmatrix}
0\\
-4\\
4
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
x_1 + 2x_2 -x_3 = 0
-x_1 -2x_2 -3x_3 = -4
2x_1 + x_2 + 2x_3 = 4
\begin{bmatrix}
1&2&-1\\
0&0&-4\\
0&-3&4
\end{bmatrix}
=\begin{bmatrix}
0\\
-4\\
4
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
(row2 = row2 + row1)
E_{21}=\begin{bmatrix}
1&0&0\\
1&1&0\\
0&0&1
\end{bmatrix}
(row3 = row3 - 2*row1)
E_{31}=\begin{bmatrix}
1&0&0\\
0&1&0\\
-2&0&1
\end{bmatrix}
(exchange row 2 and row 3)
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
=\begin{bmatrix}
0\\
4\\
-4
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
P=\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
-4x_3 = -4 \therefore x_3 = 1
-3x_2 +4(1) = 4 \therefore x_2 = 0
x_1 + 2(0) - (1) = 0 \therefore x_1 = 1
Things could still go wrong
(even in the good case)
\begin{bmatrix}
1&2&-1\\
-1&-2&-3\\
2&1&2
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
1&1&0\\
0&0&1
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
0&1&0\\
-2&0&1
\end{bmatrix}
E_{21}
E_{31}
A
= \begin{bmatrix}
1&0&0\\
-1&1&0\\
2&0&1
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
P
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
=\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
U
A = E_{21}^{-1}E_{31}^{-1}P^{-1}U
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
= \begin{bmatrix}
1&0&0\\
-1&0&1\\
2&1&0
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4\\
\end{bmatrix}
not lower triangular
A
Things could still go wrong
(even in the good case)
\begin{bmatrix}
1&0&0\\
-1&1&0\\
2&0&1
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
A=
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
PA=
\begin{bmatrix}
1&0&0\\
2&0&1\\
-1&1&0\\
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
0&0&1\\
0&1&0
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
2&1&0\\
-1&0&1\\
\end{bmatrix}
PA=
\begin{bmatrix}
1&2&-1\\
0&-3&4\\
0&0&-4
\end{bmatrix}
U
L
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}
\overbrace{~~~~~~~~~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}
\overbrace{~~~~~~~~~~~~~~~~~~~~~}
Things could still go wrong
(even in the good case)
PA=LU
interpretation: rearrange the rows of A in advance so that they are in a nice order so that we won't get any zero pivots
The asymmetry in LU factorisation
\begin{bmatrix}
1&2&-1\\
0&4&-4\\
0&0&1
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
-1&2&-3\\
2&1&2
\end{bmatrix}=
\begin{bmatrix}
1 & 0 & 0\\
-1 & 1 & 0\\
2 & -\frac{3}{4} & 1\\
\end{bmatrix}
A
L
U
(diagonal elements are 1)
(diagonal elements are not 1)
\begin{bmatrix}
a&d&e\\
0&b&f\\
0&0&c
\end{bmatrix}
=\begin{bmatrix}
a&0&0\\
0&b&0\\
0&0&c
\end{bmatrix}
\begin{bmatrix}
1&\frac{d}{a}&\frac{e}{a}\\
0&1&\frac{f}{b}\\
0&0&1
\end{bmatrix}
LDU factorisation
\begin{bmatrix}
1&2&-1\\
0&\frac{4}{4}&\frac{-4}{4}\\
0&0&1
\end{bmatrix}
\begin{bmatrix}
1&2&-1\\
-1&2&-3\\
2&1&2
\end{bmatrix}=
\begin{bmatrix}
1 & 0 & 0\\
-1 & 1 & 0\\
2 & -\frac{3}{4} & 1\\
\end{bmatrix}
A
L
U
(diagonal elements are 1)
(diagonal elements are 1)
\begin{bmatrix}
a&d&e\\
0&b&f\\
0&0&c
\end{bmatrix}
=\begin{bmatrix}
a&0&0\\
0&b&0\\
0&0&c
\end{bmatrix}
\begin{bmatrix}
1&\frac{d}{a}&\frac{e}{a}\\
0&1&\frac{f}{b}\\
0&0&1
\end{bmatrix}
\begin{bmatrix}
1&0&0\\
0&4&0\\
0&0&1
\end{bmatrix}
D
PA=LDU
Is LU factorisation unique?
prove or disprove (HW2)
What is the cost of Gauss Elimination?
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
\begin{bmatrix}
a_{11}&a_{12}&\dots&a_{1n}\\
a_{21}&a_{22}&\dots&a_{2n}\\
\dots&\dots&\dots&\dots\\
a_{n1}&a_{n2}&\dots&a_{nn}\\
\end{bmatrix}
=\begin{bmatrix}
1\\
2\\
3\\
2\\
\end{bmatrix}
A
\mathbf{x}
\mathbf{b}
Q: How many operations to get a 0 in position 2,1?
A: n as all the n elements in row 2 will change
Q: How many operations to get a 0 in all positions below pivot 1 ?
A: (n-1)(n-2) as the n-1 elements of all the n-2 rows will change
Q: How many operations to get a 0 in all positions below pivot 2 ?
A: n(n-1) as the n elements of all the n-1 rows will change
What is the cost of Gauss Elimination?
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
\begin{bmatrix}
a_{11}&a_{12}&\dots&a_{1n}\\
a_{21}&a_{22}&\dots&a_{2n}\\
\dots&\dots&\dots&\dots\\
a_{n1}&a_{n2}&\dots&a_{nn}\\
\end{bmatrix}
=\begin{bmatrix}
1\\
2\\
3\\
2\\
\end{bmatrix}
A
\mathbf{x}
\mathbf{b}
n(n-1) + (n-1)(n-2) + (n-2)(n-3) + \cdots + 2(1)
\approx \frac{1}{3}n^3
Cost for reducing A to U
Associated cost of changing b
(n-1) + (n-2) + \cdots 1
=\frac{1}{2}(n)(n-1)
= O(n^3)
= O(n^2)
Why do we care about LU factorisation?
A = LU
A\mathbf{x} = \mathbf{b_1}
A\mathbf{x} = \mathbf{b_2}
A\mathbf{x} = \mathbf{b_3}
A\mathbf{x} = \mathbf{b_k}
\cdots~~\cdots
we may have to solve Ax=b for different right hand sides
irrespective of b the Gaussian elimination of A does not change
(so we can do it just once )
L contains all the information about the elimination steps
(so we can apply these steps to a new b in time to get a c)
O(n^2)
O(n^3)
We can then solve
U\mathbf{x} = \mathbf{c}
One Lin. System = 2 Triangular Systems
A\mathbf{x} = \mathbf{b}
\equiv
U\mathbf{x} = \mathbf{c}
You already know what U is after LU factorisation
How do you find c?
(triangular system)
Homework:
Prove that Lc = b
implication: c can be obtained by solving this triangular system
What is the cost of solving a triangular system?
hint: it better be less than
O(n^3)
What is the net effect?
One time cost of elimination: ?
Recurring cost of solving Lc=b and Ux =c ?
The bigger picture
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m < n
a peek into the future
m=n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
m > n
rank =
A
A
A
(we are still focusing on this nice well-behaved case)
(These two are the more interesting cases that we will come to a bit later in the case)
Our current method
A = LU
Solve Lc = b
Solve Ux = c
The bigger picture
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m < n
a peek into the future
m=n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
m > n
rank =
A
A
A
(we are still focusing on this nice well-behaved case)
(These two are the more interesting cases that we will come to a bit later in the case)
Alternative method (for the good case)
\mathbf{x} = A^{-1}\mathbf{b}
Is A invertible?
Theorem: A nxn matrix A is invertible iff it has n non-zero pivots
Proof: (the if part) A nxn matrix A is invertible if it has n non-zero pivots
Proof: (the only if part) If a nxn matrix A is invertible it has n non-0 pivots
given
to prove
given
to prove
HW2
HW2
We know that in the good case A has n non-0 pivots and is thus invertible
Is A invertible?
How to find the inverse of A
AA^{-1} = I
\begin{bmatrix}
~~&~~&~~\\
~~&A&~~\\
~~&~~&~~\\
\end{bmatrix}
\begin{bmatrix}
\uparrow&\uparrow&\uparrow\\
\mathbf{x_1}&\mathbf{x_2}&\mathbf{x_3}\\
\downarrow&\downarrow&\downarrow\\
\end{bmatrix}
=\begin{bmatrix}
1&0&0\\
0&1&0\\
0&0&1\\
\end{bmatrix}
A\mathbf{x_1}
=\begin{bmatrix}
1\\
0\\
0\\
\end{bmatrix}
A\mathbf{x_2}
=\begin{bmatrix}
0\\
1\\
0\\
\end{bmatrix}
A\mathbf{x_3}
=\begin{bmatrix}
0\\
0\\
1\\
\end{bmatrix}
n systems of linear equations (for a n x n matrix)
We already know how to solve them
Gauss Jordan method for finding
A^{-1}
\begin{bmatrix}
2&1&1&1&0&0\\
4&-6&0&0&1&0\\
-2&7&2&0&0&1
\end{bmatrix}
row 2 = row 2 - 2*row1
row 3 = row 3 + row1
\begin{bmatrix}
2&1&1&1&0&0\\
0&-8&-2&-2&1&0\\
0&8&3&1&0&1
\end{bmatrix}
(row 3 = row 3 + row2)
\begin{bmatrix}
2&1&1&1&0&0\\
0&-8&-2&-2&1&0\\
0&0&1&-1&1&1
\end{bmatrix}
E_{31}E_{21}
E_{32}
\begin{bmatrix}
2&1&\mathbf{0}&2&-1&-1\\
0&-8&\mathbf{0}&-4&3&2\\
0&0&1&-1&1&1
\end{bmatrix}
row 2 = row 2 + 2*row3
row 1 = row 1 - row3
E_{13}E_{23}
row 1 = row 1 + 1/8*row2
\begin{bmatrix}
2&\mathbf{0}&\mathbf{0}&\frac{12}{8}&-\frac{5}{8}&-\frac{6}{8}\\
0&-8&\mathbf{0}&-4&3&2\\
0&0&1&-1&1&1
\end{bmatrix}
E_{12}
\begin{bmatrix}
1&\mathbf{0}&\mathbf{0}&\frac{12}{16}&\frac{-5}{16}&-\frac{6}{16}\\
0&1&\mathbf{0}&\frac{4}{8}&-\frac{3}{8}&-\frac{2}{8}\\
0&0&1&-1&1&1
\end{bmatrix}
M
=E
A
\mathbf{x_1}
\mathbf{x_2}
\mathbf{x_3}
each row by the pivot
\div
\begin{bmatrix}
2&1\\
0&3
\end{bmatrix}
=\begin{bmatrix}
2&0\\
0&3
\end{bmatrix}
\begin{bmatrix}
1&\frac{1}{2}\\
0&1
\end{bmatrix}
M=\begin{bmatrix}
1/2&0&0\\
0&-1/8&0\\
0&0&1
\end{bmatrix}
Gauss Jordan method for finding
A^{-1}
\begin{bmatrix}
2&1&1&1&0&0\\
4&-6&0&0&1&0\\
-2&7&2&0&0&1
\end{bmatrix}
\begin{bmatrix}
1&\mathbf{0}&\mathbf{0}&\frac{12}{16}&\frac{-5}{16}&-\frac{6}{16}\\
0&1&\mathbf{0}&\frac{4}{8}&-\frac{3}{8}&-\frac{2}{8}\\
0&0&1&-1&1&1
\end{bmatrix}
=
A
I
I
?
E
E
\begin{bmatrix}
A&I
\end{bmatrix}
=\begin{bmatrix}
EA&EI
\end{bmatrix}
=\begin{bmatrix}
I&E
\end{bmatrix}
EA=I
\therefore E=A^{-1}
E=A^{-1}
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m=n
rank =
A
(we are still focusing on this nice well-behaved case)
Method 1
A = LU
Solve Lc = b
Solve Ux = c
The two choices
Method 2
Compute
A^{-1}
Compute
A^{-1}\mathbf{x}
Cost: \approx\frac{1}{3}n^3
Cost: HW2
Cost: HW2
Cost: HW2
Cost: HW2
Invertibility and number of solutions
domain~\mathbb{R}
range~\mathbb{R}
x_1
x_2
x_3
x_j
...
...
y_1
y_2
y_3
y_j
...
...
f
When would the function be invertible?
\(f^{-1}(y) \) exists for every \( y \)
every \( y \) is produced by a unique \( x \)
Invertibility and number of solutions
domain~\mathbb{R}
range~\mathbb{R}
x_1
x_2
x_3
x_j
...
...
y_1
y_2
y_3
y_j
...
...
f
When would the function not be invertible?
multiple \(x \)'s produce the same \( y \)
some \( y \)'s are not produced by any \( x \)
Invertibility and number of solutions
domain~\mathbb{R}
range~\mathbb{R}
\mathbf{x}_1
\mathbf{x}_2
\mathbf{x}_3
\mathbf{x}_j
...
...
\mathbf{b}_1
\mathbf{b}_2
\mathbf{b}_3
\mathbf{b}_k
...
...
A\mathbf{x}
When would the matrix not be invertible?
multiple \(\mathbf{x} \)'s produce the same \( \mathbf{b} \)
some \( \mathbf{b} \)'s are not produced by any \( \mathbf{x} \)
(infinite solutions to Ax = b)
(0 solutions to Ax = b)
What if the matrix is invertible?
a unique \(\mathbf{x} \) for every \( \mathbf{b} \)
(exactly 1 solution to Ax = b for every possible b)
Learning Objectives
(achieved)
What if we get a 0 in the pivot position (in the good case) ?
To compute inverse or not, that is the question?
How do you compute the inverse of a matrix using Gaussian Elimination?
Why do we care about LU factorisation?
What is the cost of Gaussian Elimination?
CS6015: Lecture 5
By Mitesh Khapra
CS6015: Lecture 5
PA = LU, cost of Gaussian Elimination, the practical utility of LU factorisation, computing inverse using Gauss Jordan method
- 2,874
