CS6015: Linear Algebra and Random Processes
Lecture 9: Solving Ax=b, Rank Nullity Theorem, Some unsolved mysteries
Learning Objectives
How do you find one solution for Ax=b? (if it exists)
What is the connection between rank and number of solutions?
How do you find all solutions for Ax =b?
(for today's lecture)
A couple of unsolved mysteries!
Recap: Two possibilities
Solution does not exist
GE will discover this
Solution exists
GE will discover this
We will now see how to find the solutions
(ridiculously easy)
0=1
0=1
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
x
\mathbf{x}
=1101
=\begin{bmatrix}
1\\
1\\
0\\
1
\end{bmatrix}
0=0
0=0
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
x
\mathbf{x}
=1100
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
Solving Ax = b
111112342345
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
x
\mathbf{x}
=1234
=\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
Gauss Elimination
Set the free variables to 0
Why is it okay to do this?
Solve for pivot variables by back-substitution
x1+x2+2x3=1
x_1+x_2+2x_3 = 1
x2+x3=1
x_2+x_3 = 1
x2+0=1
x_2+ 0 = 1
∴x2=1
\therefore x_2 = 1
x1+1+2(0)=1
x_1+1+2(0) = 1
∴x1=0
\therefore x_1 = 0
xparticular=010
\mathbf{x}_{particular}=\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
if we can write b as a linear combination of the 3 columns then we should also be able to write it as a linear combination of the 2 pivot/independent columns (the third column is redundant as it does not add any new information)
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
=1100
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
x1x2x3
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
\underbrace{~}
Free columns
Solving Ax = b
111112342345
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
x
\mathbf{x}
=1234
=\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
Gauss Elimination
xparticular=010
\mathbf{x}_{particular}=\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
(the complete solution)
xcomplete=xparticular+xnullspace
\mathbf{x}_{complete}=\mathbf{x}_{particular} + \mathbf{x}_{nullspace}
xcomplete=
\mathbf{x}_{complete}=
+ c−1−11
+~~~c\begin{bmatrix}
-1\\
-1\\
1\end{bmatrix}
010
\begin{bmatrix}
0\\1\\0
\end{bmatrix}
A
A
A
A
A
A
=
=
b
\mathbf{b}
+ 0
+~~\mathbf{0}
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
=1100
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
x1x2x3
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
\underbrace{~}
Free columns
Solving Ax = b
xparticular=−3800
\mathbf{x}_{particular}=\begin{bmatrix}
-3\\
8\\
0\\
0\\
\end{bmatrix}
(another example)
112123134145
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
=51318
=\begin{bmatrix}
5\\
13\\
18
\end{bmatrix}
x
\mathbf{x}
Gauss Elimination
x1+x2+x3+x4=5
x_1+x_2+x_3+x_4 = 5
x2+2x3+3x4=8
x_2+2x_3+3x_4 = 8
∴x2=8
\therefore x_2 = 8
∴x1=−3
\therefore x_1 = -3
Set the free variables to 0
Solve for pivot variables by back-substitution
x1+8+0+0=5
x_1+8+0+0 = 5
x2+2(0)+3(0)=8
x_2+2(0)+3(0)= 8
Pivot columns
\underbrace{~~~~~~~~~~}
Free columns
x1x2x3x4
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
100110120130
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
\underbrace{~~~~~~~~~~}
=580
=\begin{bmatrix}
5\\
8\\
0
\end{bmatrix}
Solving Ax = b
(another example)
xcomplete=xparticular+xnullspace
\mathbf{x}_{complete}=\mathbf{x}_{particular} + \mathbf{x}_{nullspace}
xcomplete=−3800
\mathbf{x}_{complete}=
\begin{bmatrix}
-3\\
8\\
0\\
0\\
\end{bmatrix}
+ c1−210+ d2−301
+~c\begin{bmatrix}
1\\
-2\\
1\\
0
\end{bmatrix}
+~d\begin{bmatrix}
2\\
-3\\
0\\
1
\end{bmatrix}
112123134145
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
Free columns
x1x2x3x4
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
=51318
=\begin{bmatrix}
5\\
13\\
18
\end{bmatrix}
100110120130
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
\underbrace{~~~~~~~~~~}
x
\mathbf{x}
Gauss Elimination
=580
=\begin{bmatrix}
5\\
8\\
0
\end{bmatrix}
Solving Ax = b
(the geometric view)
(switch to geogebra)
xcomplete=
\mathbf{x}_{complete}=
+ c−1−11
+~~~c\begin{bmatrix}
-1\\
-1\\
1\end{bmatrix}
010
\begin{bmatrix}
0\\1\\0
\end{bmatrix}
Nullspace: all the multiples of vector (-1,-1,1), Particular solution: vector (0,1,0)
Solving Ax = b
(the geometric view)
Nullspace: xy plane , Particular solution: vector (1,1,1)
Solving Ax = b
(the geometric view)
Rank of a matrix
What does the rank tell us about the number of solutions?
rank
= number of independent rows = number of non-zero pivots after GE = number of independent columns
(everything)
Pivot columns
Free columns
rank<n,m
rank < n,m
Non-zero pivots
zero pivots
rows with all 0s after GE
Things that we know so far....
Rows with zeros ⟹ it is possible that there are 0 solutions
Free columns ⟹ non-zero nullspace ⟹ infinite solutions (if 1 exists)
No free columns ⟹ zero nullspace ⟹ 1 solution (if 1 exists)
1 solution
1~solution
∞ solutions
\infty~solutions
0 or 1 solution
0~or~1~solution
0 or ∞ solutions
0~or~\infty~solutions
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
rank=m=n
rank=m=n
rank=m<n
rank=m < n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~~~~~}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
No Free columns
rank=n<m
rank=n < m
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~~~~~~~~~~}
Free columns
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
\underbrace{~~~~~~}
Free
Pivot
\underbrace{~~~~~~~~~}
Practice Problems
For each of the above matrices, find out the number of possible solutions for Ax = b (for any b)
112211320
\begin{bmatrix}
1&2&3\\
1&1&2\\
2&1&0
\end{bmatrix}
112220320311212122
\begin{bmatrix}
1&2&3&3&2&1\\
1&2&2&1&1&2\\
2&0&0&1&2&2
\end{bmatrix}
0123−122211112131−20
\begin{bmatrix}
0&2&2\\
1&2&1\\
2&1&3\\
3&1&1\\
-1&1&-2\\
2&1&0\\
\end{bmatrix}
Bonus Q: Also figure out what does the rref look like
124215306317226124
\begin{bmatrix}
1&2&3&3&2&1\\
2&1&0&1&2&2\\
4&5&6&7&6&4
\end{bmatrix}
Rank Nullity Theorem
rank(A)+nullity(A)=n
rank(A) + nullity(A) = n
rank(A)=dimension of column space of A
rank(A) = dimension~of~column~space~of~A
nullity(A)=dimension of nullspace of A
nullity(A) = dimension~of~nullspace~of~A
n=number of columns of A
n = number~of~columns~of~A
https://www.math.purdue.edu/files/academic/courses/2010spring/MA26200/4-9.pdf
The mystery
Why is the number of independent rows equal to the number of independent columns? (or why does GE discover independent columns)
Outline of Proof: Enough to show that the dimension of the column space of A is the same as the dimension of the column space of A⊤
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
111112342345
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
A
A
112123134145
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
100110120130
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
A⊤
A^\top
The mystery
Why is it enough to show this?
100011002100
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
111112342345
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
112123134145
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
100110120130
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
A
A
A⊤
A^\top
Outline of Proof: Enough to show that the dimension of the column space of A is the same as the dimension of the column space of A⊤
because we know that ... ...
number of independent columns of A = dimension of column space of A
number of independent columns of A⊤ = dimension of column space of A⊤
number of independent columns of A⊤ = number of independent rows of A
The mystery
Theorem: A⊤Ax=0 iff Ax=0
(number of indep. rows = number of indep. columns)
Proof (the if part): A⊤Ax=0 if Ax=0
Proof (the only if part): Ax=0 if A⊤Ax=0
Ax=0
N(A⊤A)=N(A)
Multiplying both sides by A⊤
A⊤Ax=A⊤0=0
Multiplying both sides by x⊤
A⊤Ax=0
⟹
x⊤A⊤Ax=0
(Ax)⊤Ax=0
⟹
Ax=0
The mystery
(number of indep. rows = number of indep. columns)
Previous Theorem:
Ax=0 iff A⊤(Ax)=0
⟹ N(A)=N(A⊤A)
⟹ dim(N(A))=dim(N(A⊤A))
⟹ dim(C(A))=dim(C(A⊤A))≤dim(C(A⊤))
(rank + nullity thm.)
now replace A by A⊤
A⊤x=0 iff A(A⊤x)=0
⟹ N(A⊤)=N(AA⊤)
(columns of A⊤A are lin. comb. of columns of A⊤ )
⟹ dim(N(A⊤))=dim(N(AA⊤))
⟹ dim(C(A⊤))=dim(C(AA⊤))≤dim(C(A))
The mystery
(number of indep. rows = number of indep. columns)
∴ dim(C(A))=dim(C(A⊤))
∴ rank(A)=rank(A⊤)
∴ number of indep. columns of A = number of indep. columns of A⊤
∴ number of indep. columns of A = number of indep. rows of A
Hence GE discovers independent columns while discovering independent rows!
Another mystery
The number of vectors in the basis of Rn cannot be greater than n
(coming soon
..... in Quiz 1)
Hint: You can answer this based on whatever you have learnt today!
Learning Objectives
How do you find one solution for Ax=b? (if it exists)
What is the connection between rank and number of solutions?
How do you find all solutions for Ax =b?
(achieved)
A couple of unsolved mysteries!
CS6015: Linear Algebra and Random Processes Lecture 9: Solving Ax=b, Rank Nullity Theorem, Some unsolved mysteries
CS6015: Lecture 9
By Mitesh Khapra
CS6015: Lecture 9
Lecture 9: Solving Ax=b, Rank Nullity Theorem, Some unsolved mysteries
- 2,973
