CS6015: Linear Algebra and Random Processes
Lecture 9: Solving Ax=b, Rank Nullity Theorem, Some unsolved mysteries
Learning Objectives
How do you find one solution for Ax=b? (if it exists)
What is the connection between rank and number of solutions?
How do you find all solutions for Ax =b?
(for today's lecture)
A couple of unsolved mysteries!
Recap: Two possibilities
Solution does not exist
GE will discover this
Solution exists
GE will discover this
We will now see how to find the solutions
(ridiculously easy)
0=1
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
1\\
0\\
1
\end{bmatrix}
0=0
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
Solving Ax = b
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
Gauss Elimination
Set the free variables to 0
Why is it okay to do this?
Solve for pivot variables by back-substitution
x_1+x_2+2x_3 = 1
x_2+x_3 = 1
x_2+ 0 = 1
\therefore x_2 = 1
x_1+1+2(0) = 1
\therefore x_1 = 0
\mathbf{x}_{particular}=\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
if we can write b as a linear combination of the 3 columns then we should also be able to write it as a linear combination of the 2 pivot/independent columns (the third column is redundant as it does not add any new information)
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
\underbrace{~}
Free columns
Solving Ax = b
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
Gauss Elimination
\mathbf{x}_{particular}=\begin{bmatrix}
0\\
1\\
0
\end{bmatrix}
(the complete solution)
\mathbf{x}_{complete}=\mathbf{x}_{particular} + \mathbf{x}_{nullspace}
\mathbf{x}_{complete}=
+~~~c\begin{bmatrix}
-1\\
-1\\
1\end{bmatrix}
\begin{bmatrix}
0\\1\\0
\end{bmatrix}
A
A
A
=
\mathbf{b}
+~~\mathbf{0}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
\underbrace{~}
Free columns
Solving Ax = b
\mathbf{x}_{particular}=\begin{bmatrix}
-3\\
8\\
0\\
0\\
\end{bmatrix}
(another example)
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
=\begin{bmatrix}
5\\
13\\
18
\end{bmatrix}
\mathbf{x}
Gauss Elimination
x_1+x_2+x_3+x_4 = 5
x_2+2x_3+3x_4 = 8
\therefore x_2 = 8
\therefore x_1 = -3
Set the free variables to 0
Solve for pivot variables by back-substitution
x_1+8+0+0 = 5
x_2+2(0)+3(0)= 8
Pivot columns
\underbrace{~~~~~~~~~~}
Free columns
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
\underbrace{~~~~~~~~~~}
=\begin{bmatrix}
5\\
8\\
0
\end{bmatrix}
Solving Ax = b
(another example)
\mathbf{x}_{complete}=\mathbf{x}_{particular} + \mathbf{x}_{nullspace}
\mathbf{x}_{complete}=
\begin{bmatrix}
-3\\
8\\
0\\
0\\
\end{bmatrix}
+~c\begin{bmatrix}
1\\
-2\\
1\\
0
\end{bmatrix}
+~d\begin{bmatrix}
2\\
-3\\
0\\
1
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~}
Free columns
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
=\begin{bmatrix}
5\\
13\\
18
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
\underbrace{~~~~~~~~~~}
\mathbf{x}
Gauss Elimination
=\begin{bmatrix}
5\\
8\\
0
\end{bmatrix}
Solving Ax = b
(the geometric view)
(switch to geogebra)
\mathbf{x}_{complete}=
+~~~c\begin{bmatrix}
-1\\
-1\\
1\end{bmatrix}
\begin{bmatrix}
0\\1\\0
\end{bmatrix}
Nullspace: all the multiples of vector (-1,-1,1), Particular solution: vector (0,1,0)
Solving Ax = b
(the geometric view)
Nullspace: xy plane , Particular solution: vector (1,1,1)
Solving Ax = b
(the geometric view)
Rank of a matrix
What does the rank tell us about the number of solutions?
rank
= number of independent rows = number of non-zero pivots after GE = number of independent columns
(everything)
Pivot columns
Free columns
rank < n,m
Non-zero pivots
zero pivots
rows with all 0s after GE
Things that we know so far....
Rows with zeros \(\implies \) it is possible that there are 0 solutions
Free columns \(\implies \) non-zero nullspace \(\implies\) infinite solutions (if 1 exists)
No free columns \(\implies \) zero nullspace \(\implies\) 1 solution (if 1 exists)
1~solution
\infty~solutions
0~or~1~solution
0~or~\infty~solutions
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
rank=m=n
rank=m < n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\underbrace{~~~~~~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~~~~~}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
No Free columns
rank=n < m
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
Pivot columns
\underbrace{~~~~~~~~~~~~~}
\underbrace{~~~~~~~~~~~~~~~~~~~~}
Free columns
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
\underbrace{~~~~~~}
Free
Pivot
\underbrace{~~~~~~~~~}
Practice Problems
For each of the above matrices, find out the number of possible solutions for Ax = b (for any b)
\begin{bmatrix}
1&2&3\\
1&1&2\\
2&1&0
\end{bmatrix}
\begin{bmatrix}
1&2&3&3&2&1\\
1&2&2&1&1&2\\
2&0&0&1&2&2
\end{bmatrix}
\begin{bmatrix}
0&2&2\\
1&2&1\\
2&1&3\\
3&1&1\\
-1&1&-2\\
2&1&0\\
\end{bmatrix}
Bonus Q: Also figure out what does the rref look like
\begin{bmatrix}
1&2&3&3&2&1\\
2&1&0&1&2&2\\
4&5&6&7&6&4
\end{bmatrix}
Rank Nullity Theorem
rank(A) + nullity(A) = n
rank(A) = dimension~of~column~space~of~A
nullity(A) = dimension~of~nullspace~of~A
n = number~of~columns~of~A
https://www.math.purdue.edu/files/academic/courses/2010spring/MA26200/4-9.pdf
The mystery
Why is the number of independent rows equal to the number of independent columns? (or why does GE discover independent columns)
Outline of Proof: Enough to show that the dimension of the column space of \( A \) is the same as the dimension of the column space of \( A^\top \)
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
A
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
A^\top
The mystery
Why is it enough to show this?
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
1&2&3&4\\
2&3&4&5
\end{bmatrix}
\begin{bmatrix}
1&1&1&1\\
0&1&2&3\\
0&0&0&0
\end{bmatrix}
A
A^\top
Outline of Proof: Enough to show that the dimension of the column space of \( A \) is the same as the dimension of the column space of \( A^\top \)
because we know that ... ...
number of independent columns of \(A\) = dimension of column space of \(A\)
number of independent columns of \(A^\top\) = dimension of column space of \(A^\top\)
number of independent columns of \(A^\top\) = number of independent rows of \(A\)
The mystery
Theorem: \(A^\top A\mathbf{x} = 0\) iff \(A\mathbf{x} = 0\)
(number of indep. rows = number of indep. columns)
Proof (the if part): \(A^\top A\mathbf{x} = 0\) if \(A\mathbf{x} = 0\)
Proof (the only if part): \(A\mathbf{x} = 0\) if \(A^\top A\mathbf{x} = 0\)
\(A\mathbf{x} = 0\)
\(\mathcal{N}(A^\top A) = \mathcal{N}(A) \)
Multiplying both sides by \(A^\top\)
\(A^\top A\mathbf{x} = A^\top0 = 0\)
Multiplying both sides by \(\mathbf{x}^\top\)
\(A^\top A\mathbf{x} = 0\)
\(\implies\)
\(\mathbf{x}^\top A^\top A\mathbf{x} = 0\)
\((A\mathbf{x})^\top A\mathbf{x} = 0\)
\(\implies\)
\(A\mathbf{x} = 0\)
The mystery
(number of indep. rows = number of indep. columns)
Previous Theorem:
\(A\mathbf{x} = \mathbf{0}\) iff \(A^\top(A\mathbf{x}) = \mathbf{0}\)
\( \implies \mathcal{N}(A) = \mathcal{N}(A^\top A) \)
\( \implies dim(\mathcal{N}(A)) = dim(\mathcal{N}(A^\top A)) \)
\( \implies dim(\mathcal{C}(A)) = dim(\mathcal{C}(A^\top A)) \leq dim(\mathcal{C}(A^\top))\)
(rank + nullity thm.)
now replace \( A \) by \( A^\top \)
\(A^\top\mathbf{x} = \mathbf{0}\) iff \(A(A^\top\mathbf{x}) = \mathbf{0}\)
\( \implies \mathcal{N}(A^\top) = \mathcal{N}(A A^\top) \)
(columns of \(A^\top A\) are lin. comb. of columns of \(A^\top\) )
\( \implies dim(\mathcal{N}(A^\top)) = dim(\mathcal{N}(AA^\top)) \)
\( \implies dim(\mathcal{C}(A^\top)) = dim(\mathcal{C}(AA^\top)) \leq dim(\mathcal{C}(A))\)
The mystery
(number of indep. rows = number of indep. columns)
\( \therefore dim(\mathcal{C}(A)) = dim(\mathcal{C}(A^\top)) \)
\( \therefore rank(A) = rank(A^\top) \)
\( \therefore \) number of indep. columns of A = number of indep. columns of \( A^\top \)
\( \therefore \) number of indep. columns of A = number of indep. rows of \( A \)
Hence GE discovers independent columns while discovering independent rows!
Another mystery
The number of vectors in the basis of \( \mathbb{R}^n \) cannot be greater than \( n \)
(coming soon
..... in Quiz 1)
Hint: You can answer this based on whatever you have learnt today!
Learning Objectives
How do you find one solution for Ax=b? (if it exists)
What is the connection between rank and number of solutions?
How do you find all solutions for Ax =b?
(achieved)
A couple of unsolved mysteries!
CS6015: Lecture 9
By Mitesh Khapra
CS6015: Lecture 9
Lecture 9: Solving Ax=b, Rank Nullity Theorem, Some unsolved mysteries
