CS6015: Linear Algebra and Random Processes
Lecture 7: Column space of a matrix, null space of a matrix
Learning Objectives
What is the column space of a matrix?
Why do we care about these two subspaces?
What is the null space of a matrix?
(for today's lecture)
The bigger picture
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m < n
m=n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
m > n
rank =
A
A
A
Q1. For which b's does a solution not exist?
0,1,\infty~solutions
Q2. Which x's are a solution for Ax = 0?
Q3. What's the connection between these two Qs?
A\mathbf{x}=\mathbf{b}
n pivots
1 unique solution
Find L,U
Or find
A^{-1}
Column space of a matrix
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix}
b_1\\
b_2\\
b_3\\
b_4
\end{bmatrix}
\mathbf{b}
Q: Which are the \( \mathbf{b} \)'s for which a solution to \( A\mathbf{x} = \mathbf{b}\) will not exist ?
A: For those \( \mathbf{b} \)'s which do not lie in the column space of \(A\)
column space: all linear combinations of the columns of A
(these are the b's which cannot be expressed as a linear combination of the columns of A)
The (sub)space spanned by the columns of the matrix \( A \) is called the column space of the matrix
Column space of a matrix
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix}
b_1\\
b_2\\
b_3\\
b_4
\end{bmatrix}
\mathbf{b}
Q: What is the dimension of the column space of A?
A: Same as the number of independent columns of \( A \) : 2 in this example
column space: all linear combinations of the columns of A
(a 2 dimensional subspace within a 4 dimensional space - which means a solution will not exist for many vectors in \( \mathbb{R}^4 \) )
Column space of a matrix
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix}
b_1\\
b_2\\
b_3\\
b_4
\end{bmatrix}
\mathbf{b}
Q: Can you tell me some \( \mathbf{b} \)'s for which a solution will exist?
column space: all linear combinations of the columns of A
\mathbf{b}=\mathbf{a_1}
\mathbf{b}=\mathbf{a_2}
\mathbf{b}=\mathbf{a_3}
\mathbf{x} = [1~0~0]^\top
\mathbf{x} = [0~1~0]^\top
\mathbf{x} = [0~0~1]^\top
\mathbf{b}=2\mathbf{a_1}-\mathbf{a_2}+5\mathbf{a_3}
\mathbf{x} = [2~-1~5]^\top
(and many more: all linear combinations of the columns of A)
Column space of a matrix
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix}
1\\
2\\
3\\
5
\end{bmatrix}
\mathbf{b}
Q: Will a solution exist for the given \(\mathbf{b}\)?
A: Hard to say by just looking at the RHS and the LHS
column space: all linear combinations of the columns of A
(but Gaussian Elimination will figure it out!)
(Recap)
What does Gauss Elimination tell us?
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
0=0
0=1
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&2&2\\
0&3&3\\
\end{bmatrix}
(row2 = row2 -row1)
(row3 = row3 -row1)
(row4 = row4 -row1)
\mathbf{x}
=\begin{bmatrix}
1\\
1\\
2\\
3
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
1\\
0\\
0
\end{bmatrix}
(row3 = row3 - 2row2)
(row4 = row4 - 3row2)
=\begin{bmatrix}
1\\
2\\
3\\
5
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
1\\
1\\
0\\
1
\end{bmatrix}
The bigger picture
\begin{bmatrix}
~~~&~~~&~~~\\
~~~&~~~&~~~\\
~~~&~~~&~~~\\
\end{bmatrix}
m < n
m=n
\begin{bmatrix}
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
~~~&~~~&~~~&~~~&~~~\\
\end{bmatrix}
\begin{bmatrix}
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&\\
~~~&~~~&
\end{bmatrix}
m > n
rank =
A
A
A
Q1. For which b's does a solution not exist?
0,1,\infty~solutions
Q2. Which x's are a solution for Ax = 0?
Q3. What's the connection between these two Qs?
A\mathbf{x}=\mathbf{b}
n pivots
1 unique solution
Find L,U
Or find
A^{-1}
Logically, the next Q should have been when would \( A\mathbf{x} = \mathbf{b} \) have infinite solutions ?
But, instead we are asking a different question: What are the solution to \(A\mathbf {x} = \mathbf {0}\) ?
Don't worry, all roads lead to Chennai!
Ax = 0
Linear equations
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
0\\
0\\
0\\
0
\end{bmatrix}
Q: What is a trivial solution for the above system of equations?
A: \( \mathbf{x} = \mathbf{0} \) is always a solution for \( A\mathbf{x} = 0 \) for any \( A \)
(a 0-combination of the columns of A will always produce a 0 vector)
Q: When would a non-0 comb. of the columns of \( A \) produce a 0 vector?
0-combination: a linear combination where each vector is scaled by 0 (my informal term)
A: When the columns of \( A \) are linearly dependent!
(this simply follows from the definition of independence)
Ax = 0
Linear equations
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
0\\
0\\
0\\
0
\end{bmatrix}
Q: Can you think of a solution other than the trivial solution?
A: \( \mathbf{x} = [1~~1~-1]^\top \)
Q: Can you specify one more solution to the above equation?
Q: Can you specify all the solutions to the above equation?
A: \( \mathbf{x} = [2~~2~-2]^\top \)
A: \( \mathbf{x} = [c~~c~-c]^\top \)
(a line)
Ax = 0
Linear equations
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
0\\
0\\
0\\
0
\end{bmatrix}
Q: But, in general, how do we know if the columns are dependent ?
A: Well, Gaussian Elimination will tell us!
What does Gauss Elimination tell us?
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&2&2\\
0&3&3\\
\end{bmatrix}
(r2 = r2 - r1)
(r3 = r3 - r1)
(r4 = r4 - r1)
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&0\\
0&0&0\\
\end{bmatrix}
(r3 = r3 - 2r2)
(r4 = r4 - 3r2)
number of independent columns = number of non-zero pivots after GE
Why?
We are not ready to fully understand that yet but we will get there soon
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&5\\
1&4&5\\
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&2&3\\
0&3&3\\
\end{bmatrix}
\begin{bmatrix}
1&1&2\\
0&1&1\\
0&0&1\\
0&0&0\\
\end{bmatrix}
The nullspace of A: \( \mathcal{N}(A) \)
How do we know that this is a subspace?
Proof:
To prove that if \( \mathbf{u} \in N(A) \) and \( \mathbf{v} \in N(A) \) then \( c\mathbf{u} + d\mathbf{v} \in N(A) \)
\because \mathbf{u},\mathbf{v} \in \mathcal{N}(A), ~~A\mathbf{u}=A\mathbf{v}=\mathbf{0}
A(c\mathbf{u} + d\mathbf{v}) = A (c\mathbf{u}) + A (d\mathbf{v})
= c(A\mathbf{u}) + d(A\mathbf{v})
= c(\mathbf{0}) + d(\mathbf{0})
= \mathbf{0}
Hence~proved, c\mathbf{u} + d\mathbf{v} \in \mathcal{N}(A)
The nullspace of a matrix consists of all vectors \(\mathbf{x}\) such that \(\mathbf{x}\) is a solution to \(A\mathbf{x} = \mathbf{0}\)
Summary: The two subspaces
A\mathbf{x} = \mathbf{b}
m\times n
n\times 1
m\times 1
Column space of A: \( \mathcal{C}(A) \)
Null space of A: \( \mathcal{N}(A) \)
a subspace by design
a subspace (we proved it)
subspace of \(\mathbb{R}^m \)
subspace of \(\mathbb{R}^n \)
a subspace spanned by the indep. cols. of \( A \)
the cols. of A are indep. when \( \mathcal{N}(A) = \mathbf{0}\)
\begin{bmatrix}
1&1&2\\
1&2&3\\
1&3&4\\
1&4&5\\
\end{bmatrix}
\mathbf{x}
=\begin{bmatrix}
b_1\\
b_2\\
b_3\\
b_4
\end{bmatrix}
Why do we care about these subspaces
Column space of A compactly defines all the \(\mathbf{b}'s\) for which a solution to \(A\mathbf{x} \) will exist
What does the null space tell us?
(a solution to Ax = b exists)
Suppose,
\(\mathbf{b}\) lies in the column space of \( A \)
the nullspace of \( A \) is a non-zero, non-empty subspace of \( \mathbb{R}^n \)
How many solutions exist for \(A\mathbf{x} = \mathbf{b} \)?
(a solution to Ax = b exists)
(a line or a 2d plane or a 3d plane or ...)
Why do we care about these subspaces
(a solution to Ax = b exists)
Suppose,
\(\mathbf{b}\) lies in the column space of \( A \)
the nullspace of \( A \) is a non-zero subspace of \( \mathbb{R}^n \)
How many solutions exist for \( \mathbf{b} \)?
(a line or a 2d plane or a 3d plane or ...)
Let \( \mathbf{u} \in \mathcal{N}(A), \mathbf{u} \neq \mathbf{0}\)
Let \( \mathbf{x} \in \mathbb{R}^n\) be a solution for \(A\mathbf{x} = \mathbf{b}\)
(we know that a solution exists)
A(\mathbf{x} + c\mathbf{u})
= A\mathbf{x} + cA\mathbf{u}
= \mathbf{b} + 0
=\mathbf{b}
c \in \mathbb{R}
\mathbf{x} + c\mathbf{u}
is also a solution \( \forall c \in \mathbb{R} \)
(a whole line, infinite solutions)
Why do we care about these subspaces
Is b in the column space of A?
Is nullspace of A non-zero
No
Yes
No
Yes
1~solution
\infty~solutions
0~solutions
GE = Gaussian Elimination
0=k
GE will discover this
GE will discover this
\#pivots = \#ind. cols.
GE will discover this
GE will discover this
Learning Objectives
What is the column space of a matrix?
Why do we care about these two subspaces?
What is the null space of a matrix?
(achieved)
Feedback
The involvement of TAs is not effective
Any concrete suggestions on what can be done?
More practice problems, answers to tutorials
Please refer to specific sections in the textbook
Also refer to http://web.mit.edu/18.06/www/old.shtml
More quizzes/short tests
Did you mean ungraded quizzes/tests? (logistics)
Release slides in advance
already being done (please check my webpage)
Doubts: don't entertain, take only at the end, can be handled by TAs
sorry, can't really do that
Increase the pace
trying my best, will try to makeup with extra classes
More applications
this is a theory course with applications in other courses
Latex is painful!
sorry, I don't agree
\begin{bmatrix}
1&1&2&2\\
1&2&3&2\\
1&3&5&1\\
1&4&5&3\\
\end{bmatrix}
\begin{bmatrix}
1&1&2&2\\
0&1&1&0\\
0&2&3&-1\\
0&3&3&1\\
\end{bmatrix}
\begin{bmatrix}
1&1&2&2\\
0&1&1&0\\
0&0&1&-1\\
0&0&0&1\\
\end{bmatrix}
\begin{bmatrix}
1&1&2&2\\
0&1&1&0\\
0&0&1&-1\\
0&0&0&1\\
\end{bmatrix}
\begin{bmatrix}
1&1&2&2\\
0&1&1&0\\
0&0&1&-1\\
0&0&0&1\\
\end{bmatrix}
\begin{bmatrix}
x_1\\
x_2\\
x_3\\
x_4
\end{bmatrix}
x_1 + x_2 + 2 x_3 + 2 x_4 = 0
x_2 + x_3 = 0
x_3 + (-1)x_4 = 0
x_4 = 0
CS6015: Lecture 7
By Mitesh Khapra
CS6015: Lecture 7
Lecture 7: column space of a matrix, null space of a matrix
