CS6015: Linear Algebra and Random Processes

Lecture 7: Column space of a matrix, null space of a matrix

Learning Objectives

What is the column space of a matrix?

Why do we care about these two subspaces?

What is the null space of a matrix?

(for today's lecture)

The bigger picture

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n
m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A

Q1. For which b's does a solution not exist?

0,1,\infty~solutions

Q2. Which x's are a solution for Ax = 0?

Q3. What's the connection between these two Qs?

A\mathbf{x}=\mathbf{b}

n pivots

1 unique solution

Find L,U

Or find

A^{-1}

Column space of a matrix

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix} b_1\\ b_2\\ b_3\\ b_4 \end{bmatrix}
\mathbf{b}

Q: Which are the \( \mathbf{b} \)'s for which a solution to \( A\mathbf{x} = \mathbf{b}\) will not exist ?

A: For those \( \mathbf{b} \)'s which do not lie in the column space of \(A\) 

column space: all linear combinations of the columns of A

(these are the b's which cannot be expressed as a linear combination of the columns of A)

The (sub)space spanned by the columns of the matrix \( A \) is called the column space of the matrix

Column space of a matrix

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix} b_1\\ b_2\\ b_3\\ b_4 \end{bmatrix}
\mathbf{b}

Q: What is the dimension of the column space of A?

A: Same as the number of independent columns of \( A \) : 2 in this example

column space: all linear combinations of the columns of A

(a 2 dimensional subspace within a 4 dimensional space - which means a solution will not exist for many vectors in \( \mathbb{R}^4 \) )

Column space of a matrix

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix} b_1\\ b_2\\ b_3\\ b_4 \end{bmatrix}
\mathbf{b}

Q: Can you tell me some \( \mathbf{b} \)'s  for which a solution will exist?

column space: all linear combinations of the columns of A

\mathbf{b}=\mathbf{a_1}
\mathbf{b}=\mathbf{a_2}
\mathbf{b}=\mathbf{a_3}
\mathbf{x} = [1~0~0]^\top
\mathbf{x} = [0~1~0]^\top
\mathbf{x} = [0~0~1]^\top
\mathbf{b}=2\mathbf{a_1}-\mathbf{a_2}+5\mathbf{a_3}
\mathbf{x} = [2~-1~5]^\top

(and many more: all linear combinations of the columns of A)

Column space of a matrix

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{a_1}
\mathbf{a_2}
\mathbf{a_3}
\mathbf{x}
=\begin{bmatrix} 1\\ 2\\ 3\\ 5 \end{bmatrix}
\mathbf{b}

Q: Will a solution exist for the given \(\mathbf{b}\)?

A: Hard to say by just looking at the RHS and the LHS

column space: all linear combinations of the columns of A

(but Gaussian Elimination will figure it out!)

(Recap)

What does Gauss Elimination tell us?

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 1\\ 2\\ 3\\ 4 \end{bmatrix}
0=0
0=1
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&2&2\\ 0&3&3\\ \end{bmatrix}
(row2 = row2 -row1) 
(row3 = row3 -row1) 
(row4 = row4 -row1) 
\mathbf{x}
=\begin{bmatrix} 1\\ 1\\ 2\\ 3 \end{bmatrix}
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&0\\ 0&0&0\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 1\\ 1\\ 0\\ 0 \end{bmatrix}
(row3 = row3 - 2row2) 
(row4 = row4 - 3row2) 
=\begin{bmatrix} 1\\ 2\\ 3\\ 5 \end{bmatrix}
\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&0\\ 0&0&0\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 1\\ 1\\ 0\\ 1 \end{bmatrix}

The bigger picture

\begin{bmatrix} ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ ~~~&~~~&~~~\\ \end{bmatrix}
m < n
m=n
\begin{bmatrix} ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ ~~~&~~~&~~~&~~~&~~~\\ \end{bmatrix}
\begin{bmatrix} ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~&\\ ~~~&~~~& \end{bmatrix}
m > n
rank =
A
A
A

Q1. For which b's does a solution not exist?

0,1,\infty~solutions

Q2. Which x's are a solution for Ax = 0?

Q3. What's the connection between these two Qs?

A\mathbf{x}=\mathbf{b}

n pivots

1 unique solution

Find L,U

Or find

A^{-1}

Logically, the next Q should have been when would \( A\mathbf{x} = \mathbf{b} \) have infinite solutions ?

But, instead we are asking a different question: What are the solution to \(A\mathbf {x} = \mathbf {0}\) ?

Don't worry, all roads lead to Chennai!

Ax = 0

Linear equations

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 0\\ 0\\ 0\\ 0 \end{bmatrix}

Q: What is a trivial solution for the above system of equations?

A: \( \mathbf{x} = \mathbf{0} \) is always a solution for \( A\mathbf{x} = 0 \) for any \( A \)

(a 0-combination of the columns of A will always produce a 0 vector)

Q: When would a non-0 comb. of the columns of \( A \) produce a 0 vector?

0-combination: a linear combination where each vector is scaled by 0 (my informal term)

A: When the columns of \( A \) are linearly dependent!

(this simply follows from the definition of independence)

Ax = 0

Linear equations

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 0\\ 0\\ 0\\ 0 \end{bmatrix}

Q: Can you think of a solution other than the trivial solution?

A: \( \mathbf{x} = [1~~1~-1]^\top \) 

Q: Can you specify one more solution to the above equation?

Q: Can you specify all the solutions to the above equation?

A: \( \mathbf{x} = [2~~2~-2]^\top \) 

A: \( \mathbf{x} = [c~~c~-c]^\top \) 

(a line)

Ax = 0

Linear equations

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} 0\\ 0\\ 0\\ 0 \end{bmatrix}

Q: But, in general, how do we know if the columns are dependent ?

A: Well, Gaussian Elimination will tell us!

What does Gauss Elimination tell us?

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&2&2\\ 0&3&3\\ \end{bmatrix}
(r2 = r2 - r1) 
(r3 = r3 - r1) 
(r4 = r4 - r1) 
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&0\\ 0&0&0\\ \end{bmatrix}
(r3 = r3 - 2r2) 
(r4 = r4 - 3r2) 

number of independent columns = number of non-zero pivots after GE

Why?

We are not ready to fully understand that yet but we will get there soon
\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&5\\ 1&4&5\\ \end{bmatrix}
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&2&3\\ 0&3&3\\ \end{bmatrix}
\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&1\\ 0&0&0\\ \end{bmatrix}

The nullspace of A: \( \mathcal{N}(A) \)

How do we know that this is a subspace?

Proof:

To prove that if \( \mathbf{u} \in N(A) \) and \( \mathbf{v} \in N(A) \) then \( c\mathbf{u} + d\mathbf{v} \in N(A) \)

\because \mathbf{u},\mathbf{v} \in \mathcal{N}(A), ~~A\mathbf{u}=A\mathbf{v}=\mathbf{0}
A(c\mathbf{u} + d\mathbf{v}) = A (c\mathbf{u}) + A (d\mathbf{v})
= c(A\mathbf{u}) + d(A\mathbf{v})
= c(\mathbf{0}) + d(\mathbf{0})
= \mathbf{0}
Hence~proved, c\mathbf{u} + d\mathbf{v} \in \mathcal{N}(A)

The nullspace of a matrix consists of all vectors \(\mathbf{x}\) such that \(\mathbf{x}\) is a solution to \(A\mathbf{x} = \mathbf{0}\) 

Summary: The two subspaces

A\mathbf{x} = \mathbf{b}
m\times n
n\times 1
m\times 1

Column space of A: \( \mathcal{C}(A) \)

Null space of A: \( \mathcal{N}(A) \)

a subspace by design

a subspace (we proved it)

subspace of \(\mathbb{R}^m \)

subspace of \(\mathbb{R}^n \)

a subspace spanned by the indep. cols. of \( A \)

the cols. of A are indep. when \( \mathcal{N}(A)  = \mathbf{0}\)

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ 1&4&5\\ \end{bmatrix}
\mathbf{x}
=\begin{bmatrix} b_1\\ b_2\\ b_3\\ b_4 \end{bmatrix}

Why do we care about these subspaces

Column space of A compactly defines all the \(\mathbf{b}'s\) for which a solution to \(A\mathbf{x} \) will exist

What does the null space tell us?

(a solution to Ax = b exists)

Suppose, 

\(\mathbf{b}\) lies in the column space of \( A \) 

the nullspace of \( A \) is a non-zero, non-empty subspace of \( \mathbb{R}^n \)

How many solutions exist for \(A\mathbf{x} = \mathbf{b} \)?

(a solution to Ax = b exists)
(a line or a 2d plane or a 3d plane or ...)

Why do we care about these subspaces

(a solution to Ax = b exists)

Suppose, 

\(\mathbf{b}\) lies in the column space of \( A \) 

the nullspace of \( A \) is a non-zero subspace of \( \mathbb{R}^n \)

How many solutions exist for \( \mathbf{b} \)?

(a line or a 2d plane or a 3d plane or ...)

Let \( \mathbf{u} \in \mathcal{N}(A),  \mathbf{u} \neq  \mathbf{0}\)

Let \( \mathbf{x} \in \mathbb{R}^n\) be a solution for  \(A\mathbf{x} = \mathbf{b}\)

(we know that a solution exists)
A(\mathbf{x} + c\mathbf{u})
= A\mathbf{x} + cA\mathbf{u}
= \mathbf{b} + 0
=\mathbf{b}
c \in \mathbb{R}
\mathbf{x} + c\mathbf{u}

is also a solution \( \forall c \in \mathbb{R} \)

(a whole line, infinite solutions)

Why do we care about these subspaces

Is b in the column space of A?
Is nullspace of A non-zero 
No
Yes
No
Yes
1~solution
\infty~solutions
0~solutions

GE = Gaussian Elimination

0=k

GE will discover this

GE will discover this

\#pivots = \#ind. cols.

GE will discover this

GE will discover this

Learning Objectives

What is the column space of a matrix?

Why do we care about these two subspaces?

What is the null space of a matrix?

(achieved)

Feedback

The involvement of TAs is not effective

Any concrete suggestions on what can be done?

More practice problems, answers to tutorials

Please refer to specific sections in the textbook
Also refer to http://web.mit.edu/18.06/www/old.shtml

More quizzes/short tests

Did you mean ungraded quizzes/tests? (logistics)

Release slides in advance

already being done (please check my webpage)

Doubts: don't entertain, take only at the end, can be handled by TAs

sorry, can't really do that

Increase the pace

trying my best, will try to makeup with extra classes

More applications

this is a theory course with applications in other courses

Latex is painful!

sorry, I don't agree
\begin{bmatrix} 1&1&2&2\\ 1&2&3&2\\ 1&3&5&1\\ 1&4&5&3\\ \end{bmatrix}
\begin{bmatrix} 1&1&2&2\\ 0&1&1&0\\ 0&2&3&-1\\ 0&3&3&1\\ \end{bmatrix}
\begin{bmatrix} 1&1&2&2\\ 0&1&1&0\\ 0&0&1&-1\\ 0&0&0&1\\ \end{bmatrix}
\begin{bmatrix} 1&1&2&2\\ 0&1&1&0\\ 0&0&1&-1\\ 0&0&0&1\\ \end{bmatrix}
\begin{bmatrix} 1&1&2&2\\ 0&1&1&0\\ 0&0&1&-1\\ 0&0&0&1\\ \end{bmatrix}
\begin{bmatrix} x_1\\ x_2\\ x_3\\ x_4 \end{bmatrix}
x_1 + x_2 + 2 x_3 + 2 x_4 = 0
x_2 + x_3 = 0
x_3 + (-1)x_4 = 0
x_4 = 0