Please join on https://iitmadras.webex.com/join/miteshk

Learning Objectives

How do vectors and matrices show up in Machine Learning?

What are orthogonal vectors?

How do you compute the norm of a vector?

(for today's lecture)

What do you do when Ax=b does not have a solution (intuition)?

What are orthogonal subspaces?

An example from Machine Learning

Salinity

How much oil can be recovered from a drilling site?

Site 1

Site 2

Site 3

Site m

Pressure

Density

Depth

Temp.

n var.

...

(Rely on past data)

Quantity

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

You know that some relation exists between 'Quantity' and the n variables

But you don't know what $f$ is!

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

So what do you do?

An example from Machine Learning

# of Degrees

How much salary should be offered to a candidate?

Emp. 1

Emp. 2

Emp. 3

Emp. m

# Exp.

# Projects

Univ. Rank

# Prog. Lang. known

n var.

...

(Rely on past employee data)

Salary

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

You know that some relation exists between 'Salary' and the n variables

But you don't know what $f$ is!

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

So what do you do?

An example from Machine Learning

Total Pop.

How many COVID19 cases in the next week in a locality?

Loc. 1

Loc. 2

Loc. 3

Loc. m

Pop. Density

Avg. Income

Last week's count

# masks sold

n var.

...

(Rely on past data)

Cases

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

You know that some relation exists between 'Cases' and the n variables

But you don't know what $f$ is!

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

So what do you do?

A typical Machine Learning setup

You know that some relation exists between the 'output' and the 'input' variables

But you don't know what $f$ is!

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

b_{1} = f(a_{11}, a_{12}, a_{13}, \cdots, a_{1n})

So what do you do?

You make some assumption about $f$

What is the simplest assumption you can make?

simple, interpretable, less predictive (less accurate)

Can you make more complex assumptions?

Yes,

b_{1} = a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots+ a_{1n}x_n

b_{1} = a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots+ a_{1n}x_n

b_{1} = w_kg_k(\cdots w_3g_3(w_2g_2(a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots, a_{1n}x_n)))

b_{1} = w_kg_k(\cdots w_3g_3(w_2g_2(a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots, a_{1n}x_n)))

complex, uninterpretable, more predictive (accurate)

(psst! psst! Deep Learning)

$f$ is linear

In this course

What's the connection to Linear Algebra

a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots+ a_{1n}x_n = b_{1}

a_{11}x_1 + a_{12}x_2 + a_{13}x_3 \cdots+ a_{1n}x_n = b_{1}

a_{21}x_1 + a_{22}x_2 + a_{23}x_3 \cdots+ a_{2n}x_n = b_{2}

a_{21}x_1 + a_{22}x_2 + a_{23}x_3 \cdots+ a_{2n}x_n = b_{2}

a_{31}x_1 + a_{32}x_2 + a_{33}x_3 \cdots+ a_{3n}x_n = b_{3}

a_{31}x_1 + a_{32}x_2 + a_{33}x_3 \cdots+ a_{3n}x_n = b_{3}

a_{m1}x_1 + a_{m2}x_2 + a_{m3}x_3 \cdots+ a_{mn}x_n = b_{m}

a_{m1}x_1 + a_{m2}x_2 + a_{m3}x_3 \cdots+ a_{mn}x_n = b_{m}

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

\cdots

Loc. 1

Loc. 2

Loc. 3

Loc. m

We are interested in finding the $x$ 's which best explain the relationship between the $a$ 's and the $b$ 's

In other words .....

What's the connection to Linear Algebra

Loc. 1

Loc. 2

Loc. 3

Loc. m

We are interested in finding the $x$ 's which best explain the relationship between the $a$ 's and the $b$ 's

In other words .....

..... we are interested in solving $A \mathbf{x} = \mathbf{b}$

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

A

A

\mathbf{x}

\mathbf{x}

\mathbf{b}

\mathbf{b}

What could go wrong?

Loc. 1

Loc. 2

Loc. 3

Loc. m

0 solutions

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

inconsistent equations

For example two patients with the same heart rate, blood sugar level, weight, height (inputs) but a different blood cholesterol level (output)

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =110

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =110

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =120

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =120

What can you do?

Loc. 1

Loc. 2

Loc. 3

Loc. m

0 solutions

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ a_{21}&~&a_{22}&~&a_{23}&~&a_{24}&~&a_{25}&\cdots&a_{2n}\\ a_{31}&~&a_{32}&~&a_{33}&~&a_{34}&~&a_{35}&\cdots&a_{3n}\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ b_{2}\\ b_{3}\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{n}\\ \end{bmatrix}

inconsistent equations

Find the best possible solution by tolerating some noise

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =110

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =110

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =120

72x_1 + 84x_2 + 175x_3 \cdots+ 78x_n =120

115

115

115

115

What is the geometric view?

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ 72&~&84&~&175&~&\cdots&~&\cdots&\cdots&78\\ 72&~&84&~&175&~&\cdots&~&\cdots&\cdots&78\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

\begin{bmatrix} a_{11}&~&a_{12}&~&a_{13}&~&a_{14}&~&a_{15}&\cdots&a_{1n}\\ 72&~&84&~&175&~&\cdots&~&\cdots&\cdots&78\\ 72&~&84&~&175&~&\cdots&~&\cdots&\cdots&78\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ \cdots&~&\cdots&~&\cdots&~&\cdots&~&\cdots&\cdots&\cdots\\ a_{m1}&~&a_{m2}&~&a_{m3}&~&a_{m4}&~&a_{m5}&\cdots&a_{mn}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ 110\\ 120\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

=\begin{bmatrix} b_{1}\\ 110\\ 120\\ \cdots\\ \cdots\\ b_{m}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{m}\\ \end{bmatrix}

\begin{bmatrix} x_{1}\\ x_{2}\\ x_{3}\\ \cdots\\ \cdots\\ x_{m}\\ \end{bmatrix}

A

A

\mathbf{b}

\mathbf{b}

\mathbf{b}

\mathbf{b}

column space of A

\mathbf{p}

\mathbf{p}

\begin{bmatrix} p_{1}\\ 115\\ 115\\ \cdots\\ \cdots\\ p_{m}\\ \end{bmatrix}

\begin{bmatrix} p_{1}\\ 115\\ 115\\ \cdots\\ \cdots\\ p_{m}\\ \end{bmatrix}

\mathbf{p}

\mathbf{p}

"Project" $\mathbf{b}$ into the column space of $A$

Solve $A\mathbf{\hat{x}} = \mathbf{p}$

$\mathbf{\hat{x}}$ is the best possible approximation

$\mathbf{b}$ is not in the column space of $A$

Hence no solution to $A\mathbf{x} = \mathbf{b}$

What next?

How do we project $\mathbf{b}$ into the column space of $A$ ?

But first a detour to build some concepts ...

What are orthogonal vectors?

What is the norm of a vector?

What are orthogonal subspaces?

The norm of a vector

Given a vector space $V$ , the norm of a vector is a non-negative valued function $p: V \rightarrow \mathbb{R}$ with the following properties

p(\mathbf{u} + \mathbf{v}) \leq p(\mathbf{u}) + p(\mathbf{v})

p(\mathbf{u} + \mathbf{v}) \leq p(\mathbf{u}) + p(\mathbf{v})

p(a\mathbf{u}) = |a| p(\mathbf{u})

p(a\mathbf{u}) = |a| p(\mathbf{u})

If~p(\mathbf{u}) = 0~then~\mathbf{u}=\mathbf{0}

If~p(\mathbf{u}) = 0~then~\mathbf{u}=\mathbf{0}

(triangular inequality)

\mathbf{u}

\mathbf{u}

\mathbf{v}

\mathbf{v}

\mathbf{u}+\mathbf{v}

\mathbf{u}+\mathbf{v}

Examples of norm

L_{1}~norm

L_{1}~norm

||\mathbf{x}||_1 = |x_1| + |x_2| + |x_3| + \cdots + |x_n|

||\mathbf{x}||_1 = |x_1| + |x_2| + |x_3| + \cdots + |x_n|

(\mathcal{l}_{1}~norm)

(\mathcal{l}_{1}~norm)

L_{2}~norm

L_{2}~norm

(\mathcal{l}_{2}~norm)

(\mathcal{l}_{2}~norm)

(Euclidean~norm)

(Euclidean~norm)

(taxicab~norm)

(taxicab~norm)

||\mathbf{x}||_2 = (|x_1|^2 + |x_2|^2 + |x_3|^2 + \cdots + |x_n|^2)^{\frac{1}{2}}

||\mathbf{x}||_2 = (|x_1|^2 + |x_2|^2 + |x_3|^2 + \cdots + |x_n|^2)^{\frac{1}{2}}

= \sqrt{x_1^2 + x_2^2 + x_3^2 + \cdots + x_n^2}

= \sqrt{x_1^2 + x_2^2 + x_3^2 + \cdots + x_n^2}

L_{p}~norm

L_{p}~norm

(\mathcal{l}_{p}~norm)

(\mathcal{l}_{p}~norm)

||\mathbf{x}||_p = (|x_1|^p + |x_2|^p + |x_3|^p + \cdots + |x_n|^p)^{\frac{1}{p}}

||\mathbf{x}||_p = (|x_1|^p + |x_2|^p + |x_3|^p + \cdots + |x_n|^p)^{\frac{1}{p}}

= \sqrt{\mathbf{x}^\top\mathbf{x}}

= \sqrt{\mathbf{x}^\top\mathbf{x}}

[x_1~x_2~x_3~\cdots~x_n]

[x_1~x_2~x_3~\cdots~x_n]

\begin{bmatrix} x_1\\ x_2\\ x_3\\ \cdots\\ x_n \end{bmatrix}

\begin{bmatrix} x_1\\ x_2\\ x_3\\ \cdots\\ x_n \end{bmatrix}

\mathbf{x}^\top

\mathbf{x}^\top

\mathbf{x}

\mathbf{x}

(most commonly used)

= \sum_{i=1}^{n}(|x_i|^p)^{\frac{1}{p}}

= \sum_{i=1}^{n}(|x_i|^p)^{\frac{1}{p}}

Orthogonal vectors

Vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if the angle between them is $90\degree$

Condition for orthogonality

By Pythagoras' Theorem

(||\mathbf{u}+\mathbf{v}||_2)^2 = (||\mathbf{u}||_2)^2 + (||\mathbf{v}||_2)^2

(||\mathbf{u}+\mathbf{v}||_2)^2 = (||\mathbf{u}||_2)^2 + (||\mathbf{v}||_2)^2

\implies (\mathbf{u}+\mathbf{v})^\top(\mathbf{u}+\mathbf{v}) = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies (\mathbf{u}+\mathbf{v})^\top(\mathbf{u}+\mathbf{v}) = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies (\mathbf{u}^\top+\mathbf{v}^\top)(\mathbf{u}+\mathbf{v}) = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies (\mathbf{u}^\top+\mathbf{v}^\top)(\mathbf{u}+\mathbf{v}) = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies \mathbf{u}^\top\mathbf{u}+\mathbf{v}^\top\mathbf{u} + \mathbf{u}^\top\mathbf{v}+\mathbf{v}^\top\mathbf{v} = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies \mathbf{u}^\top\mathbf{u}+\mathbf{v}^\top\mathbf{u} + \mathbf{u}^\top\mathbf{v}+\mathbf{v}^\top\mathbf{v} = \mathbf{u}^\top\mathbf{u} + \mathbf{v}^\top\mathbf{v}

\implies \mathbf{u}^\top\mathbf{v} = 0

\implies \mathbf{u}^\top\mathbf{v} = 0

(the dot product of the two vectors will be 0)

\mathbf{u}

\mathbf{u}

\mathbf{v}

\mathbf{v}

\mathbf{u}+\mathbf{v}

\mathbf{u}+\mathbf{v}

Orthogonal subspaces

Two subspaces $S_1$ and $S_2$ are orthogonal if every vector $\mathbf{u} \in S_1$ is orthogonal to every vector in $\mathbf{v} \in S_2$

(switch to geogebra)

Orthogonal subspaces

\mathcal{C}(A)

\mathcal{C}(A)

\mathcal{N}(A^\top)

\mathcal{N}(A^\top)

inside~\mathbb{R}^m

inside~\mathbb{R}^m

\begin{bmatrix} \leftarrow \mathbf{a_1}^\top \rightarrow\\ \leftarrow \mathbf{a_2}^\top \rightarrow\\ \leftarrow \mathbf{a_3}^\top \rightarrow\\ \end{bmatrix}

\begin{bmatrix} \leftarrow \mathbf{a_1}^\top \rightarrow\\ \leftarrow \mathbf{a_2}^\top \rightarrow\\ \leftarrow \mathbf{a_3}^\top \rightarrow\\ \end{bmatrix}

\begin{bmatrix} \uparrow&\uparrow&\uparrow\\ \mathbf{a_1}&\mathbf{a_2}&\mathbf{a_3}\\ \downarrow&\downarrow&\downarrow\\ \end{bmatrix}

\begin{bmatrix} \uparrow&\uparrow&\uparrow\\ \mathbf{a_1}&\mathbf{a_2}&\mathbf{a_3}\\ \downarrow&\downarrow&\downarrow\\ \end{bmatrix}

A

A

A^\top

A^\top

\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}

\begin{bmatrix} x_1\\ x_2\\ x_3 \end{bmatrix}

\mathbf{x}

\mathbf{x}

= \begin{bmatrix} 0\\0\\0 \end{bmatrix}

= \begin{bmatrix} 0\\0\\0 \end{bmatrix}

\mathbf{x} \in \mathcal{N}(A^\top)

\mathbf{x} \in \mathcal{N}(A^\top)

\implies \mathbf{a_1}^\top\mathbf{x} = 0

\implies \mathbf{a_1}^\top\mathbf{x} = 0

\mathbf{a_2}^\top\mathbf{x} = 0

\mathbf{a_2}^\top\mathbf{x} = 0

\mathbf{a_3}^\top\mathbf{x} = 0

\mathbf{a_3}^\top\mathbf{x} = 0

\implies (p\mathbf{a_1}+q\mathbf{a_2}+r\mathbf{a_3})^\top\mathbf{x} = 0

\implies (p\mathbf{a_1}+q\mathbf{a_2}+r\mathbf{a_3})^\top\mathbf{x} = 0

\implies \mathcal{C}(A) \perp \mathcal{N}(A^\top)

\implies \mathcal{C}(A) \perp \mathcal{N}(A^\top)

The 4 fundamental subspaces

\mathcal{C}(A)

\mathcal{C}(A)

\mathcal{N}(A^\top)

\mathcal{N}(A^\top)

inside~\mathbb{R}^m

inside~\mathbb{R}^m

\mathcal{C}(A^\top)

\mathcal{C}(A^\top)

\mathcal{N}(A)

\mathcal{N}(A)

inside~\mathbb{R}^n

inside~\mathbb{R}^n

dim=r

dim=r

dim=n-r

dim=n-r

dim=r

dim=r

dim=m-r

dim=m-r

\mathcal{C}(A) \perp \mathcal{N}(A^\top)

\mathcal{C}(A) \perp \mathcal{N}(A^\top)

\mathcal{C}(A^\top) \perp \mathcal{N}(A)

\mathcal{C}(A^\top) \perp \mathcal{N}(A)

orthogonal complements

Example: $\mathcal{C}(A) \perp \mathcal{N}(A^\top)$

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ \end{bmatrix}

\begin{bmatrix} 1&1&2\\ 1&2&3\\ 1&3&4\\ \end{bmatrix}

A

A

\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&0\\ \end{bmatrix}

\begin{bmatrix} 1&1&2\\ 0&1&1\\ 0&0&0\\ \end{bmatrix}

U

U

Basis(\mathcal{C}(A))

Basis(\mathcal{C}(A))

=\begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix}

=\begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix}

,\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix}

,\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix}

Basis(\mathcal{N}(A^\top))

Basis(\mathcal{N}(A^\top))

=\begin{bmatrix} 1\\ 1\\ -1\\ \end{bmatrix}

=\begin{bmatrix} 1\\ 1\\ -1\\ \end{bmatrix}

\begin{bmatrix} 1&1&1\\ 1&2&3\\ 2&3&4\\ \end{bmatrix}

\begin{bmatrix} 1&1&1\\ 1&2&3\\ 2&3&4\\ \end{bmatrix}

A^\top

A^\top

\begin{bmatrix} 1&1&1\\ 0&1&2\\ 0&0&0\\ \end{bmatrix}

\begin{bmatrix} 1&1&1\\ 0&1&2\\ 0&0&0\\ \end{bmatrix}

A^\top\mathbf{x} = \mathbf{0}

A^\top\mathbf{x} = \mathbf{0}

\mathbf{x}=\begin{bmatrix} 1\\ -2\\ 1\\ \end{bmatrix}

\mathbf{x}=\begin{bmatrix} 1\\ -2\\ 1\\ \end{bmatrix}

special solution

(switch to geogebra)

Practice Problems

For each of the above matrices, verify whether the null space of $A$ is orthogonal to the column space of $A^\top$

\begin{bmatrix} 1&2&3\\ 1&1&2\\ 2&1&0 \end{bmatrix}

\begin{bmatrix} 1&2&3\\ 1&1&2\\ 2&1&0 \end{bmatrix}

\begin{bmatrix} 1&2&3&3&2&1\\ 1&2&2&1&1&2\\ 2&0&0&1&2&2 \end{bmatrix}

\begin{bmatrix} 1&2&3&3&2&1\\ 1&2&2&1&1&2\\ 2&0&0&1&2&2 \end{bmatrix}

\begin{bmatrix} 0&2&2\\ 1&2&1\\ 2&1&3\\ 3&1&1\\ -1&1&-2\\ 2&1&0\\ \end{bmatrix}

\begin{bmatrix} 0&2&2\\ 1&2&1\\ 2&1&3\\ 3&1&1\\ -1&1&-2\\ 2&1&0\\ \end{bmatrix}

\begin{bmatrix} 1&2&3&3&2&1\\ 2&1&0&1&2&2\\ 4&5&6&7&6&4 \end{bmatrix}

CS6015: Linear Algebra and Random Processes

Lecture 11: A tiny bit of ML, vector norms, orthogonal vectors, orthogonal subspaces

CS6015: Lecture 11

More from Mitesh Khapra