CS6015: Linear Algebra and Random Processes

Lecture 1: Logistics, Syllabus, Introduction to Vectors and Matrices

Logistics

Course Instructor: Mitesh M. Khapra

Lecture Hours : Slot D -  Mon 11 - 12, Tue 10 - 11, Wed 9 - 10, Thu 12 - 1 

TA1

TA2

TA3

TA4

TA5

Website for constant updates and TA details!!

TA6

TA7

TA8

TA9

TA10

TA11

TA12

Aman Kumar

CS20M010

Himani Shrotriya

CS20M024

Manideep Ladi

CS20M036

Pranay Sonagra

CS20M048

P Krishna

CS19D002

Tahir Javed

CS20D407

Priyanka Bedekar

CS20M050

Rigved Sah

CS20M053

Prachi Sahu

CS20M047

Sayan Chandra

CS20M057

Sumit Negi

CS20M067

Yash Madhani

CS20S002

Evaluation Pattern

Will discuss by the end of this week

Syllabus

Upto lecture 39: http://cse.iitm.ac.in/~miteshk/CS6015_2020.html

Clarification: point estimation, interval estimation, hypothesis testing will not be taught in this course (no statistics!)

आरम्भः

Let's Begin

Learning Objectives

What is a linear transformation?

What is a linear equation?

What is a linear combination?

What is at the heart of this course?

What is a vector?

What is a matrix?

What happens when a matrix meets a vector?

(for today's lecture)

4x = 12
x + y = 3
xy = 2
x^2 - 3x + 2 = 0

To the absolute number multiplied by four times the [coefficient of the] square, add the square of the [coefficient of the] middle term; the square root of the same, less the [coefficient of the] middle term, being divided by twice the [coefficient of the] square is the value.

\frac{-b \pm \sqrt{b^2 - 4ac}}{2a}

Brahmagupta (628 AD)

Linear Algebra

Algebra is the study of mathematical symbols and rules for manipulating these symbols

Linear Algebra

Linear transformations

T(\mathbf{x} + \mathbf{y}) = T(\mathbf{x}) + T (\mathbf{y})
T(a\mathbf{x}) = aT(\mathbf{x})
\mathbf{x}, \mathbf{y} \in \mathbb{R}^n, a \in \mathbb{R}

(switch to geogebra)

\mathbf{x} = \{x_1, x_2, x_3\}
\mathbf{y} = \{y_1, y_2, y_3\}
\mathbf{x}, \mathbf{y} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m

Linear Algebra

Linear transformations

f(x) =3 \mathbb{x}
f(x_1) = 3x_1
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
f: \mathbb{R} \rightarrow \mathbb{R}
f(x_2) = 3x_2
f(x_1 + x_2) = 3(x_1 + x_2)
= 3(x_1) + 3(x_2)
= f(x_1) + f(x_2)
f(ax_1) = 3(ax_1)
= a(3x_1)
= af(x_1)
x_1, x_2 \in \mathbb{R}

Linear Algebra

Linear transformations

f(\mathbf{x}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3 \end{bmatrix}
f: \mathbb{R^3} \rightarrow \mathbb{R^3}
\mathbf{x}, \mathbf{y} \in \mathbb{R}^3
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}
\mathbf{y} =\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix}
\mathbf{x} + \mathbf{y} =\begin{bmatrix} x_1 + y_1 \\ x_2 + y_2 \\ x_3 + y_3 \end{bmatrix}
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
a\mathbf{x} =\begin{bmatrix} ax_1 \\ ax_2 \\ ax_3 \end{bmatrix}

Linear Algebra

Linear transformations

f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3 \end{bmatrix}
f(\mathbf{y}) =\begin{bmatrix} 3y_1 - y_2 \\ 3y_3 \\ y_2 - 2 y_3 \end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix} 3(x_1 + y_1) - (x_2 + y_2) \\ 3 (x_3 + y_3) \\ (x_2 + y_2) - 2 (x_3 + y_3) \end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix} (3x_1 - x_2) + (3y_1 - y_2) \\ 3x_3 + 3y_3 \\ (x_2 - 2x_3) + (y_2 - 2y_3) \end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3 \end{bmatrix}
+\begin{bmatrix} 3y_1 - y_2 \\ 3y_3 \\ y_2 - 2 y_3 \end{bmatrix}
= f(\mathbf{x}) + f(\mathbf{y})
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}
\mathbf{y} =\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix}\in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m

Linear Algebra

Linear transformations

f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3 \end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix} 3(ax_1) - ax_2 \\ 3(ax_3) \\ ax_2 - 2(ax_3) \end{bmatrix}
= af(\mathbf{x})
f(a\mathbf{x}) =a \begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2x_3 \end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix} a(3x_1 - x_2) \\ a(3x_3) \\ a(x_2 - 2x_3) \end{bmatrix}
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m

Linear Algebra

Linear transformations

f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_1x_3 \\ x_2 - 2 x_3 \end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix} 3(ax_1) - ax_2 \\ 3(ax_1)(ax_3) \\ ax_2 - 2(ax_3) \end{bmatrix}
\neq af(\mathbf{x})
f(a\mathbf{x}) \neq a \begin{bmatrix} 3x_1 - x_2 \\ 3x_1x_3 \\ x_2 - 2x_3 \end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix} a(3x_1 - x_2) \\ a^2(3x_1x_3) \\ a(x_2 - 2x_3) \end{bmatrix}
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m

Linear Algebra

Linear transformations

f: \mathbb{R} \rightarrow \mathbb{R}
f(x) = 3x + 2
f(ax) = 3(ax) + 2
\neq af(x)
x \in \mathbb{R}
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
f(ax) = 3ax + 2
f(ax) \neq a(3x + 2)
f(0\cdot x) \neq 0\cdot f(x)

Algebraically: Degree 1 polynomials

g: \mathbb{R^3} \rightarrow \mathbb{R^3}
g(\mathbf{x}) =\begin{bmatrix} 3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3 \end{bmatrix}

Linear Algebra

Linear equations

f(x) = 3x + 2
y = 3x + 2
3x - y = -2
x + 3y = 2

System of linear equations

3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2

Geometrically: Flat surfaces

(switch to geogebra)

Notes: draw few lines, planes, curves, explain the notion of flatness

Linear Algebra

Linear equations

3x - y = -2
x + 3y = 2

System of linear equations

3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2

How is the geometry connected to the algebra?

Collection of all points which satisfy the given algebraic equation

Linear Algebra

Linear equations

f(x) = 3x + 2
y = 3x + 2
3x - y = -2
x + 3y = 2

System of linear equations

3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2

A (major) goal of this course

How many solutions exist?

Find the solutions?

(generalize to n variables)

Linear Algebra

Linear combinations

\mathbf{x} \in \mathbb{R}^n
\mathbf{y} \in \mathbb{R}^n
a\mathbf{x} + b\mathbf{y}
a \in \mathbb{R}
b \in \mathbb{R}
T(\mathbf{x} + \mathbf{y}) = T(\mathbf{x}) + T (\mathbf{y})
T(a\mathbf{x}) = aT(\mathbf{x})
T(a\mathbf{x} + b\mathbf{y}) = aT(\mathbf{x}) + bT (\mathbf{y})
\equiv

(will show up in HW1)

Linear Algebra

Linear combinations

\mathbf{x} \in \mathbb{R}^n
\mathbf{y} \in \mathbb{R}^n
a\mathbf{x} + b\mathbf{y}
a \in \mathbb{R}
b \in \mathbb{R}

Geometrically: If you take all linear combinations of n vectors you will get a flat surface

Algebraically: Degree 1 polynomials

(I will get to the geometric view soon)

Linear Algebra

Linear transformations

Linear equations

Linear combinations

Linear independence

(sorry for sneaking this in)

Linear Algebra

Linear equations

Linear transformation

(the gist of this course)

A\mathbf{x} = \mathbf{b}
A\mathbf{x} = \mathbf{0}
A\mathbf{x} = \lambda\mathbf{x}
matrix
vector
vector
matrix
vector
vector
matrix
vector
vector
scalar

Linear equations

(why? we will see soon)
(why? we will see soon)

Linear combinations

(will show up when we learn about this transformation)

Linear independence

(will show up while solving these equations)

Linear Algebra

Linear equations

(the heart of this course)

A\mathbf{x}
matrix
vector

Linear equations

We are now ready to dive into the course!

What is a vector?

Linear equations

Just a point in an n-dimensional space

(switch to geogebra)

\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} \in \mathbb{R}^3
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\x_4 \\x_5 \end{bmatrix} \in \mathbb{R}^5
\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \\\cdots \\x_n \end{bmatrix} \in \mathbb{R}^n

Algebraic view

Geometric view

Note: draw a few vectors in 2d and 3d, talk about magnitude and direction

What do you do with vectors?

Linear equations

\mathbf{x} =\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}

(in this course)

\mathbf{y} =\begin{bmatrix} y_1 \\ y_2 \\ y _3 \end{bmatrix} \in \mathbb{R}^3

Add

\mathbf{x} + \mathbf{y} =\begin{bmatrix} x_1 + y_1 \\ x_2 + y_2 \\ x_3 + y_3 \end{bmatrix} \in \mathbb{R}^3

Scale

a\mathbf{x} =\begin{bmatrix} ax_1 \\ ax_2 \\ ax_3 \end{bmatrix}
=a\begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} \in \mathbb{R}^3

Linear combinations

a\mathbf{x} + b\mathbf{y}=\begin{bmatrix} ax_1 + by_1\\ ax_2 + by_2\\ ax_3 + by_3 \end{bmatrix}

(switch to geogebra)

Linear equations

Notes: 
- show how it spans a plane, 
- 2d - 2 vectors, v, 1.5v, 2v, then -v, -1.5v, -2v, then scale u also
- 2d - show a case when it will not span the space
- 3d - 3 vectors, show case when it will not span the space
\mathbf{x} =\begin{bmatrix} a \\ b \end{bmatrix}
=b\begin{bmatrix} 0 \\ 1 \end{bmatrix}
+a\begin{bmatrix} 1 \\ 0 \end{bmatrix}
\mathbf{u} =\begin{bmatrix} 3 \\ 2 \end{bmatrix}
\mathbf{v} =\begin{bmatrix} 1 \\ 4 \end{bmatrix}
\begin{bmatrix} 0 \\ 1 \end{bmatrix}
= \frac{-1}{10} (\mathbf{u} - 3\mathbf{v})
\begin{bmatrix} 1 \\ 0 \end{bmatrix}
= \frac{1}{5} (2\mathbf{u} - \mathbf{v})
= p\mathbf{u} + q\mathbf{v}
= r\mathbf{u} + s\mathbf{v}
=(bp)\mathbf{u} + (bq)\mathbf{v}
+(ar)\mathbf{u} + (as)\mathbf{v}
=(bp + ar)\mathbf{u} + (bq + as)\mathbf{v}

What is a matrix?

Linear equations

Collection of row vectors

A =\begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \\ \end{bmatrix} \in \mathbb{R}^{3 \times 3}

Collection of column vectors

or

A =\begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \cdots & \cdots & \cdots & \cdots \\ \cdots & \cdots & \cdots & \cdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \\ \end{bmatrix} \in \mathbb{R}^{m \times n}

m need not be equal to n

square matrices

m = n

rectangular matrices

m \neq n

When a matrix meets a vector!

Linear equations

\underbrace{A}_{\mathbb{R}^{m \times n}}\underbrace{\mathbf{x}}_{\mathbb{R}^n} = \underbrace{\mathbf{y}}_{\mathbb{R}^m}
f: \mathbb{R}^n \rightarrow \mathbb{R}^m

When a matrix meets a vector!

Linear equations

A =\begin{bmatrix} 1 & 2 & 0 \\ 2 & 1 & 2\\ 0& 2 & 1\\ \end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix} \in \mathbb{R}^{3}
= \begin{bmatrix} ~~~~\\ ~~~~\\ ~~~~\\ \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ \end{matrix}
5
10
7
y_i = \sum_{j=1}^n A_{ij} x_j
y_1 = A_{11}x_1 + A_{12}x_2 + A_{13}x_3
= \sum_{j=1}^n A_{1j} x_j
y_2 = A_{21}x_1 + A_{22}x_2 + A_{23}x_3
= \sum_{j=1}^n A_{2j} x_j
y_3 = A_{31}x_1 + A_{32}x_2 + A_{33}x_3
= \sum_{j=1}^n A_{3j} x_j
\begin{bmatrix} 1\\ 2\\ 0\\ \end{bmatrix}
\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix}
\begin{bmatrix} 2\\ 1\\ 2\\ \end{bmatrix}
\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ \end{matrix}
\begin{bmatrix} 0\\ 2\\ 1\\ \end{bmatrix}
\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ \end{matrix}

When a matrix meets a vector!

Linear equations

A =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ 4\\ \end{bmatrix} \in \mathbb{R}^{4}
= \begin{bmatrix} ~~~~\\ ~~~~\\ ~~~~\\ \end{bmatrix}
11
14
15
y_i = \sum_{j=1}^n A_{ij} x_j
y_1
= \sum_{j=1}^n A_{1j} x_j
y_2
= \sum_{j=1}^n A_{2j} x_j
y_3
= \sum_{j=1}^n A_{3j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}x_4
= A_{21}x_1 + A_{22}x_2 + A_{23}x_3 + A_{24}x_4
= A_{31}x_1 + A_{32}x_2 + A_{33}x_3 + A_{34}x_4
\begin{bmatrix} 1\\ 2\\ 3\\ 4\\ \end{bmatrix}
\begin{bmatrix} 1\\ 2\\ 2\\ 0\\ \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ +\\ \end{matrix}
\begin{bmatrix} 1\\ 2\\ 3\\ 4 \end{bmatrix}
\begin{bmatrix} 2\\ 1\\ 2\\ 1 \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ \end{matrix}
\begin{bmatrix} 1\\ 2\\ 3\\ 4 \end{bmatrix}
\begin{bmatrix} 0\\ 2\\ 1\\ 2 \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ \end{matrix}

When a matrix meets a vector!

Linear equations

A =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix} \in \mathbb{R}^{3}
y_1
= \sum_{j=1}^n A_{1j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}?

The number of columns should be the same as the number of rows in the vector to perform

A\mathbf{x}
\begin{bmatrix} 1\\ 2\\ 3\\ ~\\ \end{bmatrix}
\begin{bmatrix} 1\\ 2\\ 2\\ 0\\ \end{bmatrix}
\begin{matrix} *\\ *\\ *\\ *\\ \end{matrix}
\begin{matrix} ~\\ +\\ +\\ +\\ \end{matrix}

When a matrix meets a vector!

Linear equations

A =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ 4 \end{bmatrix} \in \mathbb{R}^{4}
y_1
= \sum_{j=1}^n A_{1j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}x_4
\begin{bmatrix} 1\\ 2\\ 0\\ \end{bmatrix}
+
1
\begin{bmatrix} 2\\ 1\\ 2\\ \end{bmatrix}
+
2
\begin{bmatrix} 2\\ 2\\ 1\\ \end{bmatrix}
+
3
\begin{bmatrix} 0\\ 1\\ 2\\ \end{bmatrix}
4
= \begin{bmatrix} 11\\ 14\\ 15\\ \end{bmatrix}
y_2
= \sum_{j=1}^n A_{2j} x_j
y_3
= \sum_{j=1}^n A_{3j} x_j
= A_{21}x_1 + A_{22}x_2 + A_{23}x_3 + A_{24}x_4
= A_{31}x_1 + A_{32}x_2 + A_{33}x_3 + A_{34}x_4

When a matrix meets a vector!

Linear equations

A =\begin{bmatrix} 1 & 2 & 2 & 0\\ 2 & 1 & 2 & 1\\ 0& 2 & 1 &2\\ \end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ 4 \end{bmatrix} \in \mathbb{R}^{4}
= \begin{bmatrix} 11\\ 14\\ 15\\ \end{bmatrix}
\mathbf{a}_1
\mathbf{a}_2
\mathbf{a}_3
\mathbf{a}_4
x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + x_3 \mathbf{a}_3 + x_4 \mathbf{a}_4
\mathbf{y} =

The product          is a linear combination of the  columns of A with each column scaled by the corresponding entry in x

A\mathbf{x}

(the algebraic view)

\begin{bmatrix} 1\\ 2\\ 0\\ \end{bmatrix}
1
+
\begin{bmatrix} 2\\ 1\\ 2\\ \end{bmatrix}
2
+
\begin{bmatrix} 2\\ 2\\ 1\\ \end{bmatrix}
3
+
\begin{bmatrix} 0\\ 1\\ 2\\ \end{bmatrix}
4

When a matrix meets a vector!

Linear equations

(the geometric view)

A =\begin{bmatrix} 1 & 2 & 0 \\ 2 & 1 & 2\\ 0& 2 & 1\\ \end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix} \in \mathbb{R}^{3}
A\mathbf{x}= \begin{bmatrix} 5\\ 10\\ 7\\ \end{bmatrix}

(switch to geogebra)

A =\begin{bmatrix} 1 & 2 \\ 2 & 1\\ 0& 2 \\ \end{bmatrix} \in \mathbb{R}^{3 \times 2}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ \end{bmatrix} \in \mathbb{R}^{2}
A\mathbf{x} =\begin{bmatrix} 5\\ 4\\ 4 \end{bmatrix} \in \mathbb{R}^{3}
A =\begin{bmatrix} 1 & 2 & 0 \\ 2 & 1 & 2\\ 0& 2 & 1\\ \end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix} 1\\ 2\\ 3\\ \end{bmatrix} \in \mathbb{R}^{3}
A\mathbf{x}= \begin{bmatrix} 5\\ 10\\ 7\\ \end{bmatrix}

Learning Objectives

What is a linear transformation?

What is a linear equation?

What is a linear combination?

What is at the heart of this course?

What is a vector?

What is a matrix?

What happens when a matrix meets a vector?

(achieved)