CS6015: Linear Algebra and Random Processes
Lecture 1: Logistics, Syllabus, Introduction to Vectors and Matrices
Logistics
Course Instructor: Mitesh M. Khapra
Lecture Hours : Slot D - Mon 11 - 12, Tue 10 - 11, Wed 9 - 10, Thu 12 - 1
TA1
TA2
TA3
TA4
TA5
Website for constant updates and TA details!!
TA6
TA7
TA8
TA9
TA10
TA11
TA12
Evaluation Pattern
Quiz (6 Quizzes: n-1) : 70%, End-Sem: 30%
Syllabus
Upto lecture 39: http://cse.iitm.ac.in/~miteshk/CS6015_2020.html
Clarification: point estimation, interval estimation, hypothesis testing will not be taught in this course (no statistics!)
आरम्भः
Let's Begin
Learning Objectives
What is a linear transformation?
What is a linear equation?
What is a linear combination?
What is at the heart of this course?
What is a vector?
What is a matrix?
What happens when a matrix meets a vector?
(for today's lecture)
4x = 12
x + y = 3
xy = 2
x^2 - 3x + 2 = 0
To the absolute number multiplied by four times the [coefficient of the] square, add the square of the [coefficient of the] middle term; the square root of the same, less the [coefficient of the] middle term, being divided by twice the [coefficient of the] square is the value.
\frac{-b \pm \sqrt{b^2 - 4ac}}{2a}
Brahmagupta (628 AD)
Linear Algebra
Algebra is the study of mathematical symbols and rules for manipulating these symbols
Linear Algebra
Linear transformations
T(\mathbf{x} + \mathbf{y}) = T(\mathbf{x}) + T (\mathbf{y})
T(a\mathbf{x}) = aT(\mathbf{x})
\mathbf{x}, \mathbf{y} \in \mathbb{R}^n, a \in \mathbb{R}
(switch to geogebra)
\mathbf{x} = \{x_1, x_2, x_3\}
\mathbf{y} = \{y_1, y_2, y_3\}
\mathbf{x}, \mathbf{y} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
Linear Algebra
Linear transformations
f(x) =3 \mathbb{x}
f(x_1) = 3x_1
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
f: \mathbb{R} \rightarrow \mathbb{R}
f(x_2) = 3x_2
f(x_1 + x_2) = 3(x_1 + x_2)
= 3(x_1) + 3(x_2)
= f(x_1) + f(x_2)
f(ax_1) = 3(ax_1)
= a(3x_1)
= af(x_1)
x_1, x_2 \in \mathbb{R}
Linear Algebra
Linear transformations
f(\mathbf{x}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3
\end{bmatrix}
f: \mathbb{R^3} \rightarrow \mathbb{R^3}
\mathbf{x}, \mathbf{y} \in \mathbb{R}^3
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix}
\mathbf{y} =\begin{bmatrix}
y_1 \\ y_2 \\ y_3
\end{bmatrix}
\mathbf{x} + \mathbf{y} =\begin{bmatrix}
x_1 + y_1 \\ x_2 + y_2 \\ x_3 + y_3
\end{bmatrix}
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
a\mathbf{x} =\begin{bmatrix}
ax_1 \\ ax_2 \\ ax_3
\end{bmatrix}
Linear Algebra
Linear transformations
f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3
\end{bmatrix}
f(\mathbf{y}) =\begin{bmatrix}
3y_1 - y_2 \\ 3y_3 \\ y_2 - 2 y_3
\end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix}
3(x_1 + y_1) - (x_2 + y_2) \\ 3 (x_3 + y_3) \\ (x_2 + y_2) - 2 (x_3 + y_3)
\end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix}
(3x_1 - x_2) + (3y_1 - y_2) \\ 3x_3 + 3y_3 \\ (x_2 - 2x_3) + (y_2 - 2y_3)
\end{bmatrix}
f(\mathbf{x} + \mathbf{y}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3
\end{bmatrix}
+\begin{bmatrix}
3y_1 - y_2 \\ 3y_3 \\ y_2 - 2 y_3
\end{bmatrix}
= f(\mathbf{x}) + f(\mathbf{y})
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix}
\mathbf{y} =\begin{bmatrix}
y_1 \\ y_2 \\ y_3
\end{bmatrix}\in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
Linear Algebra
Linear transformations
f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3
\end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix}
3(ax_1) - ax_2 \\ 3(ax_3) \\ ax_2 - 2(ax_3)
\end{bmatrix}
= af(\mathbf{x})
f(a\mathbf{x}) =a \begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2x_3
\end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix}
a(3x_1 - x_2) \\ a(3x_3) \\ a(x_2 - 2x_3)
\end{bmatrix}
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
Linear Algebra
Linear transformations
f: \mathbb{R^3} \rightarrow \mathbb{R^3}
f(\mathbf{x}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_1x_3 \\ x_2 - 2 x_3
\end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix}
3(ax_1) - ax_2 \\ 3(ax_1)(ax_3) \\ ax_2 - 2(ax_3)
\end{bmatrix}
\neq af(\mathbf{x})
f(a\mathbf{x}) \neq a \begin{bmatrix}
3x_1 - x_2 \\ 3x_1x_3 \\ x_2 - 2x_3
\end{bmatrix}
f(a\mathbf{x}) =\begin{bmatrix}
a(3x_1 - x_2) \\ a^2(3x_1x_3) \\ a(x_2 - 2x_3)
\end{bmatrix}
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix} \in \mathbb{R}^3
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
Linear Algebra
Linear transformations
f: \mathbb{R} \rightarrow \mathbb{R}
f(x) = 3x + 2
f(ax) = 3(ax) + 2
\neq af(x)
x \in \mathbb{R}
T: \mathbb{R}^n \rightarrow \mathbb{R}^m
f(ax) = 3ax + 2
f(ax) \neq a(3x + 2)
f(0\cdot x) \neq 0\cdot f(x)
Algebraically: Degree 1 polynomials
g: \mathbb{R^3} \rightarrow \mathbb{R^3}
g(\mathbf{x}) =\begin{bmatrix}
3x_1 - x_2 \\ 3x_3 \\ x_2 - 2 x_3
\end{bmatrix}
Linear Algebra
Linear equations
f(x) = 3x + 2
y = 3x + 2
3x - y = -2
x + 3y = 2
System of linear equations
3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2
Geometrically: Flat surfaces
(switch to geogebra)
Notes: draw few lines, planes, curves, explain the notion of flatness
Linear Algebra
Linear equations
3x - y = -2
x + 3y = 2
System of linear equations
3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2
How is the geometry connected to the algebra?
Collection of all points which satisfy the given algebraic equation
Linear Algebra
Linear equations
f(x) = 3x + 2
y = 3x + 2
3x - y = -2
x + 3y = 2
System of linear equations
3x + y - z = -2
x + 3y + z = 2
2x - 3y + z = 4
3x_1 + x_2 - x_3 + x_4 = -2
3x_2 + x_4 = 2
2x_1 - 3x_2 + 4x_3 = 4
x_1 + x_3 = 2
A (major) goal of this course
How many solutions exist?
Find the solutions?
(generalize to n variables)
Linear Algebra
Linear combinations
\mathbf{x} \in \mathbb{R}^n
\mathbf{y} \in \mathbb{R}^n
a\mathbf{x} + b\mathbf{y}
a \in \mathbb{R}
b \in \mathbb{R}
T(\mathbf{x} + \mathbf{y}) = T(\mathbf{x}) + T (\mathbf{y})
T(a\mathbf{x}) = aT(\mathbf{x})
T(a\mathbf{x} + b\mathbf{y}) = aT(\mathbf{x}) + bT (\mathbf{y})
\equiv
(will show up in HW1)
Linear Algebra
Linear combinations
\mathbf{x} \in \mathbb{R}^n
\mathbf{y} \in \mathbb{R}^n
a\mathbf{x} + b\mathbf{y}
a \in \mathbb{R}
b \in \mathbb{R}
Geometrically: If you take all linear combinations of n vectors you will get a flat surface
Algebraically: Degree 1 polynomials
(I will get to the geometric view soon)
Linear Algebra
Linear transformations
Linear equations
Linear combinations
Linear independence
(sorry for sneaking this in)
Linear Algebra
Linear equations
Linear transformation
(the gist of this course)
A\mathbf{x} = \mathbf{b}
A\mathbf{x} = \mathbf{0}
A\mathbf{x} = \lambda\mathbf{x}
matrix
vector
vector
matrix
vector
vector
matrix
vector
vector
scalar
Linear equations
(why? we will see soon)
(why? we will see soon)
Linear combinations
(will show up when we learn about this transformation)
Linear independence
(will show up while solving these equations)
Linear Algebra
Linear equations
(the heart of this course)
A\mathbf{x}
matrix
vector
Linear equations
We are now ready to dive into the course!
What is a vector?
Linear equations
Just a point in an n-dimensional space
(switch to geogebra)
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix} \in \mathbb{R}^3
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3 \\x_4 \\x_5
\end{bmatrix} \in \mathbb{R}^5
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3 \\\cdots \\x_n
\end{bmatrix} \in \mathbb{R}^n
Algebraic view
Geometric view
Note: draw a few vectors in 2d and 3d, talk about magnitude and direction
What do you do with vectors?
Linear equations
\mathbf{x} =\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix}
(in this course)
\mathbf{y} =\begin{bmatrix}
y_1 \\ y_2 \\ y _3
\end{bmatrix} \in \mathbb{R}^3
Add
\mathbf{x} + \mathbf{y} =\begin{bmatrix}
x_1 + y_1 \\ x_2 + y_2 \\ x_3 + y_3
\end{bmatrix} \in \mathbb{R}^3
Scale
a\mathbf{x} =\begin{bmatrix}
ax_1 \\ ax_2 \\ ax_3
\end{bmatrix}
=a\begin{bmatrix}
x_1 \\ x_2 \\ x_3
\end{bmatrix} \in \mathbb{R}^3
Linear combinations
a\mathbf{x} + b\mathbf{y}=\begin{bmatrix}
ax_1 + by_1\\ ax_2 + by_2\\ ax_3 + by_3
\end{bmatrix}
(switch to geogebra)
Linear equations
Notes: - show how it spans a plane, - 2d - 2 vectors, v, 1.5v, 2v, then -v, -1.5v, -2v, then scale u also - 2d - show a case when it will not span the space - 3d - 3 vectors, show case when it will not span the space
\mathbf{x} =\begin{bmatrix}
a \\ b
\end{bmatrix}
=b\begin{bmatrix}
0 \\ 1
\end{bmatrix}
+a\begin{bmatrix}
1 \\ 0
\end{bmatrix}
\mathbf{u} =\begin{bmatrix}
3 \\ 2
\end{bmatrix}
\mathbf{v} =\begin{bmatrix}
1 \\ 4
\end{bmatrix}
\begin{bmatrix}
0 \\ 1
\end{bmatrix}
= \frac{-1}{10} (\mathbf{u} - 3\mathbf{v})
\begin{bmatrix}
1 \\ 0
\end{bmatrix}
= \frac{1}{5} (2\mathbf{u} - \mathbf{v})
= p\mathbf{u} + q\mathbf{v}
= r\mathbf{u} + s\mathbf{v}
=(bp)\mathbf{u} + (bq)\mathbf{v}
+(ar)\mathbf{u} + (as)\mathbf{v}
=(bp + ar)\mathbf{u} + (bq + as)\mathbf{v}
What is a matrix?
Linear equations
Collection of row vectors
A =\begin{bmatrix}
a_{11} & a_{12} & a_{13} \\
a_{21} & a_{22} & a_{23} \\
a_{31} & a_{32} & a_{33} \\
\end{bmatrix} \in \mathbb{R}^{3 \times 3}
Collection of column vectors
or
A =\begin{bmatrix}
a_{11} & a_{12} & \cdots & a_{1n} \\
a_{21} & a_{22} & \cdots & a_{2n} \\
\cdots & \cdots & \cdots & \cdots \\
\cdots & \cdots & \cdots & \cdots \\
a_{m1} & a_{m2} & \cdots & a_{mn} \\
\end{bmatrix} \in \mathbb{R}^{m \times n}
m need not be equal to n
square matrices
m = n
rectangular matrices
m \neq n
When a matrix meets a vector!
Linear equations
\underbrace{A}_{\mathbb{R}^{m \times n}}\underbrace{\mathbf{x}}_{\mathbb{R}^n} = \underbrace{\mathbf{y}}_{\mathbb{R}^m}
f: \mathbb{R}^n \rightarrow \mathbb{R}^m
When a matrix meets a vector!
Linear equations
A =\begin{bmatrix}
1 & 2 & 0 \\
2 & 1 & 2\\
0& 2 & 1\\
\end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix} \in \mathbb{R}^{3}
= \begin{bmatrix}
~~~~\\
~~~~\\
~~~~\\
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
\end{matrix}
5
10
7
y_i = \sum_{j=1}^n A_{ij} x_j
y_1 = A_{11}x_1 + A_{12}x_2 + A_{13}x_3
= \sum_{j=1}^n A_{1j} x_j
y_2 = A_{21}x_1 + A_{22}x_2 + A_{23}x_3
= \sum_{j=1}^n A_{2j} x_j
y_3 = A_{31}x_1 + A_{32}x_2 + A_{33}x_3
= \sum_{j=1}^n A_{3j} x_j
\begin{bmatrix}
1\\
2\\
0\\
\end{bmatrix}
\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix}
\begin{bmatrix}
2\\
1\\
2\\
\end{bmatrix}
\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
\end{matrix}
\begin{bmatrix}
0\\
2\\
1\\
\end{bmatrix}
\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
\end{matrix}
When a matrix meets a vector!
Linear equations
A =\begin{bmatrix}
1 & 2 & 2 & 0\\
2 & 1 & 2 & 1\\
0& 2 & 1 &2\\
\end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
4\\
\end{bmatrix} \in \mathbb{R}^{4}
= \begin{bmatrix}
~~~~\\
~~~~\\
~~~~\\
\end{bmatrix}
11
14
15
y_i = \sum_{j=1}^n A_{ij} x_j
y_1
= \sum_{j=1}^n A_{1j} x_j
y_2
= \sum_{j=1}^n A_{2j} x_j
y_3
= \sum_{j=1}^n A_{3j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}x_4
= A_{21}x_1 + A_{22}x_2 + A_{23}x_3 + A_{24}x_4
= A_{31}x_1 + A_{32}x_2 + A_{33}x_3 + A_{34}x_4
\begin{bmatrix}
1\\
2\\
3\\
4\\
\end{bmatrix}
\begin{bmatrix}
1\\
2\\
2\\
0\\
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
+\\
\end{matrix}
\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
\begin{bmatrix}
2\\
1\\
2\\
1
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
\end{matrix}
\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix}
\begin{bmatrix}
0\\
2\\
1\\
2
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
\end{matrix}
When a matrix meets a vector!
Linear equations
A =\begin{bmatrix}
1 & 2 & 2 & 0\\
2 & 1 & 2 & 1\\
0& 2 & 1 &2\\
\end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix} \in \mathbb{R}^{3}
y_1
= \sum_{j=1}^n A_{1j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}?
The number of columns should be the same as the number of rows in the vector to perform
A\mathbf{x}
\begin{bmatrix}
1\\
2\\
3\\
~\\
\end{bmatrix}
\begin{bmatrix}
1\\
2\\
2\\
0\\
\end{bmatrix}
\begin{matrix}
*\\
*\\
*\\
*\\
\end{matrix}
\begin{matrix}
~\\
+\\
+\\
+\\
\end{matrix}
When a matrix meets a vector!
Linear equations
A =\begin{bmatrix}
1 & 2 & 2 & 0\\
2 & 1 & 2 & 1\\
0& 2 & 1 &2\\
\end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix} \in \mathbb{R}^{4}
y_1
= \sum_{j=1}^n A_{1j} x_j
= A_{11}x_1 + A_{12}x_2 + A_{13}x_3 + A_{14}x_4
\begin{bmatrix}
1\\
2\\
0\\
\end{bmatrix}
+
1
\begin{bmatrix}
2\\
1\\
2\\
\end{bmatrix}
+
2
\begin{bmatrix}
2\\
2\\
1\\
\end{bmatrix}
+
3
\begin{bmatrix}
0\\
1\\
2\\
\end{bmatrix}
4
= \begin{bmatrix}
11\\
14\\
15\\
\end{bmatrix}
y_2
= \sum_{j=1}^n A_{2j} x_j
y_3
= \sum_{j=1}^n A_{3j} x_j
= A_{21}x_1 + A_{22}x_2 + A_{23}x_3 + A_{24}x_4
= A_{31}x_1 + A_{32}x_2 + A_{33}x_3 + A_{34}x_4
When a matrix meets a vector!
Linear equations
A =\begin{bmatrix}
1 & 2 & 2 & 0\\
2 & 1 & 2 & 1\\
0& 2 & 1 &2\\
\end{bmatrix} \in \mathbb{R}^{3 \times 4}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
4
\end{bmatrix} \in \mathbb{R}^{4}
= \begin{bmatrix}
11\\
14\\
15\\
\end{bmatrix}
\mathbf{a}_1
\mathbf{a}_2
\mathbf{a}_3
\mathbf{a}_4
x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + x_3 \mathbf{a}_3 + x_4 \mathbf{a}_4
\mathbf{y} =
The product is a linear combination of the columns of A with each column scaled by the corresponding entry in x
A\mathbf{x}
(the algebraic view)
\begin{bmatrix}
1\\
2\\
0\\
\end{bmatrix}
1
+
\begin{bmatrix}
2\\
1\\
2\\
\end{bmatrix}
2
+
\begin{bmatrix}
2\\
2\\
1\\
\end{bmatrix}
3
+
\begin{bmatrix}
0\\
1\\
2\\
\end{bmatrix}
4
When a matrix meets a vector!
Linear equations
(the geometric view)
A =\begin{bmatrix}
1 & 2 & 0 \\
2 & 1 & 2\\
0& 2 & 1\\
\end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix} \in \mathbb{R}^{3}
A\mathbf{x}= \begin{bmatrix}
5\\
10\\
7\\
\end{bmatrix}
(switch to geogebra)
A =\begin{bmatrix}
1 & 2 \\
2 & 1\\
0& 2 \\
\end{bmatrix} \in \mathbb{R}^{3 \times 2}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
\end{bmatrix} \in \mathbb{R}^{2}
A\mathbf{x} =\begin{bmatrix}
5\\
4\\
4
\end{bmatrix} \in \mathbb{R}^{3}
A =\begin{bmatrix}
1 & 2 & 0 \\
2 & 1 & 2\\
0& 2 & 1\\
\end{bmatrix} \in \mathbb{R}^{3 \times 3}
\mathbf{x} =\begin{bmatrix}
1\\
2\\
3\\
\end{bmatrix} \in \mathbb{R}^{3}
A\mathbf{x}= \begin{bmatrix}
5\\
10\\
7\\
\end{bmatrix}