CS6015: Linear Algebra and Random Processes

Lecture 17: Change of basis

Learning Objectives

How do you translate a point from one basis to another?

How do you translate a transformation from one basis to another?

The Eigenstory

real

imaginary

distinct

repeating

\(A^\top\)

\(A^{-1}\)

\(AB\)

\(A^\top A\)

(basis)

powers of A

PCA

optimisation

diagonalisation

\(A+B\)

\(U\)

\(R\)

\(A^2\)

\(A + kI\)

How to compute eigenvalues?

What are the possible values?

What are the eigenvalues of some special matrices ?

What is the relation between the eigenvalues of related matrices?

What do eigen values reveal about a matrix?

What are some applications in which eigenvalues play an important role?

Identity

Projection

Reflection

Markov

Rotation

Singular

Orthogonal

Rank one

Symmetric

Permutation

det(A - \lambda I) = 0

trace

determinant

invertibility

rank

nullspace

columnspace

(positive semidefinite matrices)

positive pivots

(independent eigenvectors)

(orthogonal eigenvectors)

... ...

(symmetric)

(where are we?)

(characteristic equation)

(desirable)

HW5

distinct values

independent eigenvectors

\(\implies\)

steady state

(Markov matrices)

... ... but before that a slight detour to understand "change of basis"

Basis \(\implies\) some coordinate system

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

Standard basis:

\mathbf{y}=\begin{bmatrix} 0\\1 \end{bmatrix}

\mathbf{x}=\begin{bmatrix} 1\\0 \end{bmatrix}

Every vector is expressed as a linear combination of these two vectors

\begin{bmatrix} a\\b \end{bmatrix}=a\mathbf{x} + b\mathbf{y}

\begin{bmatrix} 1\\1 \end{bmatrix}=1\cdot\mathbf{x} + 1\cdot\mathbf{y}

But we are free to choose any other basis

a, b are the coordinates in this coordinate system

\begin{bmatrix} 1\\1 \end{bmatrix}

Free to choose any basis

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

Alternative basis:

\mathbf{\hat{y}}=\begin{bmatrix} -1\\1 \end{bmatrix}

\mathbf{\hat{x}}=\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

Every vector is expressed as a linear combination of these two vectors

\begin{bmatrix} a\\b \end{bmatrix}=a\mathbf{\hat{x}} + b\mathbf{\hat{y}}

\begin{bmatrix} 1\\1 \end{bmatrix}\implies1\cdot\mathbf{\hat{x}} + 1\cdot\mathbf{\hat{y}}

a, b are the coordinates in this coordinate system

Same coordinates, different points

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

The vectors \(\begin{bmatrix}1\\1\end{bmatrix}\) are not the same in this two basis

\begin{bmatrix} 1\\1 \end{bmatrix}\implies1\cdot\mathbf{\hat{x}} + 1\cdot\mathbf{\hat{y}}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}\implies1\cdot\mathbf{x} + 1\cdot\mathbf{y}

=\begin{bmatrix} 1\\2 \end{bmatrix}

=\begin{bmatrix} 1\\1 \end{bmatrix}

Translating from one basis to another

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

Vector in new basis

\mathbf{\hat{a}} = 1\begin{bmatrix} 2\\1 \end{bmatrix} + 1\begin{bmatrix} -1\\1 \end{bmatrix}

\mathbf{\hat{a}}

\mathbf{\hat{a}} = \begin{bmatrix} 2&-1\\1&1 \end{bmatrix}\begin{bmatrix} 1\\1 \end{bmatrix}

new basis

Vector in std. basis

(to the standard basis)

Translating form one basis to another

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

=\begin{bmatrix} 1\\1 \end{bmatrix}

A=\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}

This is the matrix which took us from our new basis to the standard basis

(the other direction )

\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}\mathbf{\hat{a}}

=\mathbf{a}

Vector in new basis

Vector in std basis

Suppose you are given a vector in the std. basis then how do you represent it in the new basis?

\mathbf{\hat{a}}=\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}^{-1}

\mathbf{a}

Translating form one basis to another

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

=\begin{bmatrix} 1\\1 \end{bmatrix}

This is the matrix which took us from our standard basis to the new basis

(the other direction )

\mathbf{\hat{a}}=\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}^{-1}

\begin{bmatrix} 1\\1 \end{bmatrix}

\mathbf{\hat{a}}=\frac{1}{3}\begin{bmatrix} 1&1\\-1&2 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

=\begin{bmatrix} \frac{2}{3}\\~\\\frac{1}{3} \end{bmatrix}

A=\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

\frac{2}{3}\begin{bmatrix} 2\\1 \end{bmatrix}

+~\frac{1}{3}\begin{bmatrix} -1\\1 \end{bmatrix}

=\begin{bmatrix} 1\\1 \end{bmatrix}

\frac{2}{3}

\frac{1}{3}

Translating transformations

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

This transformation rotates a vector in the standard basis by 90 degrees

A=\begin{bmatrix} 0&-1\\1&0 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

What would the corresponding matrix be for the new basis?

(i.e., if v is a vector in the new basis, what is the matrix that will rotate it by 90 degrees)

\begin{bmatrix} -1\\1 \end{bmatrix}

\mathbf{v} = \begin{bmatrix}a\\b\end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

Translating transformations

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\mathbf{v}

\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

A vector in the new basis

\underbrace{~~~~~~~~~~~~~~~~~}

the same vector represented in the standard basis

\begin{bmatrix} 0&-1\\1&0 \end{bmatrix}

\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}

the desired transformation applied in the standard basis

\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}^{-1}

\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}

The result translated back to the new basis

\mathbf{v} = \begin{bmatrix}a\\b\end{bmatrix}

Translating transformations

\begin{bmatrix} 1\\0 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

\begin{bmatrix} 2&-1\\1&1 \end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 2\\1 \end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

A vector in the new basis

\underbrace{~~~~~~~~~~~~~~~~~}

the same vector represented in the standard basis

\begin{bmatrix} 0&-1\\1&0 \end{bmatrix}

\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}

the desired transformation applied in the standard basis

\begin{bmatrix} 1&1\\-1&2 \end{bmatrix}

\underbrace{~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~}

The result translated back to the new basis

\mathbf{v} = \begin{bmatrix}\frac{2}{3}\\\frac{1}{3}\end{bmatrix}

\frac{1}{3}

\begin{bmatrix}\frac{2}{3}\\\frac{1}{3}\end{bmatrix}

\begin{bmatrix} 1\\1 \end{bmatrix}

\begin{bmatrix}\frac{2}{3}\\\frac{1}{3}\end{bmatrix}

\begin{bmatrix} -1\\1 \end{bmatrix}

\begin{bmatrix} 0\\1 \end{bmatrix}

Translating transformations

B^{-1}AB\mathbf{x}

New Basis

Desired transformation

Vector in new basis

(we will see this form soon)

Puzzle: If \(A\) is the transformation that we are interested in then in what situation would we prefer the basis \(B\) as opposed to the standard basis?

Hint: What if \(B^{-1}AB\) was "simpler" than \(A\) ?

(of course, you need to think what simpler means - that's a part of the puzzle!)

Learning Objectives (achieved)