# Day 16:

Inner products and orthogonality

### Inner products

Definition. Given a vector space $$V$$, a function $$\langle\cdot,\cdot\rangle:V\times V\to\mathbb{R}$$  is called an inner product if given any $$u,v,w\in V$$ and $$\alpha\in\mathbb{R}$$ we have

1. $$\langle u,v\rangle = \langle v,u\rangle$$
2. $$\langle \alpha u,v\rangle = \alpha \langle u,v\rangle$$
3. $$\langle u+v,w\rangle = \langle u,w\rangle + \langle v,w\rangle$$
4. $$\langle u,u\rangle\geq 0$$
5. If $$\langle u,u\rangle = 0$$, then $$u=0$$.

If a space $$V$$ has an inner product, then $$V$$ is called an inner product space.

These properties have names:

• Property 1 is called symmetry of the inner product
• Properties 2 and 3 mean that the inner product is linear in the first argument
• A function satisfying Property 4 is called positive semi-definite.
• A function satisfying Property 5 is called definite.

### Dot Product

Given two vectors

$u = \begin{bmatrix}u_{1}\\ u_{2}\\ \vdots\\ u_{d}\end{bmatrix}\quad\text{and}\quad v = \begin{bmatrix}v_{1}\\ v_{2}\\ \vdots\\ v_{d}\end{bmatrix}$

in $$\R^{d}$$  their dot product is given by

$u\cdot v=u^{\top}v = \begin{bmatrix}u_{1} & u_{2} & \cdots & u_{d}\end{bmatrix}\begin{bmatrix}v_{1}\\ v_{2}\\ \vdots\\ v_{d}\end{bmatrix} = \sum_{i=1}^{d}u_{i}v_{i}$

The dot product is an inner product on $$\mathbb{R}^{d}$$, and hence $$\mathbb{R}^{d}$$ with the dot product is an inner product space.

Unless explicitly stated otherwise, this is the inner product on $$\mathbb{R}^{d}$$.

### An inner product on $$\mathbb{P}_{n}$$.

Given two vectors $$f(x),g(x)\in\mathbb{P}_{n}$$ set

$\langle f(x),g(x)\rangle = \int_{0}^{1}f(x)g(x)\, dx.$

This function is an inner product on $$\mathbb{P}_{n}$$.

This function is clearly symmetric. If $$f(x),g(x),h(x)\in\mathbb{P}_{n}$$, and $$a\in\mathbb{R}$$, then

$\langle f(x)+ag(x),h(x)\rangle = \int_{0}^{1}(f(x)+ag(x))h(x)\, dx$

$= \int_{0}^{1} f(x)h(x)+ag(x)h(x)\, dx = \int_{0}^{1}f(x)h(x)\, dx + a\int_{0}^{1}g(x)h(x)\,dx$

$=\langle f(x),h(x)\rangle + a\langle g(x),h(x)\rangle$

Hence, this function is linear in the first argument. Finally,

$\langle f(x),f(x)\rangle = \int_{0}^{1} [f(x)]^{2}\,dx$ Clearly, $$\langle f(x),f(x)\rangle\geq 0$$, with equality iff $$f(x)=0$$ for all $$x\in[0,1]$$.

### Other inner products

Example. For $$f(x),g(x)\in\mathbb{P}_{n}$$ and any two numbers $$a<b$$ we can define the inner product

$\langle f(x),g(x)\rangle = \int_{a}^{b}f(x)g(x)\, dx.$

Example. Given vectors $$x,y\in\mathbb{R}^{d}$$, and a diagonal matrix $$A\in\mathbb{R}^{d\times d}$$ with all positive entries on the diagonal, the function

$\langle x,y\rangle = x^{\top}Ay$

is an inner product in $$\mathbb{R}^{d}$$.

Example. For $$f(x),g(x)\in\mathbb{P}_{n}$$ and any set $$\{x_{1},x_{2},\ldots,x_{n},x_{n+1}\}\subset\mathbb{R}$$ with $$n+1$$ distinct elements, we can define the inner product

$\langle f(x),g(x)\rangle = \sum_{i=1}^{n+1}f(x_{i})g(x_{i}).$

### Orthogonality

Definition. We say that two vectors $$u$$ and $$v$$ are orthogonal if

$\langle u,v\rangle=0.$

Example. Consider the inner product space $$\mathbb{P}_{2}$$ with the inner product given on the previous slide. The vectors $$1$$ and $$x-\frac{1}{2}$$ are orthogonal, since

$\langle 1,x-\tfrac{1}{2}\rangle = \int_{0}^{1}x-\frac{1}{2}\, dx = 0.$

Example. The vectors $$x=[1\ \ -1]^{\top}$$ and $$y=[2\ \ 2]^{\top}$$ in $$\mathbb{R}^{2}$$ are orthogonal since

$x\cdot y = [1\ \ -1]\begin{bmatrix}2\\ 2\end{bmatrix} =(1)(2) + (-1)(2) = 0.$

### Orthogonality of subspaces

Definition. Let $$U$$ and $$W$$ be subspaces of the same inner product space $$V$$. We say that $$U$$ and $$W$$ are orthogonal if

$\langle u,w\rangle = 0\quad\text{for all }u\in U\text{ and }w\in W.$

Theorem. Assume $$A$$ is an $$m\times n$$ matrix. Then $$N(A)$$ and $$C(A^{\top})$$ are orthogonal subspaces.

Corollary. Assume $$A$$ is an $$m\times n$$ matrix. Then $$N(A^{\top})$$ and $$C(A)$$ are orthogonal subspaces.

Proof. By the definition of $$N(A)$$ and $$C(A^{\top})$$ we have $Ax=0\quad \text{and}\quad y=A^{\top}z\ \text{for some }z.$

Hence,

$y\cdot x = y^{\top}x=(A^{\top}z)^{\top}x=z^{\top}Ax=z^{\top}0=0.\ \Box$

Proof. Apply the theorem to $$A^{\top}$$. $$\Box$$

Theorem. Let $$V$$ be an inner product space. If $$S=\{x_{1},x_{2},\ldots,x_{k}\}\subset V$$ is a collection of nonzero, pairwise orthogonal vectors, then $$S$$ is independent.

Proof. Suppose there are scalars $$a_{1},a_{2},\ldots,a_{k}$$ such that

$\sum_{i=1}^{k}a_{i}x_{i} = 0.$

For $$j\in\{1,2,\ldots,k\}$$ we compute

$\left\langle \sum_{i=1}^{k}a_{i}x_{i},x_{j}\right\rangle = \sum_{i=1}^{k}a_{i}\langle x_{i},x_{j}\rangle = a_{j}\langle x_{j},x_{j}\rangle.$

Hence, for each $$j\in\{1,2,\ldots,k\}$$ we have

$a_{j}\langle x_{j},x_{j}\rangle = 0$

Since $$x_{j}\neq 0$$ we conclude that $$\langle x_{j},x_{j}\rangle >0$$, and hence $$a_{j} = 0$$ for each $$j\in\{1,2,\ldots,n\}$$. $$\Box$$

### Norms

Definition. Given an inner product space $$V$$ and a vector $$v\in V$$, then norm of $$v$$ is the number

$\|v\| = \sqrt{\langle v,v\rangle}.$

Example. Consider the inner product space $$\mathbb{P}_{2}$$ with the inner product given previously. If $$f(x) = x$$, then

$\|f(x)\|= \sqrt{\int_{0}^{1}x^2\, dx} = \sqrt{\frac{1}{3}}.$

Example. Let $$x=[1\ \ -2]^{\top}\in\mathbb{R}^{2}$$, then

$\|x\| = \sqrt{1^2+(-2)^2} = \sqrt{5}.$

By John Jasper

• 475