CS6015: Linear Algebra and Random Processes
Lecture 33: Expectation, Variance and their properties, Computing expectation and variance of some known distributions
Learning Objectives
What is expectation?
What is variance?
What are some of their properties?
How do you compute expectation and variance of some standard distributions?
Expectation
00, 0, 1, 2,3, ..., 36
Does gambling pay off?
Standard pay off - 35:1 if the ball lands on your number
If you play this game a 1000 times how much do you expect to win on average ?
00
0
1
2
3
...
36
Does gambling pay off?
X: profit
\Omega
X
-1
35
p_X(35) =
\frac{1}{38} = 0.026
p_X(-1) = 1 - \frac{1}{38} = 0.974
Does gambling pay off?
P(win) = \frac{\#wins}{\#games}
\frac{1}{38} = 0.026
p_X(-1) = 1 - \frac{1}{38} = 0.974
If you play this game a 1000 times how much do you expect to win on average ?
p_X(34) =
0.026 = \frac{\#wins}{1000}
Avg. gain = ~~~~~~~(26*35 + 974*(-1))
\#wins = 26
\frac{1}{1000}
= -0.064
(Stop gambling!!)
Expectation: the formula
E[X]
E[X] = \frac{1}{1000}(26*34 + 974*(-1))
E[X] = \frac{26}{1000} * 34 + \frac{974}{1000} * (-1)
E[X] = \sum_{x\in\{-1, 34\}}x*p_X(x)
E[X] = p_X(34)*34 + p_X(-1)*(-1)
E[X] = 0.026*34 + 0.974*(-1)
Expectation: the formula
E[X]
E[X] = \sum_{x\in\mathbb{R}_X}x*p_X(x)
The expected value or expectation of a discrete random variable X whose possible values are
x_1, x_2, \dots, x_n
is denoted by and computed as
E[X]
E[X] = \sum_{i=1}^n x_i P(X = x_i) = \sum_{i=1}^n x_i*p_X(x_i)
Expectation: Insurance
A person buys a car theft insurance policy of INR 200000 at an annual premium of INR 6000. There is a 2% chance that the car may get stolen.
What is the expected gain of the insurance company at the end of 1 year?
X: profit
X: \{6000, -194000\}
p_X(6000) = 0.98
p_X(-194000) = 0.02
E[X] = \sum_{x \in \mathbb{R}_x} x*p_X(x)
\therefore E[X] = 0.98*6000 + 0.02 * (-194000)
= 2000
Expectation: Insurance
A person buys a car theft insurance policy of INR 200000. Suppose there is a 10% chance that the car may get stolen
What should the premium be so that the expected gain in still INR 2000?
X: profit
X: \{x, -(200000 - x) \}
p_X(x) = 0.90
p_X(x - 200000) = 0.10
E[X] = \sum_{x \in \mathbb{R}_x} x*p_X(x)
\therefore E[X] = 0.9*x + 0.1 * (x - 200000)
\therefore 2000 = 0.9*x + 0.1 * (x - 200000)
\therefore x = 22000
X: \{x, x - 200000 \}
Function of a Random Variable
(1,1)
(1,2)~(2,1)
(1,3)~(2,2)~(3,1)
(1,4)~(2,3)~(3,2)~(4,1)
(1,5)~(2,4)~(3,3)~(4,2)~(5,1)
(1,6)~(2,5)~(3,4)~(4,3)~(5,2)~(6,1)
(2,6)~(3,5)~(4,4)~(5,3)~(6,2)
(3,6)~(4,5)~(5,4)~(6,3)
(4,6)~(5,5)~(6,4)
(5,6)~(6,5)
(6,6)
\Omega
2
3
4
5
6
7
8
9
10
11
12
Y = g(X)
X
Y
1
2
3
E[Y] = ?
Y = \begin{cases}
1~~if~~x < 5 \\
2~~if~~5 \leq x \leq 8 \\
3~~if~~x > 8 \\
\end{cases}
Y = g(X)
Y = \begin{cases}
1~~if~~x < 5 \\
2~~if~~5 \leq x \leq 8 \\
3~~if~~x > 8 \\
\end{cases}
E[Y] = 1*p_Y(1) + 2*p_Y(2) + 3*p_Y(3)
p_Y(1) = \frac{1}{36} + \frac{2}{36} + \frac{3}{36} = \frac{6}{36}
p_Y(2) = \frac{4}{36} + \frac{5}{36} + \frac{6}{36} + \frac{5}{36} = \frac{20}{36}
p_Y(3) = \frac{4}{36} + \frac{3}{36} + \frac{2}{36} + \frac{1}{36} = \frac{10}{36}
\therefore E[Y] = 1*(\frac{1}{36} + \frac{2}{36} + \frac{3}{36})
+ 2*(\frac{4}{36} + \frac{5}{36} + \frac{6}{36} + \frac{5}{36})
+3*(\frac{4}{36} + \frac{3}{36} + \frac{2}{36} + \frac{1}{36}) = \frac{76}{36}
Y = g(X)
\frac{6}{36}
\frac{20}{36}
\frac{30}{36}
p_X(x)
p_Y(y)
1
2
3
Y = g(X)
\therefore E[Y] = 1*(\frac{1}{36} + \frac{2}{36} + \frac{3}{36})
+3*(\frac{4}{36} + \frac{3}{36} + \frac{2}{36} + \frac{1}{36}) = \frac{76}{36}
+ 2*(\frac{4}{36} + \frac{5}{36} + \frac{6}{36} + \frac{5}{36})
\therefore E[Y] = 1*p_X(2) + 1 * p_X(3) + 1 * p_X(4)
+ 2*p_X(5) + 2 * p_X(6) + 2 * p_X(7) + 2 * p_X(8)
+ 3*p_X(9) + 3 * p_X(10) + 3 * p_X(11) + 3 * p_X(12)
Y = g(X)
\therefore E[Y] = 1*(\frac{1}{36} + \frac{2}{36} + \frac{3}{36})
+3*(\frac{4}{36} + \frac{3}{36} + \frac{2}{36} + \frac{1}{36}) = \frac{76}{36}
+ 2*(\frac{4}{36} + \frac{5}{36} + \frac{6}{36} + \frac{5}{36})
\therefore E[Y] = g(2)*p_X(2) + g(3) * p_X(3) + g(4) * p_X(4)
+ g(5)*p_X(5) + g(6) * p_X(6) + g(7) * p_X(7) + g(8) * p_X(8)
+ g(9)*p_X(9) + g(10) * p_X(10) + g(11) * p_X(11) + g(12) * p_X(12)
Y = \begin{cases}
1~~if~~x < 5 \\
2~~if~~5 \leq x \leq 8 \\
3~~if~~x > 8 \\
\end{cases}
\therefore E[Y] = \sum_x g(x)*p_X(x)
E[Y] = \sum_x g(x)*p_X(x)
E[Y] = \sum_y y*p_Y(y)
\equiv
Some properties of expectation
Linearity of expectation
Y = aX + b
E[Y] = \sum_{x \in \mathbb{R}_X} g(x)p_X(x)
= \sum_{x \in \mathbb{R}_X} a*x*p_X(x) +\sum_{x \in \mathbb{R}_X} b*p_X(x)
= a*E[X] + b*1~~(\because \sum_{x \in \mathbb{R}_X} p_X(x) = 1)
= a*\sum_{x \in \mathbb{R}_X} x*p_X(x) +b*\sum_{x \in \mathbb{R}_X} p_X(x)
= \sum_{x \in \mathbb{R}_X} (ax + b)p_X(x)
= aE[X] + b
Expectation of sum of RVs
Given a set of random variables
X_1, X_2, \dots, X_n
E[\sum_{i=1}^{n} X_i] = \sum_{i=1}^n E[X_i]
Expectation as mean of population
n students
\Omega
W: weights
30
40
50
60
E[W] = \sum_{i=1}^np_W(w_i)*w_i
p_W(w_i) = \frac{1}{n}
= \frac{1}{n}\sum_{i=1}^nw_i
(centre of gravity)
Expectation as centre of gravity
A patient needs a certain blood group which only 9% of the population has?
p = 0.09
E[X] = ?
= 1*0.09 + 2 * 0.91*0.09
+ 3 * 0.91 ^2 *.09 + 4 * 0.91^3*0.09 + \dots
= 0.09(\frac{a}{1-r} + \frac{dr}{(1-r)^2})
= \frac{1}{0.09} = \frac{1}{p}
= 0.09(1 + 2 * 0.91 + 3*0.91^2 + 4*0.91^3 + \dots
(a = 1, d = 1, r = 0.91)
= 11.11
Variance of a Random Variable
Variance of a RV
Expectation summarises a random variable
(but does not capture the spread of the RV)
E[X] = 0
E[Y] = 0
E[Z] = 0
X
0
1
-1
+1
\frac{1}{2}
\frac{1}{2}
Y
-100
-50
+100
+50
\frac{1}{4}
\frac{1}{4}
\frac{1}{4}
\frac{1}{4}
Z
Recap
Variance
\sigma^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu) ^ 2
0
-1
+1
0
-100
-50
+100
+50
0
Variance
Var(X) = E[(X - E(X))^2]
0
-1
+1
-100
-50
+100
+50
0
0
\sigma^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu) ^ 2
Variance
Var(X) = E[(X - E(X))^2]
\sigma^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \mu) ^ 2
\therefore Var(X) = E[X^2] -2\mu\cdot\mu + \mu^2
\therefore Var(X) = E[X^2] -2\mu E[ X] + \mu^2
\therefore Var(X) = E[X^2] - \mu^2 = E[X^2] - (E[X])^2
\therefore Var(X) = E[X^2] -E[2\mu X] + E[\mu^2]
\therefore Var(X) = E[(X - \mu)^2] = E[X^2 -2\mu X + \mu^2]
E[X] = \mu
Variance
Var(X) = E[X^2] - (E[X])^2
g(X) = X^2
\therefore E[X^2] = E[g(X)] = \sum_{x} p_X(x)g(x)
= 2^2 * \frac{1}{36} + 3^2 * \frac{2}{36} + 4^2 * \frac{3}{36} + 5^2 * \frac{4}{36} + 6^2 * \frac{5}{36}
+ 11^2 * \frac{2}{36}+ 12^2 * \frac{1}{36}
+ 7^2 * \frac{6}{36}+ 8^2 * \frac{5}{36}+ 9^2 * \frac{4}{36}+ 10^2 * \frac{3}{36}
= 54.83
Variance
Var(X) = E[X^2] - (E[X])^2
= 54.83 - 7^2
= 5.83
Variance: Mutual Funds
MF1 expected (average) returns: 8%
(12%, 2%, 25%, -9%, 10%)
MF2 expected (average) returns: 8%
(7%, 6%, 9%, 12%, 6%)
E[X] = \frac{1}{5}*12+\frac{1}{5}*2+\frac{1}{5}*25+\frac{1}{5}*(-9)+\frac{1}{5}*10 = 8
E[Y] = \frac{1}{5}*7+\frac{1}{5}*6+\frac{1}{5}*9+\frac{1}{5}*12+\frac{1}{5}*6 = 8
E[X^2] = \frac{1}{5}*12^2+\frac{1}{5}*2^2+\frac{1}{5}*25^2+\frac{1}{5}*(-9)^2+\frac{1}{5}*10^2
= 190.8
Variance: Mutual Funds
MF1 expected (average) returns: 8%
(12%, 2%, 25%, -9%, 10%)
MF2 expected (average) returns: 8%
(7%, 6%, 9%, 12%, 6%)
Var(X) = E[X^2] - (E[X])^2
Var(X) = 190.8 - 8^2 = 126.8
Var(Y) = 69.2 - 8^2 = 5.2
Properties of Variance
Var(aX + b) = E[(aX + b - E[aX + b])^2]
= E[(aX + b - aE[X] - b)^2]
= E[(a(X - E[X]))^2]
= E[a^2(X - E[X])^2]
= a^2E[(X - E[X])^2]
=a^2Var(X)
Properties of Variance
Var(X + X) = Var (2X) = 2^2Var(X)
Variance of sum of random variables
\neq Var(X) + Var(X)
In general, variance of the sum of random variables is not equal to the sum of the variances of the random variables
except......
Properties of Variance
P(X = x | Y = y) = P(X = x)~~\forall x \in \mathbb{R}_X, y \in \mathbb{R}_Y
Independent random variables
(X and Y are independent)
X: number~on~first~die
Y: sum~of~two~dice
P(Y = 8 ) = \frac{5}{36}
P(Y = 8 | X = 1) = 0 \neq P(Y = 8)
(X and Y are not independent)
Properties of Variance
We say that n random variables
X_1, X_2, \dots, X_n
are independent if
P(X_1 = x_1, X_2 = x_2, \dots, X_n = x_n)
= P(X_1=x_1)P (X_2=x_2) \dots P(X_n = x_n)
\forall x_1\in\mathbb{R}_{X_1}, x_2\in\mathbb{R}_{X_2}, \dots, x_n\in\mathbb{R}_{X_n}
Given such n random variables
Var(\sum_{i=1}^{n} X_i) = \sum_{i=1}^n Var(X_i)
Computing expectation and variance of some standard distributions
Bernoulli random variable
E[X] = \sum_{x=0}^1 x\cdot p_X(x)
p_X(x) = p^x(1-p)^{(1-x)}
= 0*(1-p) + 1 * p
=p
Var(X) = E[X^2] - (E[X])^2
= \sum_{x=0}^1 x^2 \cdot p_X(x)
- p^2
= p - p^2 = p(1-p)
Geometric random variable: \(E[X]\)
E[X] = \sum_{x=1}^\infty x\cdot p_X(x)
p_X(x) = (1-p)^{(x-1)}p
E[X] = 1*p + 2(1-p)p + 3(1-p)^2p + \dots
(1-p)E[X] = ~~~~~~~~~ + 1(1-p)p + 2(1-p)^2p + \dots
Subtraction eqn 2 from eqn 1
pE[X] = p + p(1-p) + p(1-p)^2 + p(1-p)^3 + \dots
E[X] = 1 + (1-p) + (1-p)^2 + (1-p)^3 + \dots
E[X] = \frac{1}{1- (1-p)} = \frac{1}{p}
Geometric random variable: \(Var(X)\)
Var(X) = E[X^2] - E[X]^2
E[X^2] = \sum_{i=1}^{\infty} i^2 (1-p)^{(i-1)}p
= \sum_{i=1}^{\infty} i^2 q^{(i-1)}p
= \sum_{i=1}^{\infty} (i - 1 + 1)^2 q^{(i-1)}p
= \sum_{i=1}^{\infty} ((i - 1)^2 + 2(i-1) + 1) q^{(i-1)}p
Substitute j = i-1
= \sum_{j=0}^{\infty} j^2q^{j}p
+ 2\sum_{j=0}^{\infty} jq^{j}p
+ \sum_{j=0}^{\infty} q^{j}p
q = 1-p
= q\sum_{j=0}^{\infty} j^2q^{j-1}p
+ 2q\sum_{j=0}^{\infty} jq^{j-1}p
+ \sum_{j=0}^{\infty} q^{j}p
Geometric random variable: \(Var(X)\)
Var(X) = E[X^2] - E[X]^2
E[X^2] = \sum_{i=1}^{\infty} i^2 (1-p)^{(i-1)}p
= q\sum_{j=0}^{\infty} j^2q^{j-1}p
+ 2q\sum_{j=0}^{\infty} jq^{j-1}p
+ \sum_{j=0}^{\infty} q^{j}p
\dots
= qE[X^2] + 2qE[X]+p\frac{1}{1-q}
q = 1 - p,~~~p= 1-q
E[X^2]= qE[X^2] + 2qE[X]+1
(1-q)E[X^2]= \frac{2q}{p}+1
(steps on previous slide)
E[X^2]= \frac{q+1}{p^2}
Var(X) = E[X^2] - E[X]^2
= \frac{2q+p}{p}
= \frac{q + q + p}{p} = \frac{q + 1}{p}
=\frac{q+1}{p^2} - \frac{1}{p^2}
=\frac{q}{p^2}
=\frac{1-p}{p^2}