Distributions of Sample Statistics

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Notation

- C F Gauss

"We need notions, not notations"

Population Parameters - $\mu, \sigma, p$

Sample statistics - $\bar{X}, S, \hat{p}$

Statistics of sample statistics - $\mathbb{E}[\bar{X}], \mathrm{sd}(\bar{X}), \mathrm{var}(\bar{X}), \\\mathbb{E}[S], \mathrm{sd}(S), \mathrm{var}(S), \\\mathbb{E}[\hat{p}], \mathrm{sd}(\hat{p}), \mathrm{var}(\hat{p})$

Population Size - $N$

Size of each Sample - $n$

Number of samples - $k$

Items in a sample - $X_1, X_2, \ldots, X_n$

Computing $\mathbb{E}[\bar{X}]$

Given $\mu, \sigma$

What is your guess of $\mathbb{E}[\bar{X}]$ ?
What would you like it to be?

Close to $\mu$

Computing $\mathbb{E}[\bar{X}]$

Given $\mu, \sigma$

$\mathbb{E}[\overline{X}]$

$= \mathbb{E}\left[\left(\frac{X_1+X_2 +\ldots + X_n}{n}\right)\right]$

$= \frac{1}{n}\mathbb{E}[X_1+X_2 +\ldots + X_n]$

$= \frac{1}{n}(\mathbb{E}[X_1]+\mathbb{E}[X_2] +\ldots + \mathbb{E}[X_n])$

$= \frac{1}{n}(\mu + \mu + \ldots + \mu)$

$= \mu$

Linearity of expected value

Recall Random Sampling $\Rightarrow X_1, X_2, \ldots X_n$ are independent

$\mathbb{E}$ of sum of independent random vars is the sum of $\mathbb{E}$ of individual variables

Recall each element of pop. is equally likely in samples

$\mathbb{E}$ of any item in any sample is $\mathbb{E}$ of any item in pop. = $\mu$

Computing $\mathbb{E}[\bar{X}]$

Given $\mu, \sigma$

$\mathbb{E}[\overline{X}]$

$= \mu$

Note that this is a "nice result"

Holds generally: independent of $\sigma$ , sample size $n$

$\overline{X}$ is an unbiased estimate of $\mu$

Looks simple, but an important result

Computing $\mathbb{E}[\bar{X}]$

$\mathbb{E}[\overline{X}]$

$= \mu$

$\overline{X}$ is an unbiased estimate of $\mu$

If we take any particular sample,
its $\overline{X}$ can be less than or more than $\mu$ but without any bias

Fair dice
$\mu = 3.5$

Systematic error

Individual samples vary,
but average of $\overline{X} \approx 3.5$

Computing $\mathbb{E}[\bar{X}]$

$\mathbb{E}[\bar{X}]$

$= \mu$

Fair dice

Vary sample size, number of samples

DEMO 1 of 2 - Discrete

Computing $\mathbb{E}[\bar{X}]$

$\mathbb{E}[\bar{X}]$

$= \mu$

Sum of random variates from $\mathcal{N}(0, 1)$

DEMO 2 of 2 - Continuous

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\overline{X})$

What is your guess of $\mathrm{var}(\overline{X})$ ?
What would you like it to be?

Close to $0$ would be nice

But definitely not more than $\sigma^2$

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

Unlike the case of $\mathbb{E}[\bar{X}]$ , we will here build intuition before proof

Exercise 1

$\bar{X}$ = Average of $n$ throws of a die

Compute variance of $\bar{X}$ of 1000 samples

Change $n$ and recompute. Plot variance vs $n$

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

Exercise 1

$\bar{X}$ = Average of $n$ throws of a die

Compute variance of $\bar{X}$ over 1000 samples

Change $n$ and recompute. Plot variance vs $n$

$\mathrm{var}(\bar{X})$

$n$

$y \propto \frac{1}{x}$

$\mathrm{var}(\bar{X})$ falls inversely with $n$ ?

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

Exercise 2

$\bar{X}$ = Average of 10 numbers from $\mathcal{N}(\mu, \sigma)$

Compute variance of $\bar{X}$ over 1000 samples

Change $\mu$ and recompute. Plot variance vs $\mu$
Change $\sigma$ and recompute. Plot variance vs $\sigma$

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

Exercise 2

$\bar{X}$ = Average of 10 numbers from $\mathcal{N}(\mu, \sigma)$

Compute variance of $\bar{X}$ over 1000 samples

Change $\mu$ and recompute. Plot variance vs $\mu$

Change $\sigma$ and recompute. Plot variance vs $\sigma$

$\mathrm{var}(\bar{X})$

$\mu$

$\mathrm{var}(\bar{X})$ is independent of $\mu$ ?

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

Exercise 2

$\bar{X}$ = Average of 10 samples from $\mathcal{N}(\mu, \sigma)$

Compute variance of $\bar{X}$ over 1000 samples

Change $\mu$ and recompute. Plot variance vs $\mu$

Change $\sigma$ and recompute. Plot variance vs $\sigma$

$\mathrm{var}(\bar{X})$

$\sigma$

$y \propto x^2$

$\mathrm{var}(\bar{X})$ is independent of $\mu$ ?

$\mathrm{var}(\overline{X})$ scales as $\sigma^2$ ?

Computing $\mathrm{var}(\bar{X})$

Given $\mu, \sigma$

Compute $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$ is independent of $\mu$ ?

$\mathrm{var}(\overline{X})$ scales as $\sigma^2$ ?

$\mathrm{var}(\bar{X})$

$=\frac{\sigma^2}{n}$

Let's work towards a proof

Computing $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$

$=\mathbb{E}[(\bar{X} - \mathbb{E}[\bar{X}])^2]$

$=\mathbb{E}[(\bar{X}^2 + \mathbb{E}[\bar{X}]^2 - 2\bar{X}\mathbb{E}[\bar{X}])$

$=\mathbb{E}[\bar{X}^2] + \mathbb{E}[\bar{X}]^2 - 2\mathbb{E}[\bar{X}]\mathbb{E}[\bar{X}]$

$=\mathbb{E}[\bar{X}^2] - \mathbb{E}[\bar{X}]^2$

A helpful reformulation of $\mathrm{var}(\bar{X})$

Computing $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$

$=\mathbb{E}[\bar{X}^2] - \mathbb{E}[\bar{X}]^2$

$\mathrm{var}(a\bar{X})$

$=\mathbb{E}[(a\bar{X})^2] - \mathbb{E}[(a\bar{X})]^2$

$=a^2 (\mathbb{E}[\bar{X}^2] - \mathbb{E}[\bar{X}]^2)$

$=a^2 (\mathrm{var}(\bar{X}))$

Computing $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$

$=\mathbb{E}[\bar{X}^2] - \mathbb{E}[\bar{X}]^2$

$\mathrm{var}(a\bar{X})$

$=a^2 (\mathrm{var}(\bar{X}))$

$\mathrm{var}(X_1 + X_2)$

$=\mathbb{E}[(X_1+X_2)^2] - \mathbb{E}[(X_1+X_2)]^2$

$=\mathbb{E}[X_1^2]+\mathbb{E}[X_2^2] + 2\mathbb{E}[X_1X_2]- \mathbb{E}[X_1]^2-\mathbb{E}[X_2]^2- 2\mathbb{E}[X_1]\mathbb{E}[X_2]$

$=\mathbb{E}[X_1^2] - \mathbb{E}[X_1]^2 + \mathbb{E}[X_2^2] - \mathbb{E}[X_2]^2$

$=\mathrm{var}(X_1) + \mathrm{var}(X_2)$

Computing $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$

$=\mathbb{E}[\bar{X}^2] - \mathbb{E}[\bar{X}]^2$

$\mathrm{var}(a\bar{X})$

$=a^2 (\mathrm{var}(\bar{X}))$

$\mathrm{var}(X_1 + X_2)$

$=\mathrm{var}(X_1) + \mathrm{var}(X_2)$

$\mathrm{var}(\bar{X})$

$=\frac{1}{n^2}\mathrm{var}(X_1 + X_2 + \ldots + X_n)$

$=\mathrm{var}(\frac{X_1 + X_2 + \ldots + X_n}{n})$

$=\frac{1}{n^2}(\mathrm{var}(X_1) + \mathrm{var}(X_2) + \ldots + \mathrm{var}(X_n) )$

$=\frac{1}{n^2}(\sigma^2 + \sigma^2 + \ldots + \sigma^2)$

$=\frac{\sigma^2}{n}$

Computing $\mathrm{var}(\bar{X})$

$\mathrm{var}(\bar{X})$

$\mathrm{sd}(\bar{X})$

$=\frac{\sigma^2}{n}$

Implications

$\bar{X}$ is an unbiased estimate. But it varies from sample to sample with $\mathrm{sd} = \frac{\sigma}{\sqrt{n}}$

To half $\mathrm{sd}$ need to quadruple $n$

Cannot decrease $\sigma$ , can increase $n$

$= \frac{\sigma}{\sqrt{n}}$

Applied to Data Science

$\overline{X_{income}}$

$\mu, \sigma$

$\overline{X_{income}}$ are unbiased estimates of $\mu$
But if they vary a lot from one sample to another, propose to increase sample size

Summary

Given population parameters $\mu$ and $\sigma$ :

$\mathbb{E}[\bar{X}] = \mu$

$\mathrm{var}(\bar{X}) = \frac{\sigma^2}{n}$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Central Limit Theorem

$\mathbb{E}[\bar{X}] = \mu$

$\mathrm{var}(\bar{X}) = \frac{\sigma^2}{n}$

We know the mean and variance,
can we know more about the PDF of $\bar{X}$ ?

$\overline{X} = \frac{1}{n}(X_1 + X_2 + \ldots + X_n)$

Sum of $n$ independent and identically distributed (iid) random variables

The Central Limit Theorem has something interesting to say about such sums

Central Limit Theorem

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$X_1 + X_2 + \ldots + X_n$
will converge to $\mathcal{N}(n\mu, \sigma\sqrt{n})$ in the limiting condition of $n\to \infty$

Central Limit Theorem

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$X_1 + X_2 + \ldots + X_n$
will converge to $\mathcal{N}(n\mu, \sigma\sqrt{n})$ in the limiting condition of $n\to \infty$

Awesome result!

Independent of the PDF of the population, the PDF of the sum of many samples tends to the normal distribution

Check that our derived results on $\mathbb{E}(\overline{X})$ and $\mathrm{sd}(\overline{X})$ remain true for large $n$

Exercise

Central Limit Theorem

Independent of the PDF of the population, the PDF of the sum of many samples tends to the normal distribution

Exercise

Plot the empirical distribution of sum of $n$ iid samples from distribution $\mathcal{D}$

Do this for $n = 10, 100, 1000, 10000$

$\mathcal{D}$ = Uniform, Binomial, Bimodal, Discrete

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$X_1 + X_2 + \ldots + X_n$
will converge to $\mathcal{N}(n\mu, \sigma\sqrt{n})$ in the limiting condition of $n\to \infty$

Central Limit Theorem

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$X_1 + X_2 + \ldots + X_n$
will converge to $\mathcal{N}(n\mu, \sigma\sqrt{n})$ in the limiting condition of $n\to \infty$

Demo

Central Limit Theorem

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$\frac{X_1-\mu}{\sigma} + \frac{X_2-\mu}{\sigma} + \ldots + \frac{X_n-\mu}{\sigma}$
will have a PDF that converges to $\mathcal{N}(0, 1)$ in the limiting condition of $n\to \infty$

An alternative version of CLT

CLT - Attempt at Proof

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$\frac{X_1-\mu}{\sigma} + \frac{X_2-\mu}{\sigma} + \ldots + \frac{X_n-\mu}{\sigma}$
will have a PDF that converges to $\mathcal{N}(0, 1)$ in the limiting condition of $n\to \infty$

What's strange about proving such a theorem?

We need to compare two PDFs !?

How do we test convergence as $n\to\infty$ !?

CLT - Attempt at Proof

We need to compare two PDFs !?

Two PDFs can have the same $\mu, \sigma$ and still differ

Exercise: Come up with an example

$\mathcal{N}(0, 1/\sqrt{3})$

$\mathrm{Uniform}(-1, 1)$

$\mu = 0, \sigma = 1/\sqrt{3}$

CLT - Attempt at Proof

We need to compare two PDFs !?

Two PDFs can have the same $\mu, \sigma$ and still differ

$\Rightarrow$ Need to compare more "terms"

These terms are called moments

$\mathbb{E}[X^n]$

$= \sum_i x_i^n p(x_i)$ , discrete

$= \int_{-\infty}^\infty x^n f(x) dx$ , continuous

The $n$ th moment is given by

CLT - Attempt at Proof

We need to compare two PDFs !?

$\mathbb{E}[X^n]$

$= \sum_i x_i^n p(x_i)$ , discrete

$= \int_{-\infty}^\infty x^n f(x) dx$ , continuous

The $n$ th moment is given by

Exercise: What is the 0th moment?

Answer: 1

CLT - Attempt at Proof

We need to compare two PDFs !?

$\mathbb{E}[X^n]$

$= \sum_i x_i^n p(x_i)$ , discrete

$= \int_{-\infty}^\infty x^n f(x) dx$ , continuous

The $n$ th moment is given by

Exercise: What is the 1st moment?

Answer: $\mu$

CLT - Attempt at Proof

We need to compare two PDFs !?

$\mathbb{E}[X^n]$

$= \sum_i x_i^n p(x_i)$ , discrete

$= \int_{-\infty}^\infty x^n f(x) dx$ , continuous

The $n$ th moment is given by

Exercise: What is the 2nd moment of the mean adjusted variable $X = X' - \mu$ ?

Answer: $\sigma^2$

CLT - Attempt at Proof

We need to compare two PDFs !?

$\mathbb{E}[X^n]$

$= \sum_i x_i^n p(x_i)$ , discrete

$= \int_{-\infty}^\infty x^n f(x) dx$ , continuous

The $n$ th moment is given by

What is the 3rd moment of the mean, variance adjusted variable $X = \frac{X' - \mu}{\sigma}$ ?

Answer: Turns out to be a measure of skewness

CLT - Attempt at Proof

We need to compare two PDFs !?

Thus to compare two PDFs we have to carefully match their moments

How do we test convergence as $n\to\infty$ !?

What must be a property of a PDF that satisfies a convergence condition?

CLT - Attempt at Proof

How do we test convergence as $n\to\infty$ !?

What must be a property of a PDF that satisfies a convergence condition?

Let $Y_k = X_1 + X_2 + \ldots + X_k$ , for all $k$

If $Y_n \sim \mathcal{N}(\mu, \sigma)$ ,
then CLT requires that $Y_{2n} \sim \mathcal{N}(\mu', \sigma')$

The sum of two normally distributed r.v.s is also normally distributed?

CLT - Attempt at Proof

How do we test convergence as $n\to\infty$ !?

The sum of two normally distributed r.v.s is also normally distributed?

Turns out to be true!

For two independent r.v.s
if $X \sim \mathcal{N}(\mu_X, \sigma_X)$ and $Y \sim \mathcal{N}(\mu_Y, \sigma_Y)$

Then $X+Y \sim \mathcal{N}(\mu_X + \mu_Y, \sqrt{\sigma_X^2 + \sigma_Y^2})$

Exercise: Test this out graphically

CLT - Attempt at Proof

How do we test convergence as $n\to\infty$ !?

Thus normal distribution is an attractor
Once the distribution becomes normal,
it remains normal

CLT - Attempt at Proof

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$\frac{X_1-\mu}{\sigma} + \frac{X_2-\mu}{\sigma} + \ldots + \frac{X_n-\mu}{\sigma}$
will have a PDF that converges to $\mathcal{N}(0, 1)$ in the limiting condition of $n\to \infty$

What's strange about proving such a theorem?

We need to compare two PDFs !?

How do we test convergence as $n\to\infty$ !?

Moments

Sum of normals

CLT - Implications

What use is CLT for data science?

Independent of PDF of population, PDF of $\overline{X}$ is normal

Super elegant

" The mathematician does not study pure mathematics because it is useful; he studies it because he delights in it and he delights in it because it is beautiful.
- George Cantor

CLT - Implications

What use is CLT for data science?

PDF( $X$ )

PDF( $\overline{X}$ )

Independent of PDF of population, PDF of $\overline{X}$ is normal

Super elegant

If we know (or guess) $\mu, \sigma$ and have a measurement of $\overline{X}$ ,
we can compute how likely is getting a value greater than $\overline{X}$

Likelihood of sample mean

CLT - Implications

What use is CLT for data science?

Exercise

Likelihood of sample mean

We have a population of die throws

We take a sample of 10 die throws

And find their total to be 40

How likely is it that we do better?

CLT - Implications

What use is CLT for data science?

Exercise

Likelihood of sample mean

We have a population of die throws

We take a sample of 10 die throws

And find their total to be 40

How likely is it that we do better?

Step 1

$\mu = ?, \sigma = ?, n = ?$

$\mu = 3.5, \sigma = 1.708,\\ n = 10$

Step 2

$\mathbb{E}[X] = n\mu, \mathrm{sd}(X) =\sigma\sqrt{n}$

$\mathbb{E}[X] = 35, \mathrm{sd}(X) =5.401$

Step 3

$\mathcal{N}(35,5.41)$

$\mathrm{Pr}(X > 40) =$ Area of yellow part

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

Let us now see how we can compute such areas under $\mathcal{N}$

$\mathrm{Pr}(X > 40) =$ Area of yellow part $\approx 0.177$

Given CLT, such area computations would always involve normal distributions

Computing area under $\mathcal{N}$

Key insight:
Relate $\mathcal{N}(\mu, \sigma)$ for any $\mu, \sigma$ to $\mathcal{N}(0, 1)$

Given $X \sim \mathcal{N}(\mu, \sigma)$
What can you say about $Z = \frac{X - \mu}{\sigma}$

We already have an intuition on this

Yes, $Z \sim \mathcal{N}(0, 1)$

This transformation of $x \to \frac{x - \mu}{\sigma}$ is so important that we give it a name

Computing area under $\mathcal{N}$

This transformation of $x \to \frac{x - \mu}{\sigma}$ is so important that we give it a name

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

This is a generic definition for any random variable

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathcal{N}(0, 1)$
Calculate z-scores for x = -3, -0.5, 0, 0.5, 3

$\mu = 0$ . $\sigma =1$

x	-3	-0.5	0	0.5	3
z

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathcal{N}(0, 1)$
Calculate z-scores for x = -3, -0.5, 0, 0.5, 3

$\mu = 0$ . $\sigma =1$

x	-3	-0.5	0	0.5	3
z	-3	-0.5	0	0.5	3

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathcal{N}(1, 2)$
Calculate z-scores for x = -3, -0.5, 0, 0.5, 3

$\mu = 1$ . $\sigma =2$

x	-3	-0.5	0	0.5	3
z

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathcal{N}(1, 2)$
Calculate z-scores for x = -3, -0.5, 0, 0.5, 3

$\mu = 1$ . $\sigma =2$

x	-3	-0.5	0	0.5	3
z	-2	-0.75	-0.5	-0.25	1

Computing area under $\mathcal{N}$

$X \sim \mathcal{N}(1, 2)$

$\mu = 1$ . $\sigma =2$

x	-3	-0.5	0	0.5	3
z	-2	-0.75	-0.5	-0.25	1

$X \sim \mathcal{N}(0, 1)$

$\mu = 0$ . $\sigma =1$

x	-3	-0.5	0	0.5	3
z	-3	-0.5	0	0.5	3

-ve Z score: Left of mean
+ve Z score: Right of mean
0 Z score: Exactly mean

Higher |Z score|: farther from mean, lower prob.

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathrm{Uniform}[-1, 1]$
Calculate z-scores for x = -2, -0.5, 0, 0.5, 2

$\mu = 0$ . $\sigma =\frac{1}{\sqrt{3}} \approx 0.58$

x	-2	-0.5	0	0.5	2
z

$\mu = 0$ . $\sigma =?$

$\sigma^2 = \int_{-1}^1 (x - \mu)^2 f(x) dx$

$= \int_{-1}^1 (x - 0)^2 (1/2) dx$

$= \frac{1}{6} x^3|_{-1}^1$

$= \frac{1}{3}$

Computing area under $\mathcal{N}$

Z Score
The number of standard deviations a value is away from the mean
$z = \frac{x - \mu}{\sigma}$

Examples
$X \sim \mathrm{Uniform}[-1, 1]$
Calculate z-scores for x = -2, -0.5, 0, 0.5, 2

$\mu = 0$ . $\sigma =\frac{1}{\sqrt{3}} \approx 0.58$

x	-2	-0.5	0	0.5	2
z	-3.46	-0.87	0	0.87	3.46

Computing area under $\mathcal{N}$

Z Score - Special Significance for $\mathcal{N}$

Given $X \sim \mathcal{N}(\mu, \sigma)$ and $Z = \frac{X - \mu}{\sigma}$

Then, $Z \sim \mathcal{N}(0, 1)$

$\mathrm{Pr}(X < a)$

$= \mathrm{Pr}\left(Z < \frac{a - \mu}{\sigma}\right)$

$= \mathrm{Pr}\left(\frac{X - \mu}{\sigma} < \frac{a - \mu}{\sigma}\right)$

Seems simple, but effective!

Reduce all checks on to $\mathcal{N}(0, 1)$

Store in tables, and look-up!

Computing area under $\mathcal{N}$

Z Score - Special Significance for $\mathcal{N}$

Given $X \sim \mathcal{N}(\mu, \sigma)$ and $Z = \frac{X - \mu}{\sigma}$

Then, $Z \sim \mathcal{N}(0, 1)$

$\mathrm{Pr}(X < a)$

$= \mathrm{Pr}\left(Z < \frac{a - \mu}{\sigma}\right)$

Examples

$X \sim \mathcal{N}(1, 1.5)$

$\mathrm{Pr}(X<0)$

$= \mathrm{Pr}(Z<\frac{0-1}{1.5})$

Two yellow areas match

Computing area under $\mathcal{N}$

Z Score - Special Significance for $\mathcal{N}$

Given $X \sim \mathcal{N}(\mu, \sigma)$ and $Z = \frac{X - \mu}{\sigma}$

Then, $Z \sim \mathcal{N}(0, 1)$

$\mathrm{Pr}(X > a)$

$= \mathrm{Pr}\left(Z > \frac{a - \mu}{\sigma}\right)$

Examples

$X \sim \mathcal{N}(1, 1.5)$

$\mathrm{Pr}(X>2)$

$= \mathrm{Pr}(Z >\frac{2-1}{1.5})$

Computing area under $\mathcal{N}$

Demo

Computing area under $\mathcal{N}$

Efficiency in calculating

$\mathrm{Pr}(X > a)$

$= \mathrm{Pr}\left(Z > \frac{a - \mu}{\sigma}\right)$

When computers were expensive ...
Instead of computing $\mathrm{Pr}(Z > c)$ each time, can we simply look-up in some tables?

Remember log tables from school?

Do we need to store $\mathrm{Pr}(Z > c)$ for all values of $c$ . What about $\mathrm{Pr}(Z < c)$ ?

Computing area under $\mathcal{N}$

Some relations on the areas

+

= 1

$\mathrm{Pr}(Z > a) + \mathrm{Pr}(Z \leq a) = 1$

$\mathrm{Pr}(Z > a) = \mathrm{Pr}(Z < -a)$

=

$\mathrm{Pr}(Z < -3.5) \approx 0$

Computing area under $\mathcal{N}$

Some relations on the areas

$\mathrm{Pr}(Z > a) + \mathrm{Pr}(Z \leq a) = 1$

$\mathrm{Pr}(Z > a) = \mathrm{Pr}(Z < -a)$

$\mathrm{Pr}(Z > 3.5) \approx 0$

Efficient table look-up

Sufficient to store $\mathrm{Pr}(Z < a)$
for $a \in [-3.5, 0]$

Computing area under $\mathcal{N}$

Efficient table look-up

Sufficient to store $\mathrm{Pr}(Z < a)$
for $a \in [-3.5, 0]$

$\mathrm{Pr}(Z < -2.81)$

$= .0025$

Computing area under $\mathcal{N}$

Demo

Computing area under $\mathcal{N}$

More areas

Given $X \sim \mathcal{N}(\mu, \sigma)$ and $Z = \frac{X - \mu}{\sigma}$

Then, $Z \sim \mathcal{N}(0, 1)$

$\mathrm{Pr}(a < X < b)$

$= \mathrm{Pr}\left(\frac{a - \mu}{\sigma} < Z < \frac{b - \mu}{\sigma}\right)$

Examples

$X \sim \mathcal{N}(1, 1.5)$

$\mathrm{Pr}(-1 < X<0)$

$= \mathrm{Pr}(\frac{-1-1}{1.5} < Z<\frac{0-1}{1.5})$

Two yellow areas match

Computing area under $\mathcal{N}$

More areas

$\mathrm{Pr}(\frac{-4}{3} < Z<\frac{-2}{3})$

$\mathrm{Pr}(Z<\frac{-2}{3})$

$\mathrm{Pr}( Z<\frac{-4}{3})$

$-$

$=$

How will we compute this?

Two look-ups and subtraction

Back to our example

What use is CLT for data science?

Exercise

Likelihood of sample mean

We have a population of die throws

We take a sample of 10 die throws

And find their total to be 40

How likely is it that we do better?

Step 1

$\mu = 3.5, \sigma = 1.708,\\ n = 10$

Step 2

$\mathbb{E}[X] = 35, \mathrm{sd}(X) =5.401$

Step 3

$\mathcal{N}(35,5.401)$

$\mathrm{Pr}(X > 40) =$ Area of yellow part

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

$\mathcal{N}(35,5.401)$

$\mathrm{Pr}(X > 40) =$ Area of yellow part

$= \mathrm{Pr}\left(Z > \frac{40 -35}{5.401}\right)$

$= \mathrm{Pr}\left(Z > 0.926\right)$

$= 0.177$

We have a fighting chance to improve on 40!

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

Let us check this without CLT

We can plot the PMF $f(x)$ for each of the all possible values

$\mathrm{Pr}(Q > 40)$

$= f(41) + f(42) + \ldots + f(60)$

$= 0.157$

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

We have over-estimated our chance

Let us super-impose $\mathcal{N}(35. 5.401)$

Looks good. Where did we go wrong!?

CLT - Implications

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

Let's zoom in

CLT - Implications

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

Let's zoom in

Map these values to areas

CLT - Implications

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

Let's zoom in

Yes we are overestimating

Alternative: Take yellow
area from 41

CLT - Implications

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

$\mathrm{Pr}(X > 41)$

$= \mathrm{Pr}(Z > \frac{41-35}{5.401})$

$= 0.133$ (from CLT)

Ha! Underestimation

And its clear why

Why not 40.5?!

CLT - Implications

$\mathrm{Pr}(X > 40)$

$= 0.177$ (from CLT)

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

$\mathrm{Pr}(X > 41)$

$= 0.133$ (from CLT)

$\mathrm{Pr}(X > 40.5)$

$= \mathrm{Pr}(Z > 1.02 )$

$= 0.156$

Neat!

CLT - Implications

$\mathrm{Pr}(Q > 40)$

$= 0.157$ (PMF)

$\mathrm{Pr}(X > 40.5)$

$= 0.156$ (CLT)

Continuity correction

CLT - Implications

$\mathrm{Pr}(Q > 25$ and $Q < 36)$

What we want to estimate

What we compute

$= \mathrm{Pr}(X < 35.5) - \mathrm{Pr}( X < 25.5)$

One more example

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

Approximating distributions

In a random group of 100 people, compute the PMF of number of people with birthdays on Tuesday

Recognise this distribution?

Binomial distribution

CLT - Implications

In a random group of 100 people, compute the PMF of number of people with birthdays on Tuesday

Binomial distribution

p = 1/7, n = 100

$\mathrm{Pr}(Q = k) = \binom{n}{k} p^k (1-p)^{n - k}$

Do this for 50 values of $k$
and you get a curve like this

$\mathrm{Pr}(Q = k)$

$k$

Expensive
to compute!

CLT - Implications

Binomial distribution

p = 1/7, n = 100

$\mathrm{Pr}(Q = k) = \binom{n}{k} p^k (1-p)^{n - k}$

$\mathrm{Pr}(Q = k)$

$k$

Let us plot $\mathcal{N}(n\mu, \sqrt{n}\sigma) =$

Pretty good approximation!

$X$ can be seen as a the sum of 100 iid r.v.s with $\mu = p = 1/7$ and $\sigma = \sqrt{p(1-p)} \approx 0.35$

$\mathcal{N}(14.29, 0.35)$

CLT - Implications

Using CLT

$\mathrm{Pr}(Q = k) = \binom{n}{k} p^k (1-p)^{n - k}$

$k$

Let $X \sim$

$\mathcal{N}(14.29, 0.35)$

$\mathrm{Pr}(Q = k)$

$\approx\mathrm{Pr}(X < k + 0.5) - \mathrm{Pr}(X < k - 0.5)$

$k = 10$

$0.058$

$0.054$

$k = 15$

$0.109$

$0.111$

$k = 18$

$0.061$

$0.065$

CLT - Implications

Normal approximation of binomial distribution

Demo

Insights

Higher accuracy for $p \approx 0.5$
Higher accuracy for larger $n$

Rule of thumb:
$np > 10$ and $n(1-p) > 10$

CLT - Implications

What use is CLT for data science?

Likelihood of sample mean

Approximating distributions

Understanding bell curves in data

CLT - Implications

Understanding bell curves in data

A lot of real world data follows this trend

Weight of new-borns

Stock market returns

Measurement errors

Heights of adults

Not exactly $\mathcal{N}$ , but close

Why?

CLT - Implications

Understanding bell curves in data

If $X_1, X_2, \ldots, X_n$ are random samples from a population with mean $\mu$ and finite standard deviation $\sigma$ , then the sum
$X_1 + X_2 + \ldots + X_n$
will converge to $\mathcal{N}(n\mu, \sigma\sqrt{n})$ in the limiting condition of $n\to \infty$

CLT so far

What happens if $X_1, X_2, \ldots, X_n$ had different $\mu, \sigma$ ?

More general version of CLT

CLT - Implications

Understanding bell curves in data

More general version of CLT

If $X_1, X_2, \ldots, X_n$ are independent random variables each with mean $\mu_i$ and finite variance $\sigma_i^2$ , and let $s^2_n = \sum_i^n \sigma_i^2$ , then the sum
$\sum_{i = 1}^{n}\frac{X_i - \mu_i}{s_n}$ converges in distribution to $\mathcal{N}(0, 1)$ under some conditions*

* Roughly, no $\sigma^2_i$ is comparable to the sum $s_n^2$

CLT - Implications

Understanding bell curves in data

General CLT

If $X_1, X_2, \ldots, X_n$ are independent random variables each with mean $\mu_i$ and finite variance $\sigma_i^2$ , and let $s^2_n = \sum_i^n \sigma_i^2$ , then the sum
$\sum_{i = 1}^{n}\frac{X_i - \mu_i}{s_n}$ converges in distribution to $\mathcal{N}(0, 1)$ under some conditions*

Implications?

If we can write a r.v. $X$ as the sum of many many independent r.v.s,

Then $X$ is likely to have a
bell-shaped distribution

CLT - Implications

Example of sum of r.v.s

$X_1$

$X_2$

$X_3$

$X_4$

$X_1 + X_2$

$X_1 + X_2 + X_3$

$X_1 + X_2 + X_3 + X_4$

CLT - Implications

Example of sum of r.v.s

$X_1$

$X_2$

$X_3$

$X_4$

$X_1 + X_2$

$X_1 + X_2 + X_3$

$X_1 + X_2 + X_3 + X_4$

CLT - Implications

Understanding bell curves in data

Revisit examples

Weight of new-borns

Stock market returns

Measurement errors

Heights of adults

CLT - Implications

Choices we make

Grading in examinations

Check out how IQ scores are given

CLT - Implications

Choices we make

Fitting distributions in Machine Learning
Gaussian Mixture Models

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Recap the big picture again

$\Omega$

Population size = $N$

Size of each sample = $n$

$|\Omega| =$ $N \choose n$

$F$

0

1

$P$

$\dfrac{1}{|\Omega|}$

0

1

$P$

$\dfrac{1}{|\Omega|}$

$X$

10 km

40 km

5 km

$S^2_{\tiny{mileage}}$

$Pr(S^2_{mileage})$

Recap the big picture again

$X = S^2$

For each sample we want to compute the r.v. which is the sample variance

Then, we want to estimate $\mathbb{E}[S^2]$ and $\mathrm{var}(S^2)$ with $\mu, \sigma$

Computing $S^2$

What is the formula for $S^2$ ?

For a given sample,
$S^2 = \frac{\sum_{i =1}^n (X_i - \overline{X})^2}{n}$

Note that we compute the statistic with only information from the sample,
and not from the population

Estimating $\mathbb{E}[S^2]$

First Question

For a given sample, $S^2 = \frac{\sum_{i =1}^n (X_i - \overline{X})^2}{n}$

What is the relation of $\mathbb{E}[S^2]$ to $\mu, \sigma$ ?

Exercise

Consider each sample as three dice throws
Sample 1,000 such samples, compute $S^2$
Compute average of the 1,000 $S^2$ values

What would you like it to be?

Estimating $\mathbb{E}[S^2]$

Exercise

Consider each sample as three dice throws
Sample 1,000 such samples, compute $S^2$
Compute average of the 1,000 $S^2$ values

⚅⚃⚀

⚅⚃⚅

$\overline{X} = \frac{11}{3}$

⚀⚂⚂

$\overline{X} = \frac{7}{3}$

$S^2 = \frac{8}{9}$

$S^2 = \frac{38}{9}$

$\overline{X} = \frac{16}{3}$

$S^2 = \frac{8}{9}$

Estimating $\mathbb{E}[S^2]$

Exercise

Consider each sample as three dice throws
Sample 1,000 such samples, compute $S^2$
Compute average of the 1,000 $S^2$ values

Average $S^2 = 1.99$

1000 samples

Estimating $\mathbb{E}[S^2]$

Exercise

Consider each sample as three dice throws
Sample 1,000 such samples, compute $S^2$
Compute average of the 1,000 $S^2$ values

Average $S^2 = 1.944$

$10^6$ samples

Estimating $\mathbb{E}[S^2]$

Exercise

Consider each sample as three dice throws

$\mathbb{E}[S^2] \approx 1.944$

$\mu = 3.5, \sigma^2 = 2.916$

We already know that for fair dice,

In this case, $\mathbb{E}[S^2] < \sigma^2$

Estimating $\mathbb{E}[S^2]$

Exercise

Let us take a continuous distribution
Sample 3 values from $\mathcal{N}(0, 1)$
Compute $S^2$
Repeat for 100,000 samples

Average of $S^2 = 0.665$

Estimating $\mathbb{E}[S^2]$

Exercise

Sample 3 values from $\mathcal{N}(0, 1)$

Average of $S^2 = 0.665$

We know that $\mu = 0, \sigma = 1$

Again $\mathbb{E}[S^2] < \sigma^2$

We can play with two variables $\mu$ and $n$

Estimating $\mathbb{E}[S^2]$

Exercise

Sample 3 values from $\mathcal{N}(4, 1)$

Average of $S^2 = 0.666$ - Minor change

Again $\mathbb{E}[S^2] < \sigma^2$

Estimating $\mathbb{E}[S^2]$

Exercise

Sample 4 values from $\mathcal{N}(0, 1)$

Average of $S^2 = 0.749$

Again $\mathbb{E}[S^2] < \sigma^2$

Estimating $\mathbb{E}[S^2]$

Exercise

Sample 5 values from $\mathcal{N}(0, 1)$

Average of $S^2 = 0.801$

Plot the trend of $\mathbb{E}[S^2]$ with $n$

Again $\mathbb{E}[S^2] < \sigma^2$ , but getting close

Estimating $\mathbb{E}[S^2]$

Exercise

Sample $n$ values from $\mathcal{N}(0, 1)$

$\mathbb{E}[S^2]$ & $\sigma^2$ vs n

$\mathbb{E}[S^2]$ $\to$ $\sigma^2$ as $n \to \infty$ ?

$\mathbb{E}[S^2]$ is an under-estimate of $\sigma^2$ and gets more accurate as sample size increases

Guess

Estimating $\mathbb{E}[S^2]$

$S^2$ is an under-estimate of $\sigma^2$ and gets more accurate as sample size increases

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

We also know that $\overline{X}$ is an unbiased estimate of $\mu$

Why then do we underestimate $\sigma^2$ with $\mathbb{E}[S^2]$

We will see two arguments
Geometric and Algebraic

Estimating $\mathbb{E}[S^2]$

Why then do we underestimate $\sigma^2$ with $\mathbb{E}[S^2]$

Geometric argument

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

Consider the example of three dice throws

1

2

3

4

5

6

$\mu = 3.5$

⚀

Lets say we get 1, 3, 4

⚂

⚃

$\overline{X} = 2.67$

Average distance of samples from $\overline{X}$
$\leq$ Average distance of samples from $\mu$

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

Estimating $\mathbb{E}[S^2]$

Why then do we underestimate $\sigma^2$ with $S^2$

Geometric argument

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

Consider the example of three dice throws

1

2

3

4

5

6

$\mu = 3.5$

⚀

Problem is less severe for spread out sample

⚂

⚅

$\overline{X} = 3.33$

There is still a small under-estimation

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

Estimating $\mathbb{E}[S^2]$

Why then do we underestimate $\sigma^2$ with $S^2$

Geometric argument

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

Consider the example of three dice throws

1

2

3

4

5

6

$\mu = 3.5$

⚀

Problem is severe for clustered samples

⚀

⚁

$\overline{X} = 1.33$

Such samples are rare, but not impossible

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

Estimating $\mathbb{E}[S^2]$

Why then do we underestimate $\sigma^2$ with $S^2$

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

In each case $\mathbb{E}[S^2]$ $<$ $\sigma^2$

1

2

3

4

5

6

⚀

⚁

1

2

3

4

5

6

⚀

⚂

⚅

1

2

3

4

5

6

⚀

⚂

⚃

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

Estimating $\mathbb{E}[S^2]$

Why then do we underestimate $\sigma^2$ with $S^2$

$\mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \mu)^2}{n}\right] =$ $\sigma^2$

Algebraic argument

$\sum_{i=1}^n(X_i - \mu)^2$

$=\sum_{i=1}^n((X_i - \overline{X})+(\overline{X} - \mu))^2$

$=\sum_{i=1}^n((X_i - \overline{X})^2+(\overline{X} - \mu)^2 + 2(X_i - \overline{X})(\overline{X} - \mu))$

$0$

$=\sum_{i=1}^n(X_i - \overline{X})^2+ \sum_{i =1}^n(\overline{X} - \mu)^2$

Positive $\Rightarrow$ underestimation

Relate this term to geometric argument

$\mathbb{E}[S^2]$ $= \mathbb{E}\left[\frac{\sum_{i = 1}^n(X_i - \overline{X})^2}{n}\right]$

Estimating $\mathbb{E}[S^2]$

Let us find expected value of the error

$\mathbb{E}[\sigma^2 - S^2]$

$= \mathbb{E}[\frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2 - \frac{1}{n} \sum_{i = 1}^n(X_i - \overline{X})^2]$

$= \mathbb{E}[\frac{1}{n}\sum_{i=1}^n ((X_i^2 -2X_i\mu + \mu^2) - (X_i^2 - 2X_i\overline{X} + \overline{X}^2))]$

$= \mathbb{E}[\frac{1}{n}\sum_{i=1}^n (\mu^2 - \overline{X}^2 + 2X_i(\overline{X}-\mu)]$

$= \mathbb{E}[\mu^2 - \overline{X}^2 +\frac{1}{n} \sum_{i = 1}^n 2X_i(\overline{X}-\mu)]$

$= \mathbb{E}[\mu^2 - \overline{X}^2 +2\overline{X}(\overline{X}-\mu)]$

$= \mathbb{E}[\mu^2 + \overline{X}^2 -2\overline{X}\mu]$

$= \mathbb{E}[(\overline{X} - \mu)^2]$

$= \mathrm{Var}(\overline{X})$

$= \frac{\sigma^2}{n}$

Estimating $\mathbb{E}[S^2]$

Let us find expected value of the error

$\mathbb{E}[\sigma^2 - S^2]$

$= \frac{\sigma^2}{n}$

$\mathbb{E}[S^2] = \frac{n - 1}{n} \sigma^2$

$S^2$ & $\sigma^2$ vs $n$

Now this plot
make senses

Estimating $\mathbb{E}[S^2]$

$\mathbb{E}[S^2] = \frac{n - 1}{n} \sigma^2$

We don't quite like this systematic error (bias)

We define an unbiased variance $S_{n-1}^2$ as

$\mathbb{E}[S^2_{n-1}] = \sigma^2$

To distinguish, the usual biased variance is denoted $S_n^2$ and given as

$\mathbb{E}[S^2_{n}] = \frac{n - 1}{n}\sigma^2$

Estimating $\mathbb{E}[S^2]$

$\mathbb{E}[S^2_{n-1}] = \sigma^2$

$\mathbb{E}[S^2_{n}] = \frac{n - 1}{n}\sigma^2$

In terms of sample data

$S^2_{n}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n}$

$S^2_{n-1}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n-1}$

Implications for Data Science

$S^2_{n}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n}$

$S^2_{n-1}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n-1}$

Sample mean $\overline{X}$ is an unbiased estimate of $\mu$

With a small correction, the sample variance $S^2_{n - 1}$ is an unbiased estimate of $\sigma^2$

Both of these hold independent of the distribution function

Implications for Data Science

$S^2_{n}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n}$

$S^2_{n-1}$ $= \frac{\sum_{i = 1}^n (X_i - \overline{X})^2}{n-1}$

Because there are two "values" of variance,
need to avoid confusion

In a numerical package, check which variance definition is used

The difference can be significant for small sample sizes

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Estimating $\mathrm{var}(S^2)$

We will try to rewrite such that we have only one sampled value per term

$S_{n-1}^2 = \frac{\sum_{i=1}^n (X_i - \overline{X})^2}{n-1}$

In this term, there are two sampled quantities $X_i$ and $\overline{X}$ in each sum term

This is an example of such a term:

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

Because we always want to work with independent random variables

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

Let us play the algebraic tricks we have already seen

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \sum_{i=1}^n\left(\frac{( X_i - \overline{X}) + (\overline{X} - \mu)}{\sigma}\right)^2$

$= \sum_{i = 1}^n\left(\frac{X_i - \overline{X}}{\sigma}\right)^2 + \sum_{i = 1}^n\left(\frac{\overline{X} - \mu}{\sigma}\right)^2 + 2 \left(\frac{\overline{X} - \mu}{\sigma^2}\right)\sum_{i = 1}^n(X_i - \overline{X})$

$0$

$S_{n-1}^2 = \frac{\sum_{i=1}^n (X_i - \overline{X})^2}{n-1}$

$\frac{(n-1)S_{n-1}^2}{\sigma^2} = \sum_{i=1}^n \left(\frac{X_i - \overline{X}}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

Contains only a single random variable per term

What do these terms mean?

Turns out we cannot say much for any generic population distribution

Constrain $X \sim \mathcal{N}(\mu, \sigma)$

So what would we do?

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

Constrain $X \sim \mathcal{N}(\mu, \sigma)$

Sum of squares of $n$ i.i.d. r.v.s form a standard normal distribution

Square of one r.v. form a standard normal distribution

What do these terms mean?

What's the distribution of sum of squares of standard normal variables?

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Chi Square Distribution

Let $Z_1, Z_2, \ldots, Z_n$ be independent standard normal variables and $Q = \sum_{i = 1}^n Z_i^2$

Then, what is the distribution of $Q$ ?

Let us try plotting
for $n = 1, 2, 3, \ldots$

Chi Square Distribution

$Z_1$

What will happen if we square it?

$Q = Z_1^2$

$Q$

Chi Square Distribution

$Z_1$

$Q = Z_1^2 + Z_2^2$

$Z_2$

$Q$

Chi Square Distribution

$Z_1$

$Q = Z_1^2 + Z_2^2 + Z_3^2$

$Z_2$

$Z_3$

$Q$

Chi Square Distribution

$Z_1$

$Q = Z_1^2 + Z_2^2 + Z_3^2 + Z_4^2$

$Z_2$

$Z_3$

$Z_4$

$Q$

Chi Square Distribution

$Z_1$

$Q = Z_1^2 + Z_2^2 + Z_3^2 + Z_4^2 + Z_5^2$

$Z_2$

$Z_3$

$Z_4$

$Z_5$

$Q$

Chi Square Distribution

$Q = Z_1^2 + Z_2^2 + Z_3^2 + \ldots + Z_{100}^2$

We cannot escape the normal curve can we?!

Chi Square Distribution

$Q = Z_1^2$

$Q = Z_1^2 + Z_2^2$

$Q = Z_1^2 + Z_2^2 + Z_3^2$

$Q = Z_1^2 + Z_2^2 + Z_3^2 + Z_4^2$

$Q = Z_1^2 + Z_2^2 + Z_3^2 + Z_4^2 + Z_5^2$

$Q \sim \chi^2(1)$

$Q \sim \chi^2(2)$

$Q \sim \chi^2(3)$

$Q \sim \chi^2(4)$

$Q \sim \chi^2(5)$

Chi Square Distribution

$\chi^2(k)$ has $k$ degrees of freedom

Chi Square Distribution

$Q = Z_1^2 + Z_2^2 + Z_3^2 + Z_4^2$

$Q \sim \chi^2(4)$

Chi-square distribution with $k$ degrees of freedom $\chi^2(k)$

If $Q$ is the sum of squares of $k$ independent standard normal variables then $Q \sim \chi^2(k)$

Natural follow-up questions:
What are the mean and variance of $\chi^2(k)$

Chi Square Distribution

Mean of $\chi^2(k)$

Let us start with $\chi^2(1)$

$\mu_{\chi^2(1)} = \mathbb{E}[Z^2]$ , where $Z\sim \mathcal{N}(0, 1)$

Where have we seen $\mathbb{E}[Z^2]$ ?

$\sigma^2(Z) = \mathbb{E}[Z^2] - (\mathbb{E}[Z])^2$

$1= \mathbb{E}[Z^2] - 0$

$\mathbb{E}[Z^2] = 1$

$\Rightarrow \mathbb{E}[Q] = 1$

$Q = Z_1^2$

$Q \sim \chi^2(1)$

$Z_1$

Relate to the plots

Chi Square Distribution

Mean of $\chi^2(k)$

$\mathbb{E}[Z^2] = 1$

$Q = Z_1^2 + Z_2^2$

$Q \sim \chi^2(2)$

On to mean of $\chi^2(2)$

$\mathbb{E}[Q] = \mathbb{E}[Z_1^2 + Z_2^2]$

$=\mathbb{E}[Z_1^2] + \mathbb{E}[Z_2^2]$

Since $Z_1$ and $Z_2$ are independent

$=2$

Chi Square Distribution

Mean of $\chi^2(k)$

$\mathbb{E}[Z^2] = 1$

$Q = Z_1^2 + Z_2^2$

$Q \sim \chi^2(2)$

On to mean of $\chi^2(2)$

$\mathbb{E}[Q] = \mathbb{E}[Z_1^2 + Z_2^2]$

$=\mathbb{E}[Z_1^2] + \mathbb{E}[Z_2^2]$

$=2$

$Q = Z_1^2$

$Q \sim \chi^2(1)$

Mean shifts to the right with an added degree of freedom

Chi Square Distribution

Extending this argument, we have the nice result that
Mean of $\chi^2(k) = k$

Chi Square Distribution

Variance of $\chi^2(k)$

$\sigma^2(Z^2) = \mathbb{E}[Z^4] - (\mathbb{E}[Z^2])^2$

Let us start with $\mathrm{var}(\chi^2(1))$

$\mathbb{E}[Z^4]$ recall is a moment of the normal distribution - the 4th moment or Kurtosis

We can compute it with the integral
$\int_{-\infty}^\infty x^4 f(x) dx$ , where $f(x)$ is the PDF of $\mathcal{N}(0, 1)$

$\int_{-\infty}^\infty x^4 f(x) dx$

Area under this curve is 3

$= 3 - 1 = 2$

Chi Square Distribution

Variance of $\chi^2(k)$

$\mathrm{var}(\chi^2(1)) = 2$

Consider $Q = Z_1^2 + Z_2^2$

$\mathrm{var}(Q) = \mathrm{var}(Z_1^2 + Z_2^2)$

$= \mathrm{var}(Z_1^2) + \mathrm{var}(Z_2^2)$

Since $Z_1$ and $Z_2$ are independent

$= 2 + 2 = 4$

$\mathrm{var}(\chi^2(k)) = 2k$

Chi Square Distribution

$\chi^2(k)$ is the distribution of the sum of the squares of $k$ standard normal distributions
$k$ is referred to as the degrees of freedom
Mean of $\chi^2(k)$ is $k$ and variance is $2k$

Chi Square Distribution

Implications in data science

Being a close relative to $\mathcal{N}$ , you will find that $\chi^2$ is a commonly used distribution

Applications (some in this course)

Quantify goodness of fit

Test independence of two variables

Confidence interval estimation for $\sigma^2$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Recap: Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

What's the distribution of sum of squares of standard normal variables?

Constrain $X \sim \mathcal{N}(\mu, \sigma)$

Sum of squares of $n$ i.i.d. r.v.s form a standard normal distribution

Square of one r.v. form a standard normal distribution

What do these terms mean?

$\chi^2$ distributions

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

Sum of squares of $n$ i.i.d. r.v.s form a standard normal distribution

Square of one r.v. form a standard normal distribution

$\sim \chi^2(n)$

$\sim \chi^2(1)$

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

$\sim \chi^2(n)$

$\sim \chi^2(1)$

By equating the moments of LHS and RHS and using some uniqueness theorems

$\sim \chi^2(n-1)$

$\frac{(n-1)S^2_{n - 1}}{\sigma^2}$ has a $\chi^2$ distribution with
$n - 1$ degrees of freedom

Estimating $\mathrm{var}(S^2)$

$\sum_{i=1}^n\left(\frac{ X_i - \mu}{\sigma}\right)^2$

$= \frac{(n-1)S^2_{n-1}}{\sigma^2} + \frac{n(\overline{X} - \mu)^2}{\sigma^2}$

$\frac{(n-1)S^2_{n - 1}}{\sigma^2}$ has a $\chi^2$ distribution with
$n - 1$ degrees of freedom

$S_{n-1}^2 = \frac{\sum_{i=1}^n (X_i - \overline{X})^2}{n-1}$

If we freely choose $n-1$ terms, the last one gets fixed

What's with the
degree of freedom?!

$\sum_{i=1}^n (X_i - \overline{X}) = 0$

$S^2_{n-1}$ has $n$ terms adding up, but they are not independent

Because of this we lose one degree of freedom

Estimating $\mathrm{var}(S^2)$

$\frac{(n-1)S^2_{n - 1}}{\sigma^2}$ has a $\chi^2$ distribution with
$n - 1$ degrees of freedom

Now that we have the distribution, lets check the mean value

We already knew $\mathbb{E}[S^2_{n - 1}] = \sigma^2$

$\mathbb{E}\left[\frac{(n-1)S^2_{n-1}}{\sigma^2}\right]=$ Mean of $\chi^2(n - 1)$

$\Rightarrow \frac{(n-1)}{\sigma^2}\mathbb{E}\left[S^2_{n-1}\right]= n - 1$

$\Rightarrow \mathbb{E}\left[S^2_{n-1}\right]= \sigma^2$

Estimating $\mathrm{var}(S^2)$

$\frac{(n-1)S^2_{n - 1}}{\sigma^2}$ has a $\chi^2$ distribution with
$n - 1$ degrees of freedom

Now that we have the distribution, lets estimate the variance

$\mathrm{var}\left(\frac{(n-1)S^2_{n-1}}{\sigma^2}\right)=$ $\mathrm{var}$ of $\chi^2(n - 1)$

$\Rightarrow \frac{(n-1)^2}{\sigma^4}\mathrm{var}(S^2_{n-1})=$ $2(n - 1)$

$\mathrm{var}(aX) = a^2\mathrm{var}(X)$

$\Rightarrow \mathrm{var}(S^2_{n-1})=$ $\frac{2\sigma^4}{n - 1}$

Satistics of $S^2$

The sample statistic $S^2_{n-1}$ defined as $\frac{\sum_{i = 1}^n (X - \overline{X})^2}{n-1}$ is an unbiased estimator of the population variance $\sigma^2$ , i.e., $\mathbb{E}(S^2_{n -1}) = \sigma^2$

If the population values are normally distributed, then $\frac{(n - 1)S^2_{n - 1}}{\sigma^2} \sim \chi^2(n - 1)$ and $\mathrm{var}(S^2_{n - 1}) = \frac{2\sigma^4}{n - 1}$

On to Experiments

Recall our earlier examples

On to Experiments

Recall our earlier examples

Distribution of variance of 2 values sampled from $\mathcal{N}(0, 1)$

Now we know that we should use unbiased variance $S^2_{n - 1}$

Measured distribution of $(n - 1)S^2_{n - 1}$
vs $\chi^2(1)$

Histogram vs PDF
count is normalized
bin width = 1

On to Experiments

Recall our earlier examples

Distribution of variance of 3 values sampled from $\mathcal{N}(0, 1)$

Measured distribution of $(n - 1)S^2_{n - 1}$
vs $\chi^2(2)$

On to Experiments

Recall our earlier examples

Distribution of variance of 4 values sampled from $\mathcal{N}(0, 1)$

Measured distribution of $(n - 1)S^2_{n - 1}$
vs $\chi^2(3)$

On to Experiments

Recall our earlier examples

Distribution of variance of 5 values sampled from $\mathcal{N}(0, 1)$

Measured distribution of $(n - 1)S^2_{n - 1}$
vs $\chi^2(4)$

On to Experiments

Our other example on dice

The underlying distribution is not normal and thus we cannot estimate the distribution

On to Experiments

Sample of $n$ dice throws

$n = 2$

$n = 3$

$n = 4$

$n = 5$

$n = 7$

$n = 8$

Measured
$(n - 1)S^2_{n - 1}$
vs $\chi^2(4)$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Recall Example

Proportion as a statistic measure of fraction of card sets that contain King of Diamonds

Expectation of Proportion

Let us formalise this new statistic

$p$ - Proportion of elements in population satisfying a condition

$\hat{p}$ - Proportion of elements in sample satisfying a condition

We already know about $\mathbb{E}[\overline{X}]$ and $\mathbb{E}[S]$

Can we somehow relate $\mathbb{E}[\hat{p}]$ to these?

Expectation of Proportion

Exercise

Define a random variable such that its sample mean is related to $\hat{p}$

Indicator variable

Define a random variable $X$ such that

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

Expectation of Proportion

Indicator variable

Define a random variable $X$ such that

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

How is $\hat{p}$ related to $X$ ?

Expectation of Proportion

Indicator variable

Define a random variable $X$ such that

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

How is $\hat{p}$ related to $X$ ?

$\hat{p} = \frac{X_1 + X_2 + \ldots X_n }{n}$

$= \overline{X}$

Expectation of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\hat{p} = \overline{X}$

For $X$ , what is the population mean $\mu$ ?

We are asking for the mean of the indicator variable for all elements in the population

$\mu = \frac{\sum_{i = 1}^N X_i}{N}$

$=p$

Expectation of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\hat{p} = \overline{X}$

$\mu = p$

We already know that $\mathbb{E}[\overline{X}]= \mu$

$\Rightarrow \mathbb{E}[\hat{p}] = \mathbb{E}[\overline{X}] = \mu = p$

Expectation of Proportion

$\mathbb{E}[\hat{p}] = p$

Sample statistics of proportion are unbiased estimates of the population parameter of proportion

Variance of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\mathbb{E}[\overline{X}] = p$

What is $\mathrm{var}(\overline{X})$ ?

Variance of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\mathbb{E}[\overline{X}] = p$

For the indicator variable $X$ what is the corresponding population parameter $\sigma$ ?

True

False

PMF

$p$

$1 - p$

$\sigma^2 = \frac{\sum_{i = 1}^N (X - \mu)^2}{N}$

$= p(1-p)^2 + (1-p)(0 - p)^2$

$= p(1-p)$

Variance of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\mathbb{E}[\overline{X}] = p$

For the indicator variable $X$ what is the corresponding population parameter $\sigma$ ?

True

False

PMF

$p$

$1 - p$

$\sigma$

$= \sqrt{p(1-p)}$

Variance of Proportion

Indicator variable

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

$\mathbb{E}[\overline{X}] = p$

$\sigma$

$= \sqrt{p(1-p)}$

What is $\mathrm{var}(\overline{X})$ ?

$\mathrm{var}(\overline{X}) = \frac{\sigma^2}{n}$

$= \frac{p(1-p)}{n}$

$= \mathrm{var}(\hat{p})$

Variance of Proportion

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

What is the maximum value of $\mathrm{var}(\hat{p})$ for a given $n$ ?

True

False

PMF

$p = 0.25 \Rightarrow \mathrm{var}(\hat{p}) = \frac{0.1875}{n}$

Variance of Proportion

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

What is the maximum value of $\mathrm{var}(\hat{p})$ for a given $n$ ?

True

False

PMF

$p = 0.5 \Rightarrow \mathrm{var}(\hat{p}) = \frac{0.25}{n}$

Variance of Proportion

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

What is the maximum value of $\mathrm{var}(\hat{p})$ for a given $n$ ?

True

False

PMF

$p = 0.75 \Rightarrow \mathrm{var}(\hat{p}) = \frac{0.1875}{n}$

$\mathrm{var}(\hat{p})$ is highest when $p = 0.5$

Variance of Proportion

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

Let us take up our original problem

Recall that an outcome is a random set of 13 cards from a standard deck

$\mathbb{E}[\hat{p}] = p = \frac{1}{4}$

If our samples consist of $n = 5$ such sets of 13 cards, what are statistics of $\hat{p}$

$p = 1/4$

$\mathrm{var}(\hat{p}) = \frac{p(1 - p)}{n} = \frac{3}{80}$

Variance of Proportion

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

Let us take up our original problem

Recall that an outcome is a random set of 13 cards from a standard deck

$\mathbb{E}[\hat{p}] = p = \frac{1}{4}$

If our samples consist of $n = 20$ such sets of 13 cards, what are statistics of $\hat{p}$

$p = 1/4$

$\mathrm{var}(\hat{p}) = \frac{p(1 - p)}{n} = \frac{3}{320}$

Satistics of $\hat{p}$

$= 0$ , otherwise

$X_i = 1$ , if $i$ th element of sample satisfies condition

The sample statistic $\hat{p}$ as defined as the proportion of the population satisfying a boolean condition is related to the indicator random variable

$\mathbb{E}[\hat{p}] = p$

$\mathrm{var}(\hat{p})$

$= \frac{p(1-p)}{n}$

The statistics of $\hat{p}$ are given as

Our Roadmap

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Central Limit Theorem

Given $\mu, \sigma$

$E(S^2)$ , $Var(S^2)$

Chi Square Distribution

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$

Our Roadmap

Given μ,σ\mu, \sigmaμ,σ

E(Xˉ)E(\bar{X})E(Xˉ), Var(Xˉ)Var(\bar{X})Var(Xˉ)

Central Limit Theorem

Given μ,σ\mu, \sigmaμ,σ

E(S)E(S)E(S), Var(S)Var(S)Var(S)

Chi Square Distribution

Given ppp

E(p^)E(\hat{p})E(p^​), Var(p^)Var(\hat{p})Var(p^​)

Given $\mu, \sigma$

$E(\bar{X})$ , $Var(\bar{X})$

Given $\mu, \sigma$

$E(S)$ , $Var(S)$

Given $p$

$E(\hat{p})$ , $Var(\hat{p})$