CS6015: Linear Algebra and Random Processes

Lecture 29:  Conditional probabilities, multiplication rule, total probability theorem, Bayes' theorem, independent events

Learning Objectives

What is the multiplication rule of probability?

What are independent events?

What is the total probability theorem?

What is Bayes' theorem?

What are conditional probabilities?

What is conditional independence?

Conditional Probabilities

Change in belief

Before start of play: What is the chance of India winning?

(assume fair playing conditions & equally good teams)

India scores 395 batting first: What is the chance of India winning?

> 0.5
0.5

Change in belief

What exactly happened here?

(assume fair playing conditions & equally good teams)

A: event that India will win

B: India scores 395 runs

P(A) changes once we know that event B has occurred

P(A|B)
\neq P(A)

Change in belief

What is the probability that a randomly selected person is healthy (not infected)?

A: event that a person is healthy

B: event that the person has COVID-19 symptoms

10% of the population is infected

P(A) = 0.9
P(A|B) \neq P(A)

                is called the conditional probability of the event A given the event B

P(A|B)

The definition of P(B|A)

P(A) = \frac{5}{36}
(1 , 1) (1 , 2) (1 , 3) (1 , 4) (1 , 5) (1 , 6)
(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)

What is the probability that the sum is 8?

The definition of P(B|A)

P(A|B) = \frac{1}{6}
(1 , 1) (1 , 2) (1 , 3) (1 , 4) (1 , 5) (1 , 6)
(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)

What is the probability that the sum is 8 given that the first dice shows a 4?

A: sum is 8

B: first dice shows a 4

The definition of P(B|A)

P(B) = \frac{6}{36}

A: sum is 8

B: first dice shows a 4

\Omega
(6, 2)
(1, 1)~(1,2)~(1,3)~(1,4)~(1,5)~(1,6)
(4, 4)
(3, 5)
(5, 3)
(4, 1)~~(4,2)~~(4,3)~~(4,5)~~(4,6)
(2, 6)
(2, 1)~(2,2)~(2,3)~(2,4)~(2,5)
(3, 1)~(3,2)~(3,3)~(3,4)~(3,6)
(5, 1)~(5,2)~(5,4)~(5,5)~(5,6)
(6, 1)~(6,3)~(6,4)~(6,5)~(6,6)
A
B
A \cap B
P(A\cap B) = \frac{1}{36}
P(A|B) = \frac{P(A\cap B)}{P(B)} = \frac{\frac{1}{36}}{\frac{6}{36}} = \frac{1}{6}

                is called the conditional probability of the event A given the event B

P(A|B)
P(A|B) = \frac{P(A\cap B)}{P(B)}

conditional probability

regular probabilities

(we already know how to compute these)

Examples

I am thinking of a two digit number. Suppose I tell you that at least one of the two digits is even then what is the probability that both are even?

A: event that both digits are even

B: event that at least one digit is even

P(A) = \frac{20}{90} = \frac{2}{9}

10, 11, 12, 13, 14, 15, ...., ..., ..., ..., ..., ..., ..., ..., ..., 94, 95, 96, 97, 98, 99

(all equally likely)

but we are interested in P(A|B)

Examples

I am thinking of a two digit number. Suppose I tell you that at least one of the two digits is even then what is the probability that both are even?

A: event that both digits are even

B: event that at least one digit is even

P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{20}{90}}{\frac{65}{90}} = \frac{4}{13}

10, 11, 12, 13, 14, 15, ...., ..., ..., ..., ..., ..., ..., ..., ..., 94, 95, 96, 97, 98, 99

(all equally likely)

A
B

Examples

60% of the students in a class opt for ML. 20% of the students opt for both ML and DL. Given that a student has opted for ML what is the probability that she has also opted for DL?

A: event that student has opted for DL

B: event that the student has opted for ML

P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{0.2}{0.6} = \frac{1}{3}
A
B
\Omega

Do conditional probabilities satisfy the axioms of probability?

Axioms of Probability

P(A|B) = \frac{P(A\cap B)}{P(B)} \geq 0

: ratio of two probabilities

P(\Omega|A)= \frac{P(\Omega \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1
P(A_1 \cup A_2|B) = \frac{P((A_1 \cup A_2) \cap B)}{P(B)}
= \frac{P(A_1 \cap B)}{P(B)} + \frac{P(A_2 \cap B)}{P(B)}
= P(A_1|B) + P(A_2|B)
B
A_1
\Omega
A_2
= \frac{P((A_1 \cap B) \cup (A_2 \cap B)}{P(B)}

The multiplication principle

The chain rule of probability

P(A|B) = \frac{P(A\cap B)}{P(B)}
P(B|A) = \frac{P(B\cap A)}{P(A)}
\therefore P(A\cap B) = P(A|B)\cdot P(B)
\therefore P(B\cap A) = P(B|A)\cdot P(A)
\therefore P(A\cap B) = P(A|B)\cdot P(B) = P(B|A)\cdot P(A)

The chain rule of probability

+
B
\Omega
A

A: event that a person is infected

B: event that the test result is positive

A\cap B
A\cap B^\mathsf{c}
A^\mathsf{c}\cap B
A^\mathsf{c}\cap B^\mathsf{c}

The chain rule of probability

+
B
\Omega
A

roughly 10% of the population is infected

for an infected person the test shows a negative result in 1% of the cases

Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

for an healthy person the test shows a positive result in 5% of the cases

The chain rule of probability

+
B
\Omega
A

Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
A
A \cap B
A \cap B^\mathsf{c}
A^\mathsf{c} \cap B
A^\mathsf{c} \cap B^\mathsf{c}
A^\mathsf{c}
B|A
B^\mathsf{c}|A
B|A^\mathsf{c}
B^\mathsf{c}|A^\mathsf{c}

A:  infected

B: test positive

The chain rule of probability

for n events

P(A \cap B \cap C) = P((A \cap B) \cap C)
Let~(A \cap B)=X
\therefore P(A \cap B \cap C) = P(X \cap C)
\therefore P(A \cap B \cap C) = P(X)\cdot P(C|X)
\therefore P(A \cap B \cap C) = P(A\cap B)\cdot P(C|A \cap B)
\therefore P(A \cap B \cap C) = P(A)\cdot P(B|A)\cdot P(C|A \cap B)

The chain rule of probability

for n events

P(A \cap B \cap C \cap D)
= P(A)\cdot P(B|A)\cdot P(C|A \cap B)\cdot P(D|A \cap B \cap C)
P(A_1 \cap A_2 \cap \dots \cap A_n)
= P(A_1)\prod_{i=2}^{n} P(A_i|A_1\dots A_{i-1})

Examples

Suppose you draw 3 cards one by one without replacement. What is the probability that all the 3 cards are aces?

Using counting principles:

p = \frac{4 \choose 3}{52 \choose 3} = \frac{\frac{4!}{1!~3!}}{\frac{52!}{49!~3!}} = \frac{4*3*2}{52*51*50}

Examples

Suppose you draw 3 cards one by one without replacement. What is the probability that all the 3 cards are aces?

Using chain rule:

A_i

:the event that the i-th card is an ace

P(A_1\cap A_2\cap A_3)
= P(A_1)\cdot P(A_2|A_1)\cdot P(A_3|A_2 \cap A_1)
P(A_1)
= \frac{4}{52}
P(A_2|A_1)
= \frac{3}{51}
P(A_3|A_1\cap A_2)
= \frac{2}{50}
= \frac{4*3*2}{52*51*50}

Total Probability Theorem

Total Probability Theorem

A_1
A_5
A_4
A_3
A_2
A_6
A_7
B
A_1, A_2, \cdots A_n
\Omega

partition

\Omega
A_1 \cup A_2, \cup \dots \cup A_n = \Omega
A_i \cap A_j = \phi~\forall i \neq j
B = (B \cap A_1) \cup (B \cap A_2) \cup \dots \cup (B \cap A_n)
P(B) = P(B \cap A_1) + P(B \cap A_2) + \dots + P(B \cap A_n)
P(B) = \sum_{i=1}^{n} P (A_i) \cdot P(B | A_i)
P(B)
= P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + \dots + P(A_n)\cdot P(B|A_n)

Examples

+
B
\Omega
A

Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

A:  infected

B: test positive

P(B) = ?
P(B) = P(A)P(B|A) + P(A^\mathsf{c})(B(|A^\mathsf{c})
\therefore P(B) = 0.1*0.99 + 0.9*0.05 =0.144

Examples

What is the probability that he will come out alive?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(B^\mathsf{c}) = ?

B   :  monster encountered

      i-th path taken

A_i:
= P(A_1)P(B^\mathsf{c}|A_1) + P(A_2)P(B^\mathsf{c}|A_2) + P(A_3)P(B^\mathsf{c}|A_3)
= \frac{1}{3}\cdot0.7 + \frac{1}{3}\cdot0.4 + \frac{1}{3}\cdot0.25 = 0.45

Bayes' Theorem

Examples

If he does not come out alive what is the probability that he took path A1?

P(A_1|B) = ?

B   :  monster encountered

      i-th path taken

A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1 \cap B)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\ P(A_1 \cap B) = P(B)P(A_1|B)
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75

Examples

If he does not come out alive what is the probability that he took path A1?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_1|B) = ?

B   :  monster encountered

      i-th path taken

A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1)P(B|A_1)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\ P(A_1 \cap B) = P(B)P(A_1|B)
= 0.182

Examples

If he does not come out alive what is the probability that he took path A3?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_3|B) = ?

B   :  monster encountered

      i-th path taken

A_i:
P(A_3|B) = \frac{P(A_3 \cap B)}{P(B)}
P(A_3|B) = \frac{P(A_3)P(B|A_3)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_3 \cap B) = P(A_3)P(B|A_3) \\ P(A_3 \cap B) = P(B)P(A_3|B)
= 0.45

Breaking it down

Exploit Multiplication Rule

P(A_1)P(B|A_1) = P(B)P(A_1|B)

Exploit Total Probability Theorem

P(B) = P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)

Exploit the known probabilities

P(A_1|B) = \frac{P(A_1)\cdot P(B|A_1) }{\sum_{i=1}^nP(A_i)P(B|A_i)}

Bayes' Theorem

Breaking it down: Example 1

A   :  Ship 1 sends a signal 1

B  :  Ship 2 receives a signal 1

P(A) = 0.01
P(B|A) = 0.95
P(B|A^\mathsf{c}) = 0.05
P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.18
A
A^\mathsf{c}
10000
9900
100
495
95

Breaking it down: Example 2

A
A^\mathsf{c}
10000
9000
1000
450
990
+
B
\Omega
A

Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

A:  infected

B: test positive

P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.6875

Bayes' Theorem: 3 forms

P(A_i|B) = \frac{P(A_i)\cdot P(B|A_i) }{\sum_{j=1}^{n}P(A_j)P(B|A_j)}
P(A|B) = \frac{P(A \cap B)}{P(B)}
P(A|B) = \frac{P(A)\cdot P(B|A) }{P(B)}

Independent Events

Do we always update our beliefs

A: I had a sandwich for breakfast

If A occurs will you update your belief about B ?

What do you call such events?

B: It will rain today

No

Independent events

Example

50 girls and 70 boys in a class. Of these, 35 girls and 49 boys are good at Maths. If I tell you that a student is very good at Maths what is the probability that she is a girl?

Facts:

P(A) = \frac{50}{50+70} = \frac{5}{12}
P(B|A) = \frac{35}{50} = \frac{7}{10}

A:  student is girl

B: student is good at Maths

P(A^\mathsf{c}) = \frac{7}{12}
P(B|A^\mathsf{c}) = \frac{49}{70} = \frac{7}{10}
P(A|B) = ?
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
P(B) = P(B|A)P(A) + P(B|A')P(A') = \frac{7}{10}
= \frac{\frac{7}{10}\frac{5}{12}}{\frac{7}{10}} = \frac{5}{12}

Tow events A and B are independent if

P(A|B) = P(B)
P(A \cap B) = P(B)\cdot P(A|B)

Tow events A and B are independent if

P(A\cap B) = P(A)\cdot P(B)

Example

A:  first toss results in a head

B: exactly 2 tosses result in heads

P(A) = \frac{4}{8}

Are A and B independent ?

H H H *
H H T * * *
H T H * * *
H T T *
T H H *
T H T
T T H
T T T
A
B
A \cap B
P(B) = \frac{3}{8}
P(A \cap B) = \frac{2}{8}
P(A \cap B) \neq P(A)P(B)

Example

Are A and B independent ?

A: sum is 7

B: second dice shows an even number

A = \{ (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) \}
B = \{(1,2), (1,4), (1,6), (2,2), (2,4), (2,6),(3,2), (3,4), (3,6), \newline(4,2), (4,4), (4,6), (5,2), (5,4), (5,6), (6,2), (6,4), (6,6)\}
A \cap B = \{(1,6), (3,4), (5,2) \}
P(A) = \frac{6}{36} = \frac{1}{6}
P(A \cap B) = \frac{3}{36}= \frac{1}{12}
P(B) = \frac{18}{36} = \frac{1}{2}
\therefore P(A \cap B) = P(A)\cdot P(B)

Example

A: first answer is correct

B: second answer is correct

A quiz has two multiple choice Qs. The first Q has 4 choices of which 1 is correct and the second Q has 3 choices of which 1 is correct. If a student randomly guesses the answers what is the probability that he will answer both Qs correctly?

P(A) = \frac{1}{4}
P(B) = \frac{1}{3}
P(A \cap B) = P(A)\cdot P(B) = \frac{1}{12}

Independence: n events

We say that events                                      are pairwise independent if

A_1, A_2, A_3, \dots, A_n
P(A_i \cap A_j) = P(A_i)\cdot P(A_j)~\forall~i\neq j

We say that events                                      are mutually independent or independent if for all subsets

P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)
A_1, A_2, A_3, \dots, A_n
I \subset \{1,2,3,\dots,n\}
\{1,2,3\}
n = 3
\{1,2\}, \{1, 3\}, \{2,3\}, \{1, 2, 3\}
P(A_1 \cap A_2 ) = P(A_1)\cdot P(A_2)
P(A_1 \cap A_3 ) = P(A_1)\cdot P(A_3)
P(A_2 \cap A_3 ) = P(A_2)\cdot P(A_3)
P(A_1 \cap A_2 \cap A_3 )
= P(A_1)\cdot P(A_2)\cdot P(A_3)

Conditional independence

Conditional independence

A: it rained last night

B: forgot to turn off the sprinkler

C: a muddy puddle in the morning

Are A and B independent?

Yes

Are A and B independent given C?

No

Why? Now, if you know that A has not occurred then you will be certain that B has occurred

Conditional independence

A

B

C

One effect, two possible causes:

Causes may be initially independent but become dependent once you have some knowledge about the effect

A \perp \!\!\! \perp B
A \not\!\perp\!\!\!\perp B~|~C

Conditional independence

Is L independent of E?

No

G

D

I

L

E

Difficulty

Intelligence

Grade

GATE

Letter

Why? If you know about E, you can infer I and hence G and hence L

Conditional independence

Is L independent of E given G?

Yes

G

D

I

L

E

Difficulty

Intelligence

Grade

GATE

Letter

Why? You already know G so E does not add any new information

Conditional independence

Tow events A and B are independent if

P(A\cap B) = P(A)\cdot P(B)

Tow events A and B are conditionally independent given an event C if

P(A\cap B|C) = P(A|C)\cdot P(B|C)

Summary

Putting it all together

Conditional Probability

Compute~P(B|A)~using~P(A\cap B)~and~P(A)

Multiplication Rule

Compute~P(A\cap B)~using~P(A)~and~P(B|A)

Total Probability Theorem

Compute~P(B)~using~P(A_1), P(A_2), \dots, P(A_n)
and~P(B|A_1), P(B|A_2), \dots, P(B|A_n)

Bayes' Theorem

Compute~P(A|B)~using
P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)

Independent events

Conditional independence

Learning Objectives

(achieved)

What is the multiplication rule of probability?

What are independent events?

What is the total probability theorem?

What is Bayes' theorem?

What are conditional probabilities?

What is conditional independence?