CS6015: Linear Algebra and Random Processes
Lecture 29: Conditional probabilities, multiplication rule, total probability theorem, Bayes' theorem, independent events
Learning Objectives
What is the multiplication rule of probability?
What are independent events?
What is the total probability theorem?
What is Bayes' theorem?
What are conditional probabilities?
What is conditional independence?
Conditional Probabilities
Change in belief
Before start of play: What is the chance of India winning?
(assume fair playing conditions & equally good teams)
India scores 395 batting first: What is the chance of India winning?
> 0.5
0.5
Change in belief
What exactly happened here?
(assume fair playing conditions & equally good teams)
A: event that India will win
B: India scores 395 runs
P(A) changes once we know that event B has occurred
P(A|B)
\neq P(A)
Change in belief
What is the probability that a randomly selected person is healthy (not infected)?
A: event that a person is healthy
B: event that the person has COVID-19 symptoms
10% of the population is infected
P(A) = 0.9
P(A|B) \neq P(A)
is called the conditional probability of the event A given the event B
P(A|B)
The definition of P(B|A)
P(A) = \frac{5}{36}
| (1 , 1) | (1 , 2) | (1 , 3) | (1 , 4) | (1 , 5) | (1 , 6) |
|---|---|---|---|---|---|
| (2, 1) | (2, 2) | (2, 3) | (2, 4) | (2, 5) | (2, 6) |
| (3, 1) | (3, 2) | (3, 3) | (3, 4) | (3, 5) | (3, 6) |
| (4, 1) | (4, 2) | (4, 3) | (4, 4) | (4, 5) | (4, 6) |
| (5, 1) | (5, 2) | (5, 3) | (5, 4) | (5, 5) | (5, 6) |
| (6, 1) | (6, 2) | (6, 3) | (6, 4) | (6, 5) | (6, 6) |
What is the probability that the sum is 8?
The definition of P(B|A)
P(A|B) = \frac{1}{6}
| (1 , 1) | (1 , 2) | (1 , 3) | (1 , 4) | (1 , 5) | (1 , 6) |
|---|---|---|---|---|---|
| (2, 1) | (2, 2) | (2, 3) | (2, 4) | (2, 5) | (2, 6) |
| (3, 1) | (3, 2) | (3, 3) | (3, 4) | (3, 5) | (3, 6) |
| (4, 1) | (4, 2) | (4, 3) | (4, 4) | (4, 5) | (4, 6) |
| (5, 1) | (5, 2) | (5, 3) | (5, 4) | (5, 5) | (5, 6) |
| (6, 1) | (6, 2) | (6, 3) | (6, 4) | (6, 5) | (6, 6) |
What is the probability that the sum is 8 given that the first dice shows a 4?
A: sum is 8
B: first dice shows a 4
The definition of P(B|A)
P(B) = \frac{6}{36}
A: sum is 8
B: first dice shows a 4
\Omega
(6, 2)
(1, 1)~(1,2)~(1,3)~(1,4)~(1,5)~(1,6)
(4, 4)
(3, 5)
(5, 3)
(4, 1)~~(4,2)~~(4,3)~~(4,5)~~(4,6)
(2, 6)
(2, 1)~(2,2)~(2,3)~(2,4)~(2,5)
(3, 1)~(3,2)~(3,3)~(3,4)~(3,6)
(5, 1)~(5,2)~(5,4)~(5,5)~(5,6)
(6, 1)~(6,3)~(6,4)~(6,5)~(6,6)
A
B
A \cap B
P(A\cap B) = \frac{1}{36}
P(A|B) = \frac{P(A\cap B)}{P(B)} = \frac{\frac{1}{36}}{\frac{6}{36}} = \frac{1}{6}
is called the conditional probability of the event A given the event B
P(A|B)
P(A|B) = \frac{P(A\cap B)}{P(B)}
conditional probability
regular probabilities
(we already know how to compute these)
Examples
I am thinking of a two digit number. Suppose I tell you that at least one of the two digits is even then what is the probability that both are even?
A: event that both digits are even
B: event that at least one digit is even
P(A) = \frac{20}{90} = \frac{2}{9}
10, 11, 12, 13, 14, 15, ...., ..., ..., ..., ..., ..., ..., ..., ..., 94, 95, 96, 97, 98, 99
(all equally likely)
but we are interested in P(A|B)
Examples
I am thinking of a two digit number. Suppose I tell you that at least one of the two digits is even then what is the probability that both are even?
A: event that both digits are even
B: event that at least one digit is even
P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{20}{90}}{\frac{65}{90}} = \frac{4}{13}
10, 11, 12, 13, 14, 15, ...., ..., ..., ..., ..., ..., ..., ..., ..., 94, 95, 96, 97, 98, 99
(all equally likely)
A
B
Examples
60% of the students in a class opt for ML. 20% of the students opt for both ML and DL. Given that a student has opted for ML what is the probability that she has also opted for DL?
A: event that student has opted for DL
B: event that the student has opted for ML
P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{0.2}{0.6} = \frac{1}{3}
A
B
\Omega
Do conditional probabilities satisfy the axioms of probability?
Axioms of Probability
P(A|B) = \frac{P(A\cap B)}{P(B)} \geq 0
: ratio of two probabilities
P(\Omega|A)= \frac{P(\Omega \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1
P(A_1 \cup A_2|B) = \frac{P((A_1 \cup A_2) \cap B)}{P(B)}
= \frac{P(A_1 \cap B)}{P(B)} + \frac{P(A_2 \cap B)}{P(B)}
= P(A_1|B) + P(A_2|B)
B
A_1
\Omega
A_2
= \frac{P((A_1 \cap B) \cup (A_2 \cap B)}{P(B)}
The multiplication principle
The chain rule of probability
P(A|B) = \frac{P(A\cap B)}{P(B)}
P(B|A) = \frac{P(B\cap A)}{P(A)}
\therefore P(A\cap B) = P(A|B)\cdot P(B)
\therefore P(B\cap A) = P(B|A)\cdot P(A)
\therefore P(A\cap B) = P(A|B)\cdot P(B) = P(B|A)\cdot P(A)
The chain rule of probability
+
B
\Omega
A
A: event that a person is infected
B: event that the test result is positive
A\cap B
A\cap B^\mathsf{c}
A^\mathsf{c}\cap B
A^\mathsf{c}\cap B^\mathsf{c}
The chain rule of probability
+
B
\Omega
A
roughly 10% of the population is infected
for an infected person the test shows a negative result in 1% of the cases
Facts:
P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
for an healthy person the test shows a positive result in 5% of the cases
The chain rule of probability
+
B
\Omega
A
Facts:
P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
A
A \cap B
A \cap B^\mathsf{c}
A^\mathsf{c} \cap B
A^\mathsf{c} \cap B^\mathsf{c}
A^\mathsf{c}
B|A
B^\mathsf{c}|A
B|A^\mathsf{c}
B^\mathsf{c}|A^\mathsf{c}
A: infected
B: test positive
The chain rule of probability
for n events
P(A \cap B \cap C) = P((A \cap B) \cap C)
Let~(A \cap B)=X
\therefore P(A \cap B \cap C) = P(X \cap C)
\therefore P(A \cap B \cap C) = P(X)\cdot P(C|X)
\therefore P(A \cap B \cap C) = P(A\cap B)\cdot P(C|A \cap B)
\therefore P(A \cap B \cap C) = P(A)\cdot P(B|A)\cdot P(C|A \cap B)
The chain rule of probability
for n events
P(A \cap B \cap C \cap D)
= P(A)\cdot P(B|A)\cdot P(C|A \cap B)\cdot P(D|A \cap B \cap C)
P(A_1 \cap A_2 \cap \dots \cap A_n)
= P(A_1)\prod_{i=2}^{n} P(A_i|A_1\dots A_{i-1})
Examples
Suppose you draw 3 cards one by one without replacement. What is the probability that all the 3 cards are aces?
Using counting principles:
p = \frac{4 \choose 3}{52 \choose 3} = \frac{\frac{4!}{1!~3!}}{\frac{52!}{49!~3!}} = \frac{4*3*2}{52*51*50}
Examples
Suppose you draw 3 cards one by one without replacement. What is the probability that all the 3 cards are aces?
Using chain rule:
A_i
:the event that the i-th card is an ace
P(A_1\cap A_2\cap A_3)
= P(A_1)\cdot P(A_2|A_1)\cdot P(A_3|A_2 \cap A_1)
P(A_1)
= \frac{4}{52}
P(A_2|A_1)
= \frac{3}{51}
P(A_3|A_1\cap A_2)
= \frac{2}{50}
= \frac{4*3*2}{52*51*50}
Total Probability Theorem
Total Probability Theorem
A_1
A_5
A_4
A_3
A_2
A_6
A_7
B
A_1, A_2, \cdots A_n
\Omega
partition
\Omega
A_1 \cup A_2, \cup \dots \cup A_n = \Omega
A_i \cap A_j = \phi~\forall i \neq j
B = (B \cap A_1) \cup (B \cap A_2) \cup \dots \cup (B \cap A_n)
P(B) = P(B \cap A_1) + P(B \cap A_2) + \dots + P(B \cap A_n)
P(B) = \sum_{i=1}^{n} P (A_i) \cdot P(B | A_i)
P(B)
= P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + \dots + P(A_n)\cdot P(B|A_n)
Examples
+
B
\Omega
A
Facts:
P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
A: infected
B: test positive
P(B) = ?
P(B) = P(A)P(B|A) + P(A^\mathsf{c})(B(|A^\mathsf{c})
\therefore P(B) = 0.1*0.99 + 0.9*0.05 =0.144
Examples
What is the probability that he will come out alive?
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(B^\mathsf{c}) = ?
B : monster encountered
i-th path taken
A_i:
= P(A_1)P(B^\mathsf{c}|A_1) + P(A_2)P(B^\mathsf{c}|A_2) + P(A_3)P(B^\mathsf{c}|A_3)
= \frac{1}{3}\cdot0.7 + \frac{1}{3}\cdot0.4 + \frac{1}{3}\cdot0.25 = 0.45
Bayes' Theorem
Examples
If he does not come out alive what is the probability that he took path A1?
P(A_1|B) = ?
B : monster encountered
i-th path taken
A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1 \cap B)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\
P(A_1 \cap B) = P(B)P(A_1|B)
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
Examples
If he does not come out alive what is the probability that he took path A1?
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_1|B) = ?
B : monster encountered
i-th path taken
A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1)P(B|A_1)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\
P(A_1 \cap B) = P(B)P(A_1|B)
= 0.182
Examples
If he does not come out alive what is the probability that he took path A3?
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_3|B) = ?
B : monster encountered
i-th path taken
A_i:
P(A_3|B) = \frac{P(A_3 \cap B)}{P(B)}
P(A_3|B) = \frac{P(A_3)P(B|A_3)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_3 \cap B) = P(A_3)P(B|A_3) \\
P(A_3 \cap B) = P(B)P(A_3|B)
= 0.45
Breaking it down
Exploit Multiplication Rule
P(A_1)P(B|A_1) = P(B)P(A_1|B)
Exploit Total Probability Theorem
P(B) = P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)
Exploit the known probabilities
P(A_1|B) = \frac{P(A_1)\cdot P(B|A_1) }{\sum_{i=1}^nP(A_i)P(B|A_i)}
Bayes' Theorem
Breaking it down: Example 1
A : Ship 1 sends a signal 1
B : Ship 2 receives a signal 1
P(A) = 0.01
P(B|A) = 0.95
P(B|A^\mathsf{c}) = 0.05
P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.18
A
A^\mathsf{c}
10000
9900
100
495
95
Breaking it down: Example 2
A
A^\mathsf{c}
10000
9000
1000
450
990
+
B
\Omega
A
Facts:
P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
A: infected
B: test positive
P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.6875
Bayes' Theorem: 3 forms
P(A_i|B) = \frac{P(A_i)\cdot P(B|A_i) }{\sum_{j=1}^{n}P(A_j)P(B|A_j)}
P(A|B) = \frac{P(A \cap B)}{P(B)}
P(A|B) = \frac{P(A)\cdot P(B|A) }{P(B)}
Independent Events
Do we always update our beliefs
A: I had a sandwich for breakfast
If A occurs will you update your belief about B ?
What do you call such events?
B: It will rain today
No
Independent events
Example
50 girls and 70 boys in a class. Of these, 35 girls and 49 boys are good at Maths. If I tell you that a student is very good at Maths what is the probability that she is a girl?
Facts:
P(A) = \frac{50}{50+70} = \frac{5}{12}
P(B|A) = \frac{35}{50} = \frac{7}{10}
A: student is girl
B: student is good at Maths
P(A^\mathsf{c}) = \frac{7}{12}
P(B|A^\mathsf{c}) = \frac{49}{70} = \frac{7}{10}
P(A|B) = ?
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
P(B) = P(B|A)P(A) + P(B|A')P(A') = \frac{7}{10}
= \frac{\frac{7}{10}\frac{5}{12}}{\frac{7}{10}} = \frac{5}{12}
Tow events A and B are independent if
P(A|B) = P(B)
P(A \cap B) = P(B)\cdot P(A|B)
Tow events A and B are independent if
P(A\cap B) = P(A)\cdot P(B)
Example
A: first toss results in a head
B: exactly 2 tosses result in heads
P(A) = \frac{4}{8}
Are A and B independent ?
| H | H | H | * | ||
| H | H | T | * | * | * |
| H | T | H | * | * | * |
| H | T | T | * | ||
| T | H | H | * | ||
| T | H | T | |||
| T | T | H | |||
| T | T | T |
A
B
A \cap B
P(B) = \frac{3}{8}
P(A \cap B) = \frac{2}{8}
P(A \cap B) \neq P(A)P(B)
Example
Are A and B independent ?
A: sum is 7
B: second dice shows an even number
A = \{ (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) \}
B = \{(1,2), (1,4), (1,6), (2,2), (2,4), (2,6),(3,2), (3,4), (3,6), \newline(4,2), (4,4), (4,6), (5,2), (5,4), (5,6), (6,2), (6,4), (6,6)\}
A \cap B = \{(1,6), (3,4), (5,2) \}
P(A) = \frac{6}{36} = \frac{1}{6}
P(A \cap B) = \frac{3}{36}= \frac{1}{12}
P(B) = \frac{18}{36} = \frac{1}{2}
\therefore P(A \cap B) = P(A)\cdot P(B)
Example
A: first answer is correct
B: second answer is correct
A quiz has two multiple choice Qs. The first Q has 4 choices of which 1 is correct and the second Q has 3 choices of which 1 is correct. If a student randomly guesses the answers what is the probability that he will answer both Qs correctly?
P(A) = \frac{1}{4}
P(B) = \frac{1}{3}
P(A \cap B) = P(A)\cdot P(B) = \frac{1}{12}
Independence: n events
We say that events are pairwise independent if
A_1, A_2, A_3, \dots, A_n
P(A_i \cap A_j) = P(A_i)\cdot P(A_j)~\forall~i\neq j
We say that events are mutually independent or independent if for all subsets
P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)
A_1, A_2, A_3, \dots, A_n
I \subset \{1,2,3,\dots,n\}
\{1,2,3\}
n = 3
\{1,2\}, \{1, 3\}, \{2,3\}, \{1, 2, 3\}
P(A_1 \cap A_2 ) = P(A_1)\cdot P(A_2)
P(A_1 \cap A_3 ) = P(A_1)\cdot P(A_3)
P(A_2 \cap A_3 ) = P(A_2)\cdot P(A_3)
P(A_1 \cap A_2 \cap A_3 )
= P(A_1)\cdot P(A_2)\cdot P(A_3)
Conditional independence
Conditional independence
A: it rained last night
B: forgot to turn off the sprinkler
C: a muddy puddle in the morning
Are A and B independent?
Yes
Are A and B independent given C?
No
Why? Now, if you know that A has not occurred then you will be certain that B has occurred
Conditional independence
A
B
C
One effect, two possible causes:
Causes may be initially independent but become dependent once you have some knowledge about the effect
A \perp \!\!\! \perp B
A \not\!\perp\!\!\!\perp B~|~C
Conditional independence
Is L independent of E?
No
G
D
I
L
E
Difficulty
Intelligence
Grade
GATE
Letter
Why? If you know about E, you can infer I and hence G and hence L
Conditional independence
Is L independent of E given G?
Yes
G
D
I
L
E
Difficulty
Intelligence
Grade
GATE
Letter
Why? You already know G so E does not add any new information
Conditional independence
Tow events A and B are independent if
P(A\cap B) = P(A)\cdot P(B)
Tow events A and B are conditionally independent given an event C if
P(A\cap B|C) = P(A|C)\cdot P(B|C)
Summary
Putting it all together
Conditional Probability
Compute~P(B|A)~using~P(A\cap B)~and~P(A)
Multiplication Rule
Compute~P(A\cap B)~using~P(A)~and~P(B|A)
Total Probability Theorem
Compute~P(B)~using~P(A_1), P(A_2), \dots, P(A_n)
and~P(B|A_1), P(B|A_2), \dots, P(B|A_n)
Bayes' Theorem
Compute~P(A|B)~using
P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)
Independent events
