# CS6015: Linear Algebra and Random Processes

## Lecture 29:  Conditional probabilities, multiplication rule, total probability theorem, Bayes' theorem, independent events

> 0.5
0.5

P(A|B)
\neq P(A)

P(A) = 0.9
P(A|B) \neq P(A)

P(A|B)

### The definition of P(B|A)

P(A) = \frac{5}{36}
(1 , 1) (1 , 2) (1 , 3) (1 , 4) (1 , 5) (1 , 6)
(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)

### The definition of P(B|A)

P(A|B) = \frac{1}{6}
(1 , 1) (1 , 2) (1 , 3) (1 , 4) (1 , 5) (1 , 6)
(2, 1) (2, 2) (2, 3) (2, 4) (2, 5) (2, 6)
(3, 1) (3, 2) (3, 3) (3, 4) (3, 5) (3, 6)
(4, 1) (4, 2) (4, 3) (4, 4) (4, 5) (4, 6)
(5, 1) (5, 2) (5, 3) (5, 4) (5, 5) (5, 6)
(6, 1) (6, 2) (6, 3) (6, 4) (6, 5) (6, 6)

### The definition of P(B|A)

P(B) = \frac{6}{36}

### B: first dice shows a 4

\Omega
(6, 2)
(1, 1)~(1,2)~(1,3)~(1,4)~(1,5)~(1,6)
(4, 4)
(3, 5)
(5, 3)
(4, 1)~~(4,2)~~(4,3)~~(4,5)~~(4,6)
(2, 6)
(2, 1)~(2,2)~(2,3)~(2,4)~(2,5)
(3, 1)~(3,2)~(3,3)~(3,4)~(3,6)
(5, 1)~(5,2)~(5,4)~(5,5)~(5,6)
(6, 1)~(6,3)~(6,4)~(6,5)~(6,6)
A
B
A \cap B
P(A\cap B) = \frac{1}{36}
P(A|B) = \frac{P(A\cap B)}{P(B)} = \frac{\frac{1}{36}}{\frac{6}{36}} = \frac{1}{6}

### is called the conditional probability of the event A given the event B

P(A|B)
P(A|B) = \frac{P(A\cap B)}{P(B)}

### B: event that at least one digit is even

P(A) = \frac{20}{90} = \frac{2}{9}

### B: event that at least one digit is even

P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{20}{90}}{\frac{65}{90}} = \frac{4}{13}

A
B

### B: event that the student has opted for ML

P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{0.2}{0.6} = \frac{1}{3}
A
B
\Omega

### Axioms of Probability

P(A|B) = \frac{P(A\cap B)}{P(B)} \geq 0

### : ratio of two probabilities

P(\Omega|A)= \frac{P(\Omega \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1
P(A_1 \cup A_2|B) = \frac{P((A_1 \cup A_2) \cap B)}{P(B)}
= \frac{P(A_1 \cap B)}{P(B)} + \frac{P(A_2 \cap B)}{P(B)}
= P(A_1|B) + P(A_2|B)
B
A_1
\Omega
A_2
= \frac{P((A_1 \cap B) \cup (A_2 \cap B)}{P(B)}

### The chain rule of probability

P(A|B) = \frac{P(A\cap B)}{P(B)}
P(B|A) = \frac{P(B\cap A)}{P(A)}
\therefore P(A\cap B) = P(A|B)\cdot P(B)
\therefore P(B\cap A) = P(B|A)\cdot P(A)
\therefore P(A\cap B) = P(A|B)\cdot P(B) = P(B|A)\cdot P(A)

+
B
\Omega
A

### B: event that the test result is positive

A\cap B
A\cap B^\mathsf{c}
A^\mathsf{c}\cap B
A^\mathsf{c}\cap B^\mathsf{c}

+
B
\Omega
A

### Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

+
B
\Omega
A

### Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05
A
A \cap B
A \cap B^\mathsf{c}
A^\mathsf{c} \cap B
A^\mathsf{c} \cap B^\mathsf{c}
A^\mathsf{c}
B|A
B^\mathsf{c}|A
B|A^\mathsf{c}
B^\mathsf{c}|A^\mathsf{c}

### for n events

P(A \cap B \cap C) = P((A \cap B) \cap C)
Let~(A \cap B)=X
\therefore P(A \cap B \cap C) = P(X \cap C)
\therefore P(A \cap B \cap C) = P(X)\cdot P(C|X)
\therefore P(A \cap B \cap C) = P(A\cap B)\cdot P(C|A \cap B)
\therefore P(A \cap B \cap C) = P(A)\cdot P(B|A)\cdot P(C|A \cap B)

### for n events

P(A \cap B \cap C \cap D)
= P(A)\cdot P(B|A)\cdot P(C|A \cap B)\cdot P(D|A \cap B \cap C)
P(A_1 \cap A_2 \cap \dots \cap A_n)
= P(A_1)\prod_{i=2}^{n} P(A_i|A_1\dots A_{i-1})

### Using counting principles:

p = \frac{4 \choose 3}{52 \choose 3} = \frac{\frac{4!}{1!~3!}}{\frac{52!}{49!~3!}} = \frac{4*3*2}{52*51*50}

A_i

### :the event that the i-th card is an ace

P(A_1\cap A_2\cap A_3)
= P(A_1)\cdot P(A_2|A_1)\cdot P(A_3|A_2 \cap A_1)
P(A_1)
= \frac{4}{52}
P(A_2|A_1)
= \frac{3}{51}
P(A_3|A_1\cap A_2)
= \frac{2}{50}
= \frac{4*3*2}{52*51*50}

### Total Probability Theorem

A_1
A_5
A_4
A_3
A_2
A_6
A_7
B
A_1, A_2, \cdots A_n
\Omega

### partition

\Omega
A_1 \cup A_2, \cup \dots \cup A_n = \Omega
A_i \cap A_j = \phi~\forall i \neq j
B = (B \cap A_1) \cup (B \cap A_2) \cup \dots \cup (B \cap A_n)
P(B) = P(B \cap A_1) + P(B \cap A_2) + \dots + P(B \cap A_n)
P(B) = \sum_{i=1}^{n} P (A_i) \cdot P(B | A_i)
P(B)
= P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + \dots + P(A_n)\cdot P(B|A_n)

+
B
\Omega
A

### Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

### B: test positive

P(B) = ?
P(B) = P(A)P(B|A) + P(A^\mathsf{c})(B(|A^\mathsf{c})
\therefore P(B) = 0.1*0.99 + 0.9*0.05 =0.144

### What is the probability that he will come out alive?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(B^\mathsf{c}) = ?

### i-th path taken

A_i:
= P(A_1)P(B^\mathsf{c}|A_1) + P(A_2)P(B^\mathsf{c}|A_2) + P(A_3)P(B^\mathsf{c}|A_3)
= \frac{1}{3}\cdot0.7 + \frac{1}{3}\cdot0.4 + \frac{1}{3}\cdot0.25 = 0.45

P(A_1|B) = ?

### i-th path taken

A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1 \cap B)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\ P(A_1 \cap B) = P(B)P(A_1|B)
P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75

### If he does not come out alive what is the probability that he took path A1?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_1|B) = ?

### i-th path taken

A_i:
P(A_1|B) = \frac{P(A_1 \cap B)}{P(B)}
P(A_1|B) = \frac{P(A_1)P(B|A_1)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_1 \cap B) = P(A_1)P(B|A_1) \\ P(A_1 \cap B) = P(B)P(A_1|B)
= 0.182

### If he does not come out alive what is the probability that he took path A3?

P(A_1)=P(A_2)=P(A_3) = \frac{1}{3}
P(B|A_1) = 0.3
P(B|A_2) = 0.6
P(B|A_3) = 0.75
P(A_3|B) = ?

### i-th path taken

A_i:
P(A_3|B) = \frac{P(A_3 \cap B)}{P(B)}
P(A_3|B) = \frac{P(A_3)P(B|A_3)}{P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)}
P(A_3 \cap B) = P(A_3)P(B|A_3) \\ P(A_3 \cap B) = P(B)P(A_3|B)
= 0.45

### Exploit Multiplication Rule

P(A_1)P(B|A_1) = P(B)P(A_1|B)

### Exploit Total Probability Theorem

P(B) = P(A_1)\cdot P(B|A_1) + P(A_2)\cdot P(B|A_2) + P(A_3)\cdot P(B|A_3)

### Exploit the known probabilities

P(A_1|B) = \frac{P(A_1)\cdot P(B|A_1) }{\sum_{i=1}^nP(A_i)P(B|A_i)}

### B  :  Ship 2 receives a signal 1

P(A) = 0.01
P(B|A) = 0.95
P(B|A^\mathsf{c}) = 0.05
P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.18
A
A^\mathsf{c}
10000
9900
100
495
95

A
A^\mathsf{c}
10000
9000
1000
450
990
+
B
\Omega
A

### Facts:

P(A) = 0.1
P(B^\mathsf{c}|A) = 0.01
\implies P(B|A) = 0.99
\implies P(B^\mathsf{c}|A^\mathsf{c}) = 0.95
P(B|A^\mathsf{c}) = 0.05

### B: test positive

P(A|B) = ?
P(A|B) = \frac{P(A)P(B|A)}{P(A)P(B|A) + P(A^\mathsf{c})P(B|A^\mathsf{c})} = 0.6875

### Bayes' Theorem: 3 forms

P(A_i|B) = \frac{P(A_i)\cdot P(B|A_i) }{\sum_{j=1}^{n}P(A_j)P(B|A_j)}
P(A|B) = \frac{P(A \cap B)}{P(B)}
P(A|B) = \frac{P(A)\cdot P(B|A) }{P(B)}

### Facts:

P(A) = \frac{50}{50+70} = \frac{5}{12}
P(B|A) = \frac{35}{50} = \frac{7}{10}

### B: student is good at Maths

P(A^\mathsf{c}) = \frac{7}{12}
P(B|A^\mathsf{c}) = \frac{49}{70} = \frac{7}{10}
P(A|B) = ?
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
P(B) = P(B|A)P(A) + P(B|A')P(A') = \frac{7}{10}
= \frac{\frac{7}{10}\frac{5}{12}}{\frac{7}{10}} = \frac{5}{12}

### Tow events A and B are independent if

P(A|B) = P(B)
P(A \cap B) = P(B)\cdot P(A|B)

### Tow events A and B are independent if

P(A\cap B) = P(A)\cdot P(B)

### B: exactly 2 tosses result in heads

P(A) = \frac{4}{8}

### Are A and B independent ?

H H H *
H H T * * *
H T H * * *
H T T *
T H H *
T H T
T T H
T T T
A
B
A \cap B
P(B) = \frac{3}{8}
P(A \cap B) = \frac{2}{8}
P(A \cap B) \neq P(A)P(B)

### B: second dice shows an even number

A = \{ (1,6), (2,5), (3,4), (4,3), (5,2), (6,1) \}
B = \{(1,2), (1,4), (1,6), (2,2), (2,4), (2,6),(3,2), (3,4), (3,6), \newline(4,2), (4,4), (4,6), (5,2), (5,4), (5,6), (6,2), (6,4), (6,6)\}
A \cap B = \{(1,6), (3,4), (5,2) \}
P(A) = \frac{6}{36} = \frac{1}{6}
P(A \cap B) = \frac{3}{36}= \frac{1}{12}
P(B) = \frac{18}{36} = \frac{1}{2}
\therefore P(A \cap B) = P(A)\cdot P(B)

### A quiz has two multiple choice Qs. The first Q has 4 choices of which 1 is correct and the second Q has 3 choices of which 1 is correct. If a student randomly guesses the answers what is the probability that he will answer both Qs correctly?

P(A) = \frac{1}{4}
P(B) = \frac{1}{3}
P(A \cap B) = P(A)\cdot P(B) = \frac{1}{12}

### We say that events                                      are pairwise independent if

A_1, A_2, A_3, \dots, A_n
P(A_i \cap A_j) = P(A_i)\cdot P(A_j)~\forall~i\neq j

### We say that events                                      are mutually independent or independent if for all subsets

P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)
A_1, A_2, A_3, \dots, A_n
I \subset \{1,2,3,\dots,n\}
\{1,2,3\}
n = 3
\{1,2\}, \{1, 3\}, \{2,3\}, \{1, 2, 3\}
P(A_1 \cap A_2 ) = P(A_1)\cdot P(A_2)
P(A_1 \cap A_3 ) = P(A_1)\cdot P(A_3)
P(A_2 \cap A_3 ) = P(A_2)\cdot P(A_3)
P(A_1 \cap A_2 \cap A_3 )
= P(A_1)\cdot P(A_2)\cdot P(A_3)

### Causes may be initially independent but become dependent once you have some knowledge about the effect

A \perp \!\!\! \perp B
A \not\!\perp\!\!\!\perp B~|~C

### Tow events A and B are independent if

P(A\cap B) = P(A)\cdot P(B)

### Tow events A and B are conditionally independent given an event C if

P(A\cap B|C) = P(A|C)\cdot P(B|C)

### Conditional Probability

Compute~P(B|A)~using~P(A\cap B)~and~P(A)

### Multiplication Rule

Compute~P(A\cap B)~using~P(A)~and~P(B|A)

### Total Probability Theorem

Compute~P(B)~using~P(A_1), P(A_2), \dots, P(A_n)
and~P(B|A_1), P(B|A_2), \dots, P(B|A_n)

### Bayes' Theorem

Compute~P(A|B)~using
P(\cap_{i \in I} A_i) = \prod_{i=1}^{n}P(A_i)