Bayesian Methods

A  brief  introduction to

T. Paternina

Machine Learning Journal Club

05/04/2017

Overview

  1. Quick reminder on probability
  2. Bayes' rule and HIV diagnosis
  3. Inference & comparing models
  4. Why Bayes'?

Quick reminder

P(A,B) = P(A)*P(B)
P(A,B)=P(A)∗P(B)P(A,B) = P(A)*P(B)
P(X=6) = {1 \over6}
P(X=6)=16P(X=6) = {1 \over6}
P(X=6,Y=6) = {1 \over6}* {1 \over6}
P(X=6,Y=6)=16∗16P(X=6,Y=6) = {1 \over6}* {1 \over6}

***

Quick reminder

P(A \cup B) = P(A)+P(B)
P(A∪B)=P(A)+P(B)P(A \cup B) = P(A)+P(B)
P(X=6 \cup X=5) = {1 \over6} + {1 \over6}
P(X=6∪X=5)=16+16P(X=6 \cup X=5) = {1 \over6} + {1 \over6}

***

or

Quick reminder

P(A)+P(B)=1
P(A)+P(B)=1P(A)+P(B)=1
P(X\ne6) = 1- {1 \over6}
P(X≠6)=1−16P(X\ne6) = 1- {1 \over6}

***

Bayes' rule

Bayes' rule

Total

43 9 52
44 4 48
Total 87 13 100
P(A|B) = {P(A,B) \over P(B)}
P(A∣B)=P(A,B)P(B)P(A|B) = {P(A,B) \over P(B)}

The probability of an event A conditioned by an event B is equal to the joint probability of A & B over the probability of B.

P(\space\space\space|\space\space\space) = {P(\space\space\space,\space\space\space) \over P(\space\space\space)}
P(   ∣   )=P(   ,   )P(   )P(\space\space\space|\space\space\space) = {P(\space\space\space,\space\space\space) \over P(\space\space\space)}
={44/100\over 48/100} = {44\over48}
=44/10048/100=4448={44/100\over 48/100} = {44\over48}
\approx 91.6\%
≈91.6%\approx 91.6\%

example from wikipedia (https://en.wikipedia.org/wiki/Contingency_table)

HIV diagnosis

HIV diagnosis

(U.S. military style)

(x2)

(x2)

(x2)

HIV+

HIV diagnosis

(U.S. military style)

P(HIV+|ELISA+)?
P(HIV+∣ELISA+)?P(HIV+|ELISA+)?

ELISA sensitivity (TP) = 93%

ELISA specificity (TN) = 99%

HIV+ prevalence = 1.48/1000

P(ELISA+|HIV+)
P(ELISA+∣HIV+)P(ELISA+|HIV+)
P(ELISA-|HIV-)
P(ELISA−∣HIV−)P(ELISA-|HIV-)
P(HIV+)
P(HIV+)P(HIV+)

Posterior probability

Prior probability

P(HIV+)
P(HIV+)P(HIV+)
P(HIV-)
P(HIV−)P(HIV-)

Prior probability

P(+|HIV+)
P(+∣HIV+)P(+|HIV+)
P(-|HIV-)
P(−∣HIV−)P(-|HIV-)
P(-|HIV+)
P(−∣HIV+)P(-|HIV+)
P(+|HIV-)
P(+∣HIV−)P(+|HIV-)
P(A|B) = {P(A,B) \over P(B)}
P(A∣B)=P(A,B)P(B)P(A|B) = {P(A,B) \over P(B)}
P(HIV+|+) = {P(HIV+,+) \over P(+)}= {P(+|HIV+)P(HIV+)\over P(+,HIV+) +P(+,HIV-)}
P(HIV+∣+)=P(HIV+,+)P(+)=P(+∣HIV+)P(HIV+)P(+,HIV+)+P(+,HIV−)P(HIV+|+) = {P(HIV+,+) \over P(+)}= {P(+|HIV+)P(HIV+)\over P(+,HIV+) +P(+,HIV-)}

0.00148

0.99852

0.93

0.07

0.99

0.01

= {0.00148 * 0.93 \over (0.00148*0.93)+ (0.99852*0.01)} \approx 0.12
=0.00148∗0.93(0.00148∗0.93)+(0.99852∗0.01)≈0.12 = {0.00148 * 0.93 \over (0.00148*0.93)+ (0.99852*0.01)} \approx 0.12

Posterior probability

Updating probabilities

0.00148
0.001480.00148
0.12
0.120.12

1st ELISA

Prior probability

Updating probabilities

0.00148
0.001480.00148
0.12
0.120.12

1st ELISA

Prior probability

0.93
0.930.93

2n ELISA

Prior probability

Updating probabilities

0.00148
0.001480.00148
0.12
0.120.12

1st ELISA

0.93
0.930.93

2n ELISA

0.999
0.9990.999

3rd ELISA

Prior probability

Prior probability

Prior probability

Bayesian Inference & comparing models

Effectiveness of contraception

40

20 standard

20 pill

9 months

16

4

p:

probability of a pregnancy coming from the treatment group.

p 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
prior 0.06 0.06 0.06 0.06 0.52 0.06 0.06 0.06 0.06

Testing several models

k:

number of pregnancies in the treatment group.

What's the value of p?

Computing likelihoods

Likelihood:

probability of the observed data given a particular model.

P(data|model)
P(data∣model)P(data|model)
P(k=4|n=20,p)
P(k=4∣n=20,p)P(k=4|n=20,p)
p 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
prior 0.06 0.06 0.06 0.06 0.52 0.06 0.06 0.06 0.06
L 0.0898 0.2182 0.1304 0.035 0.0046 0.0003 0 0 0
P(p|k=4,n=20)={L*P(p) \over P(k=4,n=20)}
P(p∣k=4,n=20)=L∗P(p)P(k=4,n=20)P(p|k=4,n=20)={L*P(p) \over P(k=4,n=20)}
={{L * P(p) \over \sum_{i} L_iP(p_i)}}
=L∗P(p)∑iLiP(pi)={{L * P(p) \over \sum_{i} L_iP(p_i)}}
p 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
prior 0.06 0.06 0.06 0.06 0.52 0.06 0.06 0.06 0.06
L 0.0898 0.2182 0.1304 0.035 0.0046 0.0003 0 0 0
post. 0.1748 0.4248 0.2539 0.0681 0.0780 0.0005 0 0 0

Once you go Bayesian...

  • Updating degree of belief as we gain information.

 

  • Perform model parameter estimation from data.

 

  • Quantify the level of uncertainty in a model.

References and resources

 

 

Made with Slides.com