COMP3010: Algorithm Theory and Design
Daniel Sutantyo, Department of Computing, Macquarie University
9.0 - Basic Probability
Topics
- Basic probability
- Probabilistic analysis
- using probability theory to analyse the running time of an algorithm
- Randomised algorithms
- add elements of randomness to an algorithm
9.0 - Basic Probability
- In probability theory, we are interested in the probability that an event \(E\) happens, and we denote this using the notation \(P(E)\)
- the set of all events is called the sample space \(S\)
- \(P(E)\) assigns a numerical value that says how likely it is for one of the events in the sample space to occur
- for example, if we are flipping a coin, then there are two possible events:
- the coin comes up head, with probability \(P(\text{head})\)
- the coin comes up tail, with probability \(P(\text{tail})\)
- \(S = \{\text{head},\text{tail}\}\)
- \(P(\text{head}) = P(\text{tail}) = 1/2\) if the coin is balanced
Events and Sample Space
9.0 - Basic Probability
- Another example:
- if we are flipping two coins, then \(S = \{HH,HT,TH,TT\}\) where
- \(HH\) denotes the event that both coins come up heads
- \(HT\) denotes the event that the first coin comes up head and the second coin comes up tail
- \(TH\) denotes the event that the first coin comes up tail and the second coin comes up head
- \(TT\) denotes the event that both coins come up tails
- the probability of each event is \(1/4\)
- if we don't differentiate between the first coin and the second coin, we can say \(S = \{HH,HT,TT\}\) with \(P(HT) = 1/2\)
- if we are flipping two coins, then \(S = \{HH,HT,TH,TT\}\) where
9.0 - Basic Probability
Events and Sample Space
- Make sure you define the event \(E\) clearly
- \(HT\) - the event that one coin comes up head and the other one comes up tail
- \(P(HT) = 1/2\)
- \(HT\) - the event that the first coin comes up head and the second coin comes up tail
- \(P(HT) = 1/4\)
- \(HT\) - the event that one coin comes up head and the other one comes up tail
- You can also use derive these probabilities from \(P(H)\) and \(P(T)\) if the events are independent and mutually exclusive
- \(P(HT) = P(H) \times P(T) + P(T) \times P(H)\)
- \(P(HT) = P(H) \times P(T)\)
9.0 - Basic Probability
Events and Sample Space
Probability Distribution
- More formally, we are going to call \(P\) is what we call a probability distribution, and it is a function that maps the events in \(S\) to real numbers such that
- \(0 \le P(E) \le 1\) for any event \(E \in S\) (with \(P(S) = 1\))
- For this unit, we will limit our discussion to discrete probability distributions, that is, there is a finite number of events
9.0 - Basic Probability
Uniform Distribution
- Furthermore, we are going to assume a uniform probability distribution on \(S\), that is, each elementary event in \(S\) has the same probability of occurring
- elementary events in \(S\) are mutually exclusive, that is if \(E_1\) and \(E_2\) are two elementary events, then only one of them can occur
- example:
- \(H\) and \(T\) are elementary events
- \(HT\) and \(TH\) are elementary events (both definitions)
- if we are flipping 2 coins or more then these events are not elementary:
- \(E_1 = \) there are 0 or more tails
- \(E_2 = \) there are 1 or more tails
- \(E_3 = \) there are 2 or more tails
9.0 - Basic Probability
- If there are \(n\) elementary events and they are equally likely to occur, then the probability of each elementary event is \(1/n\)
- In this unit, we are also going to assume that events are independent, that is, the probability of an elementary event happening is not dependent on another elementary event happening
- \(P(E_1 \text{\ and\ } E_2) = P(E_1) \times P(E_2)\)
- \(P(E_1 \text{\ and\ } E_2) = P(E_1) \times P(E_2)\)
- You do not need to specify that an event is elementary or not, just make sure to be careful when there is an overlap between two or more events
9.0 - Basic Probability
Uniform Distribution
Random Variable
- Sometimes we need a more compact way to write some probabilities
- for example, suppose that we are rolling a dice
- we can use \(P(1)\), \(P(2)\), \(P(3)\), etc to denote the events
- or we can use a random variable \(X\)
- let \(X\) denotes the number that shows up on the dice
- \(X\) is a variable, but its value is random, so we call it a random variable (and it is often a numerical value)
- usage:
- \(P(X = 1) = 1/6\)
- \(P(X = 2) = 1/6\)
- etc
- for example, suppose that we are rolling a dice
9.0 - Basic Probability
- Why do we bother?
- we can write probability of some events more succinctly:
- \(P(X \le 0) = 0\)
- \(P(X \le 3) = 1/2\)
- \(P(X \le 5) = 5/6\)
- \(P(X \le 6) = 1\)
- random variables is what we use to write a probability density functions, but this is not something we're going to discuss in this unit
- we can write probability of some events more succinctly:
9.0 - Basic Probability
Random Variable
- Another example:
- Suppose that we are rolling two dice
- Let \(X\) be the sum of the values shown on the dice
- \(P(X = 3) = 2/36 = 1/18\)
- \(P(X = 6) = 5/36\)
- Let \(X\) be the maximum of the two values shown on the dice
- \(P(X = 2) = 3/36 = 1/12\)
- \(P(X = 6) = 11/36\)
9.0 - Basic Probability
Random Variable
- You do not need to write \(X = x\) once you define \(X\), you can just write \(P(x)\)
- let \(X\) be the value shown on a dice roll
- you can write \(P(X=1) = 1/6\)
- or you can also just write \(P(1) = 1/6\)
- the difference is that, if we don't use a random variable, then we have to define 6 events
- \(P(1)\) is the probability that the value is 1
- \(P(2)\) is the probability that the value is 2
- if we use a random variable, then we (kind of) only define one event
- \(P(X = x)\) is the probability that the value is \(x\), \(x \in \{1,2,3,4,5,6\}\)
- \(P(1)\) is basically \(P(x)\) with \(x = 1\)
- let \(X\) be the value shown on a dice roll
9.0 - Basic Probability
Random Variable
Expected Values
- The expected value of a random variable \(X\) is the average of the value it can take
- if we roll a dice, then the average of the possible values is 3.5 (from 1+2+3+4+5+6 divided by 6)
- We use \(E[X]\) to denote the expected value of \(X\) (notice the square brackets), and we compute it using the formula
- \[E[X] = \sum_{x} x \cdot P(x)\]
where the summation is over all the possible values of that \(X\) can take and their probabilities
- \[E[X] = \sum_{x} x \cdot P(x)\]
9.0 - Basic Probability
- Example:
- if we roll a dice, then the expected value is
- \[\left(1 \times \frac{1}{6}\right) + \left(2 \times \frac{1}{6}\right) + \left(3 \times\frac{1}{6}\right) + \left(4 \times \frac{1}{6}\right) + \left(5 \times \frac{1}{6}\right) + \left(6 \times \frac{1}{6}\right) = 3.5\]
- Example:
- if we flip a coin then the expected value is ???
9.0 - Basic Probability
Expected Values
Expected Values
9.0 - Basic Probability
- Remember that a random variable \(X\) normally takes a numerical value, so if you want to find the expected value of \(X\), then you normally need to assign a numerical value to its outcomes
- For example, suppose that you are playing a game with your friend:
- if the result is head, you double your money
- if the result is tail, you lose half your money
- if you bet $10, then the expected value of one game is
- \(0.5 \times \$10 + 0.5 \times (-\$5) = \$2.5\)
- you should play this game as much as you can
More on Probabilities
9.0 - Basic Probability
- Probability can be counterintuitive, or at least, it is easy to have the wrong intuition
- Example:
- if a family has two children, and at least one is a girl, what is the probability that both children are girls?
- if a family has two children, and the eldest is a girl, what is the probability that both children are girls
9.0 - Basic Probability
- If a family has two children, and at least one is a girl, what is the probability that both children are girls?
- some people think the answer is 1/2, because you either have one girl or two girls
- but the sample space actually has three events:
- the first child is a girl, the second child is a boy
- the first child is a boy, the second child is a girl
- both children are girls
- so the probability that both children are girls is 1/3
More on Probabilities
9.0 - Basic Probability
- If a family has two children, and the eldest is a girl, what is the probability that both children are girls
- there are only two possibilities: both children are girls, or the the second child is a boy
- the probability is 1/2
More on Probabilities
9.0 - Basic Probability
More on Probabilities
- The Monty Hall Problem:
- you are a participant in a TV game show and you are given a choice of 3 doors to open.
- behind one of the doors is car
- the other doors contain nothing behind them (or maybe there is a goat)
- after you made a choice, the host opens another door (that you didn't choose) and reveals that there is a goat behind it
- should you choose another door?
- you are a participant in a TV game show and you are given a choice of 3 doors to open.
9.0 - Basic Probability
More on Probabilities
9.0 - Basic Probability
More on Probabilities
- Your intuition may tell you that there is no point in changing your choice, since there is 50-50 chance anyway
- actually this is an incorrect, it is always better to change the your choice
9.0 - Basic Probability
More on Probabilities
9.0 - Basic Probability
More on Probabilities
9.0 - Basic Probability
More on Probabilities
- The door that you chose at the start has \(1/n\) chance of having the car behind it, where \(n\) is the number of doors
- The doors that you did not choose at the start has \((1 - 1/n)\) chance of having the car behind it
- if there are 100 doors, the door that you chose have \(1/100\) chance of having the car
- the 99 doors that you didn't choose have \(99/100\) chance of having the car
- if you remove 1 door, the remaining 98 doors still have \(99/100\) chance of having the car
- if you remove 2 doors, the remaining 97 doors still have \(99/100\) chance of having the car
- if you remove 98 doors, the last door has \(99/100\) chance of having the car
9.0 - Basic Probability
More on Probabilities
- With only three doors, the door that you chose has \(1/3\) chance of having the car behind it, and the other two doors have \(2/3\) chance
- if one door is opened, then the other door now has the \(2/3\) chance
- does it mean it contains the car, no, it's just more likely
- with 100 doors, can the door that you chose at the start contain the car?
- yes, of course, but it is 1 in 100 chance
9.0 - Basic Probability
More on Probabilities
- Finally, most of us are pretty bad at randomising
- if I ask you to give me 5 random numbers from 1 to 10, there is a very high probability that you would give me 5 distinct numbers
9.0 - Basic Probability
More on Probabilities
- In this unit we are only going to do some basic probability
- most common setting: there is an array with \(n\) integers, pick a random element, probability is \(1/n\)
- Please understand that we are doing this very informally, and if you want to do more on probabilities, you should pick up a good reference
- CLRS Appendix C has a good treatment of probabilities, still at beginners level, which really is enough for most computer scientists
- Good website for beginners: https://seeing-theory.brown.edu/basic-probability
COMP3010 - 9.0 - Basic Probability
By Daniel Sutantyo
