COMP3010 - 9.0 - Basic Probability

COMP3010: Algorithm Theory and Design

Daniel Sutantyo, Department of Computing, Macquarie University

9.0 - Basic Probability

Topics

Basic probability
Probabilistic analysis
- using probability theory to analyse the running time of an algorithm
Randomised algorithms
- add elements of randomness to an algorithm

9.0 - Basic Probability

In probability theory, we are interested in the probability that an event $E$ happens, and we denote this using the notation $P(E)$
- the set of all events is called the sample space $S$
- $P(E)$ assigns a numerical value that says how likely it is for one of the events in the sample space to occur
- for example, if we are flipping a coin, then there are two possible events:
  - the coin comes up head, with probability $P(\text{head})$
  - the coin comes up tail, with probability $P(\text{tail})$
  - $S = \{\text{head},\text{tail}\}$
  - $P(\text{head}) = P(\text{tail}) = 1/2$ if the coin is balanced

Events and Sample Space

9.0 - Basic Probability

Another example:
- if we are flipping two coins, then $S = \{HH,HT,TH,TT\}$ where
  - $HH$ denotes the event that both coins come up heads
  - $HT$ denotes the event that the first coin comes up head and the second coin comes up tail
  - $TH$ denotes the event that the first coin comes up tail and the second coin comes up head
  - $TT$ denotes the event that both coins come up tails
- the probability of each event is $1/4$
- if we don't differentiate between the first coin and the second coin, we can say $S = \{HH,HT,TT\}$ with $P(HT) = 1/2$

9.0 - Basic Probability

Events and Sample Space

Make sure you define the event $E$ clearly
- $HT$ - the event that one coin comes up head and the other one comes up tail
  - $P(HT) = 1/2$
- $HT$ - the event that the first coin comes up head and the second coin comes up tail
  - $P(HT) = 1/4$
You can also use derive these probabilities from $P(H)$ and $P(T)$ if the events are independent and mutually exclusive
- $P(HT) = P(H) \times P(T) + P(T) \times P(H)$
- $P(HT) = P(H) \times P(T)$

9.0 - Basic Probability

Events and Sample Space

Probability Distribution

More formally, we are going to call $P$ is what we call a probability distribution, and it is a function that maps the events in $S$ to real numbers such that
- $0 \le P(E) \le 1$ for any event $E \in S$ (with $P(S) = 1$)
For this unit, we will limit our discussion to discrete probability distributions, that is, there is a finite number of events

9.0 - Basic Probability

Uniform Distribution

Furthermore, we are going to assume a uniform probability distribution on $S$, that is, each elementary event in $S$ has the same probability of occurring
- elementary events in $S$ are mutually exclusive, that is if $E_1$ and $E_2$ are two elementary events, then only one of them can occur
- example:
  - $H$ and $T$ are elementary events
  - $HT$ and $TH$ are elementary events (both definitions)
  - if we are flipping 2 coins or more then these events are not elementary:
    - $E_1 = $ there are 0 or more tails
    - $E_2 = $ there are 1 or more tails
    - $E_3 = $ there are 2 or more tails

9.0 - Basic Probability

If there are $n$ elementary events and they are equally likely to occur, then the probability of each elementary event is $1/n$
In this unit, we are also going to assume that events are independent, that is, the probability of an elementary event happening is not dependent on another elementary event happening
- $P(E_1 \text{\ and\ } E_2) = P(E_1) \times P(E_2)$
You do not need to specify that an event is elementary or not, just make sure to be careful when there is an overlap between two or more events

9.0 - Basic Probability

Uniform Distribution

Random Variable

Sometimes we need a more compact way to write some probabilities
- for example, suppose that we are rolling a dice
  - we can use $P(1)$, $P(2)$, $P(3)$, etc to denote the events
  - or we can use a random variable $X$
    - let $X$ denotes the number that shows up on the dice
    - $X$ is a variable, but its value is random, so we call it a random variable (and it is often a numerical value)
    - usage:
      - $P(X = 1) = 1/6$
      - $P(X = 2) = 1/6$
      - etc

9.0 - Basic Probability

Why do we bother?
- we can write probability of some events more succinctly:
  - $P(X \le 0) = 0$
  - $P(X \le 3) = 1/2$
  - $P(X \le 5) = 5/6$
  - $P(X \le 6) = 1$
- random variables is what we use to write a probability density functions, but this is not something we're going to discuss in this unit

9.0 - Basic Probability

Random Variable

Another example:
- Suppose that we are rolling two dice
- Let $X$ be the sum of the values shown on the dice
  - $P(X = 3) = 2/36 = 1/18$
  - $P(X = 6) = 5/36$
- Let $X$ be the maximum of the two values shown on the dice
  - $P(X = 2) = 3/36 = 1/12$
  - $P(X = 6) = 11/36$

9.0 - Basic Probability

Random Variable

You do not need to write $X = x$ once you define $X$, you can just write $P(x)$
- let $X$ be the value shown on a dice roll
  - you can write $P(X=1) = 1/6$
  - or you can also just write $P(1) = 1/6$
- the difference is that, if we don't use a random variable, then we have to define 6 events
  - $P(1)$ is the probability that the value is 1
  - $P(2)$ is the probability that the value is 2
- if we use a random variable, then we (kind of) only define one event
  - $P(X = x)$ is the probability that the value is $x$, $x \in \{1,2,3,4,5,6\}$
  - $P(1)$ is basically $P(x)$ with $x = 1$

9.0 - Basic Probability

Random Variable

Expected Values

The expected value of a random variable $X$ is the average of the value it can take
- if we roll a dice, then the average of the possible values is 3.5 (from 1+2+3+4+5+6 divided by 6)
We use $E[X]$ to denote the expected value of $X$ (notice the square brackets), and we compute it using the formula
- \[E[X] = \sum_{x} x \cdot P(x)\]
  where the summation is over all the possible values of that $X$ can take and their probabilities

9.0 - Basic Probability

Example:
- if we roll a dice, then the expected value is
- \[\left(1 \times \frac{1}{6}\right) + \left(2 \times \frac{1}{6}\right) + \left(3 \times\frac{1}{6}\right) + \left(4 \times \frac{1}{6}\right) + \left(5 \times \frac{1}{6}\right) + \left(6 \times \frac{1}{6}\right) = 3.5\]
Example:
- if we flip a coin then the expected value is ???

9.0 - Basic Probability

Expected Values

9.0 - Basic Probability

Remember that a random variable $X$ normally takes a numerical value, so if you want to find the expected value of $X$, then you normally need to assign a numerical value to its outcomes
For example, suppose that you are playing a game with your friend:
- if the result is head, you double your money
- if the result is tail, you lose half your money
- if you bet $10, then the expected value of one game is
  - $0.5 \times \$10 + 0.5 \times (-\$5) = \$2.5$
  - you should play this game as much as you can

More on Probabilities

9.0 - Basic Probability

Probability can be counterintuitive, or at least, it is easy to have the wrong intuition
Example:
- if a family has two children, and at least one is a girl, what is the probability that both children are girls?
- if a family has two children, and the eldest is a girl, what is the probability that both children are girls

https://mathwithbaddrawings.com/2013/10/14/the-riddle-of-the-odorless-incense/

9.0 - Basic Probability

If a family has two children, and at least one is a girl, what is the probability that both children are girls?
- some people think the answer is 1/2, because you either have one girl or two girls
- but the sample space actually has three events:
  - the first child is a girl, the second child is a boy
  - the first child is a boy, the second child is a girl
  - both children are girls
- so the probability that both children are girls is 1/3

More on Probabilities

9.0 - Basic Probability

More on Probabilities

https://en.wikipedia.org/wiki/Monty_Hall_problem

The Monty Hall Problem:
- you are a participant in a TV game show and you are given a choice of 3 doors to open.
  - behind one of the doors is car
  - the other doors contain nothing behind them (or maybe there is a goat)
- after you made a choice, the host opens another door (that you didn't choose) and reveals that there is a goat behind it
- should you choose another door?

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

The door that you chose at the start has $1/n$ chance of having the car behind it, where $n$ is the number of doors
The doors that you did not choose at the start has $(1 - 1/n)$ chance of having the car behind it
- if there are 100 doors, the door that you chose have $1/100$ chance of having the car
- the 99 doors that you didn't choose have $99/100$ chance of having the car
  - if you remove 1 door, the remaining 98 doors still have $99/100$ chance of having the car
  - if you remove 2 doors, the remaining 97 doors still have $99/100$ chance of having the car
  - if you remove 98 doors, the last door has $99/100$ chance of having the car

9.0 - Basic Probability

More on Probabilities

With only three doors, the door that you chose has $1/3$ chance of having the car behind it, and the other two doors have $2/3$ chance
- if one door is opened, then the other door now has the $2/3$ chance
- does it mean it contains the car, no, it's just more likely
- with 100 doors, can the door that you chose at the start contain the car?
  - yes, of course, but it is 1 in 100 chance

9.0 - Basic Probability

More on Probabilities

In this unit we are only going to do some basic probability
- most common setting: there is an array with $n$ integers, pick a random element, probability is $1/n$
Please understand that we are doing this very informally, and if you want to do more on probabilities, you should pick up a good reference
- CLRS Appendix C has a good treatment of probabilities, still at beginners level, which really is enough for most computer scientists
- Good website for beginners: https://seeing-theory.brown.edu/basic-probability