COMP3010: Algorithm Theory and Design

Daniel Sutantyo, Department of Computing, Macquarie University

9.0 - Basic Probability

Topics

  • Basic probability
  • Probabilistic analysis
    • using probability theory to analyse the running time of an algorithm
  • Randomised algorithms
    • add elements of randomness to an algorithm

9.0 - Basic Probability

  • In probability theory, we are interested in the probability that an event \(E\) happens, and we denote this using the notation \(P(E)\)
    • the set of all events is called the sample space \(S\)
    • \(P(E)\) assigns a numerical value that says how likely it is for one of the events in the sample space to occur
    • for example, if we are flipping a coin, then there are two possible events:
      • the coin comes up head, with probability \(P(\text{head})\)
      • the coin comes up tail, with probability \(P(\text{tail})\)
      • \(S = \{\text{head},\text{tail}\}\)
      • \(P(\text{head}) = P(\text{tail}) = 1/2\) if the coin is balanced

Events and Sample Space

9.0 - Basic Probability

  • Another example:
    • if we are flipping two coins, then \(S = \{HH,HT,TH,TT\}\) where
      • \(HH\) denotes the event that both coins come up heads
      • \(HT\) denotes the event that the first coin comes up head and the second coin comes up tail
      • \(TH\) denotes the event that the first coin comes up tail and the second coin comes up head
      • \(TT\) denotes the event that both coins come up tails
    • the probability of each event is \(1/4\)
    • if we don't differentiate between the first coin and the second coin, we can say \(S = \{HH,HT,TT\}\) with \(P(HT) = 1/2\)

9.0 - Basic Probability

Events and Sample Space

  • Make sure you define the event \(E\) clearly
    • \(HT\) - the event that one coin comes up head and the other one comes up tail
      • \(P(HT) = 1/2\)
    • \(HT\) - the event that the first coin comes up head and the second coin comes up tail
      • \(P(HT) = 1/4\)
  • You can also use derive these probabilities from \(P(H)\) and \(P(T)\) if the events are independent and mutually exclusive
    • \(P(HT) = P(H) \times P(T) + P(T) \times P(H)\)
    • \(P(HT) = P(H) \times P(T)\)

9.0 - Basic Probability

Events and Sample Space

Probability Distribution

  • More formally, we are going to call \(P\) is what we call a probability distribution, and it is a function that maps the events in \(S\) to real numbers such that
    • \(0 \le P(E) \le 1\) for any event \(E \in S\)  (with \(P(S) = 1\))
  • For this unit, we will limit our discussion to discrete probability distributions, that is, there is a finite number of events 

9.0 - Basic Probability

Uniform Distribution

  • Furthermore, we are going to assume a uniform probability distribution on \(S\), that is, each elementary event in \(S\) has the same probability of occurring
    • elementary events in \(S\) are mutually exclusive, that is if \(E_1\) and \(E_2\) are two elementary events, then only one of them can occur
    • example:
      • \(H\) and \(T\) are elementary events
      • \(HT\) and \(TH\) are elementary events (both definitions)
      • if we are flipping 2 coins or more then these events are not elementary:
        • \(E_1 = \) there are 0 or more tails
        • \(E_2 = \) there are 1 or more tails
        • \(E_3 = \) there are 2 or more tails

9.0 - Basic Probability

  • If there are \(n\) elementary events and they are equally likely to occur, then the probability of each elementary event is \(1/n\)
  • In this unit, we are also going to assume that events are independent, that is, the probability of an elementary event happening is not dependent on another elementary event happening
    • \(P(E_1 \text{\ and\ } E_2) = P(E_1) \times P(E_2)\)
       
  • You do not need to specify that an event is elementary or not, just make sure to be careful when there is an overlap between two or more events

9.0 - Basic Probability

Uniform Distribution

Random Variable

  • Sometimes we need a more compact way to write some probabilities
    • for example, suppose that we are rolling a dice
      • we can use \(P(1)\), \(P(2)\), \(P(3)\), etc to denote the events
      • or we can use a random variable \(X\) 
        • let \(X\) denotes the number that shows up on the dice
        • \(X\) is a variable, but its value is random, so we call it a random variable (and it is often a numerical value)
        • usage:
          • \(P(X = 1) = 1/6\)
          • \(P(X = 2) = 1/6\)
          • etc

9.0 - Basic Probability

  • Why do we bother?
    • we can write probability of some events more succinctly:
      • \(P(X \le 0) = 0\)
      • \(P(X \le 3) = 1/2\)
      • \(P(X \le 5) = 5/6\)
      • \(P(X \le 6) = 1\)
    • random variables is what we use to write a probability density functions, but this is not something we're going to discuss in this unit

9.0 - Basic Probability

Random Variable

  • Another example:
    • Suppose that we are rolling two dice
    • Let \(X\) be the sum of the values shown on the dice
      • \(P(X = 3) = 2/36 = 1/18\)
      • \(P(X = 6) = 5/36\)
    • Let \(X\) be the maximum of the two values shown on the dice
      • \(P(X = 2) = 3/36 = 1/12\)
      • \(P(X = 6) = 11/36\)

9.0 - Basic Probability

Random Variable

  • You do not need to write \(X = x\) once you define \(X\), you can just write \(P(x)\)
    • let \(X\) be the value shown on a dice roll
      • you can write \(P(X=1) = 1/6\)
      • or you can also just write \(P(1) = 1/6\)
    • the difference is that, if we don't use a random variable, then we have to define 6 events
      • \(P(1)\) is the probability that the value is 1
      • \(P(2)\) is the probability that the value is 2
    • if we use a random variable, then we (kind of) only define one event
      • \(P(X = x)\) is the probability that the value is \(x\), \(x \in \{1,2,3,4,5,6\}\)
      • \(P(1)\) is basically \(P(x)\) with \(x = 1\)

9.0 - Basic Probability

Random Variable

Expected Values

  • The expected value of a random variable \(X\) is the average of the value it can take
    • if we roll a dice, then the average of the possible values is 3.5 (from 1+2+3+4+5+6 divided by 6) 
  • We use \(E[X]\) to denote the expected value of \(X\) (notice the square brackets), and we compute it using the formula
    • \[E[X] = \sum_{x} x \cdot P(x)\]
      where the summation is over all the possible values of that \(X\) can take and their probabilities

9.0 - Basic Probability

  • Example:
    • if we roll a dice, then the expected value is
    • \[\left(1 \times \frac{1}{6}\right) + \left(2 \times \frac{1}{6}\right) + \left(3 \times\frac{1}{6}\right) + \left(4 \times \frac{1}{6}\right) + \left(5 \times \frac{1}{6}\right) + \left(6 \times \frac{1}{6}\right) = 3.5\] 
  • Example:
    • if we flip a coin then the expected value is ???

9.0 - Basic Probability

Expected Values

Expected Values

9.0 - Basic Probability

  • Remember that a random variable \(X\) normally takes a numerical value, so if you want to find the expected value of \(X\), then you normally need to assign a numerical value to its outcomes
  • For example, suppose that you are playing a game with your friend:
    • if the result is head, you double your money
    • if the result is tail, you lose half your money
    • if you bet $10, then the expected value of one game is
      • \(0.5 \times \$10 + 0.5 \times (-\$5) = \$2.5\)  
      • you should play this game as much as you can

More on Probabilities

9.0 - Basic Probability

  • Probability can be counterintuitive, or at least, it is easy to have the wrong intuition
  • Example:
    • if a family has two children, and at least one is a girl, what is the probability that both children are girls?
    • if a family has two children, and the eldest is a girl, what is the probability that both children are girls

9.0 - Basic Probability

  • If a family has two children, and at least one is a girl, what is the probability that both children are girls?
    • some people think the answer is 1/2, because you either have one girl or two girls
    • but the sample space actually has three events:
      • the first child is a girl, the second child is a boy
      • the first child is a boy, the second child is a girl
      • both children are girls
    • so the probability that both children are girls is 1/3

More on Probabilities

9.0 - Basic Probability

  • If a family has two children, and the eldest is a girl, what is the probability that both children are girls
    • ​there are only two possibilities: both children are girls, or the the second child is a boy
    • the probability is 1/2

More on Probabilities

9.0 - Basic Probability

More on Probabilities

  • The Monty Hall Problem:
    • you are a participant in a TV game show and you are given a choice of 3 doors to open.
      • behind one of the doors is car
      • the other doors contain nothing behind them (or maybe there is a goat)
    • after you made a choice, the host opens another door (that you didn't choose) and reveals that there is a goat behind it
    • should you choose another door?

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

  • Your intuition may tell you that there is no point in changing your choice, since there is 50-50 chance anyway
    • actually this is an incorrect, it is always better to change the your choice

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

9.0 - Basic Probability

More on Probabilities

  • The door that you chose at the start has \(1/n\) chance of having the car behind it, where \(n\) is the number of doors
  • The doors that you did not choose at the start has \((1 - 1/n)\) chance of having the car behind it
    • if there are 100 doors, the door that you chose have \(1/100\) chance of having the car
    • the 99 doors that you didn't choose have \(99/100\) chance of having the car
      • if you remove 1 door, the remaining 98 doors still have \(99/100\) chance of having the car
      • if you remove 2 doors, the remaining 97 doors still have \(99/100\) chance of having the car
      • if you remove 98 doors, the last door has \(99/100\) chance of having the car

9.0 - Basic Probability

More on Probabilities

  • With only three doors, the door that you chose has \(1/3\) chance of having the car behind it, and the other two doors have \(2/3\) chance
    • if one door is opened, then the other door now has the \(2/3\) chance
    • does it mean it contains the car, no, it's just more likely
    • with 100 doors, can the door that you chose at the start contain the car? 
      • yes, of course, but it is 1 in 100 chance

9.0 - Basic Probability

More on Probabilities

  • Finally, most of us are pretty bad at randomising
    • if I ask you to give me 5 random numbers from 1 to 10, there is a very high probability that you would give me 5 distinct numbers

9.0 - Basic Probability

More on Probabilities

  • In this unit we are only going to do some basic probability
    • most common setting: there is an array with \(n\) integers, pick a random element, probability is \(1/n\)
  • Please understand that we are doing this very informally, and if you want to do more on probabilities, you should pick up a good reference
    • CLRS Appendix C has a good treatment of probabilities, still at beginners level, which really is enough for most computer scientists
    • Good website for beginners: https://seeing-theory.brown.edu/basic-probability