COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

6.0 - Divide and Conquer -  Recurrence Equation

The topics so far ...

6.0 - Divide and Conquer

  • Complexity
  • Correctness
  • Brute force
  • Dynamic Programming
  • Greedy Algorithm

 

Reference for this week: CLRS Chapter 4

Divide and Conquer

Optimal Substructure

The topics so far ...

6.0 - Divide and Conquer

Divide and Conquer

Optimal Substructure

  • Complexity
  • Correctness
  • Brute force
  • Dynamic Programming
  • Greedy Algorithm

 

Reference for this week: CLRS Chapter 4

6.0 - Divide and Conquer

Complexity of Divide and Conquer Algorithm

  • How did you work out the complexity of divide and conquer algorithm?
    • in the first week, we mostly talked about the complexity notations, i.e. the \(O\), \(\Theta\), and \(\Omega\) notations
    • it was assumed that you know the complexity of some simple algorithms like binary search, mergesort, and quicksort
    • how did you work it out?

6.0 - Divide and Conquer

Mergesort

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\)
  • total number of operations: \(n\log n\)

13

4

19

11

32

22

7

32

22

7

32

22

7

13

4

19

11

13

4

19

11

6.0 - Divide and Conquer

Binary Search

  • number of levels: \(\log n\)
  • number of operations on each level : \(1\)
  • total number of operations: \(\log n\)

13

4

19

11

32

7

22

13

19

32

22

13

19

13

6.0 - Divide and Conquer

Recursion Tree

6.0 - Divide and Conquer

Recursion Tree - Mergesort

\(T(n) = 2T(n/2) + n\)

the number of operations:

\(T(1) = 1\)

6.0 - Divide and Conquer

Recursion Tree - Binary Search

\(T(n) = T(n/2) + 1\)

the number of operations:

\(T(1) = 1\)

Recurrence Equation

6.0 - Divide and Conquer

  • We characterise the running times of divide-and-conquer algorithms using a recurrence equation (or just recurrence)
  • Formally, if \(T(n)\) is the running time of the algorithm:

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • where
    • \(D(n)\) is the cost of dividing the problem into subproblems
    • \(C(n)\) is the cost of combining solutions of the subproblems to create a solution for the problem

6.0 - Divide and Conquer

Mergesort

13

4

19

11

32

22

7

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\)
  • total number of operations: \(n\log n\)
  • recurrence equation

\(T(n) = 2T(n/2) + n\)

6.0 - Divide and Conquer

Mergesort

13

4

19

11

32

22

7

32

22

7

13

4

19

11

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\)
  • total number of operations: \(n\log n\)
  • recurrence equation

\(T(n) = 2T(n/2) + n\)

6.0 - Divide and Conquer

Mergesort

13

4

19

11

32

22

7

32

22

7

32

22

7

13

4

19

11

13

4

19

11

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\)
  • total number of operations: \(n\log n\)
  • recurrence equation

\(T(n) = 2T(n/2) + n\)

6.0 - Divide and Conquer

Mergesort

19

32

7

22

11

13

4

13

4

19

11

32

7

22

22

7

32

11

19

4

13

\(T(n) = 2T(n/2) + n\)

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\) (cost of combining)
  • total number of operations: \(n\log n\)
  • recurrence equation

6.0 - Divide and Conquer

Mergesort

19

32

7

22

11

13

4

13

4

19

11

32

7

22

22

7

32

11

19

4

13

\(T(n) = 2T(n/2) + n\)

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\) (cost of combining)
  • total number of operations: \(n\log n\)
  • recurrence equation

6.0 - Divide and Conquer

Mergesort

  • number of levels: \(\log n\)
  • number of operations on each level : \(n\) (cost of combining)
  • total number of operations: \(n\log n\)
  • recurrence equation

19

32

7

22

11

13

4

13

4

19

11

32

7

22

22

7

32

11

19

4

13

\(T(n) = 2T(n/2) + n\)

Recurrence Equation

6.0 - Divide and Conquer

  • We characterise the running times of divide-and-conquer algorithms using a recurrence equation (or just recurrence)
  • Formally, if \(T(n)\) is the running time of the algorithm:

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • where
    • \(D(n)\) is the cost of dividing the problem into subproblems
    • \(C(n)\) is the cost of combining solutions of the subproblems to create a solution for the problem

6.0 - Divide and Conquer

Binary Search

  • number of levels: \(\log n\)
  • number of operations on each level : \(1\) (cost of dividing)
  • total number of operations: \(\log n\)
  •  recurrence equation: 

13

4

19

11

32

7

22

13

19

32

22

13

19

13

\(T(n) = T(n/2) + 1\)

... or is it?

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • we don't have to use equality because sometimes we can't say for sure, so it is okay to have

Recurrence Equation

6.0 - Divide and Conquer

\(T(n) \le aT(n/b) + f(n)\)

\(T(n) \ge aT(n/b) + f(n)\)

or

  • it should be okay to just use equality most of the time in this unit, we don't have to be too precise
  • we also ignore the details on boundary conditions and non-exact division

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

Recurrence Equation

6.0 - Divide and Conquer

  • we also ignore the details on boundary conditions and non-exact division
    • e.g. merge sort - \(T(n) = 2T(n/2) + n\)
    • what if \(n\) is odd?
      • shouldn't we have something like  \(T(n) = T(\lfloor{n/2}\rfloor) + T(\lceil{n/2}\rceil + n \)

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

Recurrence Equation

6.0 - Divide and Conquer

  • we'll keep it simple:

\(n\)

\[\frac{n}{2}\]

\[\frac{n}{2}\]

\(n\)

\[\frac{n}{3}\]

\[\frac{n}{3}\]

\[\frac{n}{3}\]

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • how does this relate to the complexity notation?
    • \(T(n)\) is the running time of the algorithm
    • the complexity notation is the bound for the running time of the algorithm
    • e.g.
      • \(T(n) = \frac{1}{2}n^2 + \frac{1}{2}n\)
      • \(T(n) = \Theta(n^2)\)
      • \(T(n) \le aT(n/b) + f(n)\) \(\rightarrow\) \(T(n) = O(g(n))\)
      • \(T(n) \ge aT(n/b) + f(n)\) \(\rightarrow\) \(T(n) = \Omega(g(n))\)

Recurrence Equation

6.0 - Divide and Conquer

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • you can also use complexity notation to describe \(D(n)\) and/or \(C(n)\):
    • e.g.
      • \(T(n) = 2T(n/2) + n\)
      • \(T(n) = 2T(n/2) + \Theta(n)\) or \(T(n) = 2T(n/2) + O(n)\)
  • remember our goal is to find the asymptotic notation for \(T(n)\), that is eventually we want to be able to say that
    • \(T(n) = O(g(n))\) or \(T(n) = \Omega(g(n))\) or \(T(n) = \Theta(g(n))\)

Recurrence Equation

6.0 - Divide and Conquer

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le c$}\\ aT(n/b) + D(n) + C(n) &\text{otherwise} \end{cases}\)

  • summary:
    • you can use inequality as well \(<, \le, =, \ge, >)\)
    • don't worry about floor or ceiling when it comes to division
    • don't forget that we want an asymptotic bound out of this (we're not trying to be exact)

Recurrence Equation

6.0 - Divide and Conquer

Example - Linear Search

6.0 - Divide and Conquer

public static int linearSearch(int[] a, int i, int j, int target){
    if (i > j)
        return -1;
    if (a[i] == target)
    	return i;
    return linearSearch(a,i+1,j,target);
}
public static int linearSearch(int[] a, int i, int j, int target) {
	if (i > j) 
		return -1;
	int m = (i+j)/2;
	if (a[m] == target)
		return m;
	return Math.max(linearSearch(a,i,m-1,target), linearSearch(a,m+1,j,target));
}

Example - Linear Search

6.0 - Divide and Conquer

public static int linearSearch(int[] a, int i, int j, int target) {
	if (i > j) 
		return -1;
	int m = (i+j)/2;
	if (a[m] == target)
		return m;
	return Math.max(linearSearch(a,i,m-1,target), linearSearch(a,m+1,j,target));
}

linearSearch(a[], 0, 8, target)

linearSearch(a[], 0, 3, target)

linearSearch(a[], 5, 8, target)

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le 1$}\\ 2T(n/2) + \Theta(1)&\text{otherwise} \end{cases}\)

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le 1$}\\ 2T(n/2) + \Theta(1)&\text{otherwise} \end{cases}\)

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le 1$}\\ 3T(n/3) + \Theta(1)&\text{otherwise} \end{cases}\)

  • If we divide the array into two parts:
  • If we divide the array into three parts:

Example - Linear Search

6.0 - Divide and Conquer

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

22

7

19

4

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

22

7

19

4

Example - Quicksort

6.0 - Divide and Conquer

22

7

32

11

19

4

13

22

7

32

11

19

4

22

7

19

4

Example - Quicksort

6.0 - Divide and Conquer

13

32

11

22

7

19

4

Example - Quicksort

6.0 - Divide and Conquer

13

32

11

22

7

19

4

13

32

11

22

7

19

13

32

11

22

7

19

Example - Quicksort

6.0 - Divide and Conquer

13

32

11

22

7

19

4

13

32

11

22

7

19

13

32

11

22

7

19

13

32

11

22

19

13

32

11

22

19

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le 1$}\\ 2T(n/2) + \Theta(n)&\text{otherwise} \end{cases}\)

\(T(n) = \begin{cases} \Theta(1) &\text{if $n \le 1$}\\ T(n-1) + \Theta(n)&\text{otherwise} \end{cases}\)

  • Average case:
  • Worst case:

Example - Quicksort

6.0 - Divide and Conquer

  • We have discussed how to derive the recursive relation for \(T(n)\), and we mentioned that this is going to lead to an asymptotic bound for the running time of the algorithm, but we haven't actually showed how this can be done
  • We are going to discuss three methods:
    • recursion-tree method
    • substitution method
    • the master method

What's next

6.0 - Divide and Conquer