COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

5.1 - Greedy Algorithm

Greedy Algorithm

  • What did we learn so far:
    • greedy algorithm doesn't go through all the subproblems
    • it chooses the locally optimal solution in the hope that it contributes to the globally optimal solution
    • .... and it may not work
    • what I didn't say: greedy algorithm is mostly for optimisation problems
       
  • Chapter 16, in particular Section 16.2, pages 423-426, is a very good read (remember that I based DP and Greedy topics heavily on CLRS)

5.1 - Greedy Algorithm

Greedy Algorithm - Examples

  • Prim's and Kruskal's algorithm (finding the minimum spanning tree)
  • Dijkstra's algorithm (finding shortest path)
  • The coin change problem (or rod making)
  • Fractional knapsack
  • Huffman encoding (Strings algorithm)

Non examples (i.e. problems you cannot solve using greedy algorithm)

  • Knapsack
  • Longest common subsequence
  • Travelling salesman
  • ... and a lot more

 

5.1 - Greedy Algorithm

Greedy Algorithm - Shortest Path

5.1 - Greedy Algorithm

10

15

3

A

F

C

D

3

14

  • Find the shortest path from A to F

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

  • Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

  • Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

  • Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

  • Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

  • Even if greedy algorithm fails to give you the right answer, sometimes you still use it anyway because it runs so fast
    • maybe we can accept the answer that it produces is close enough
    • we use greedy algorithm in approximation algorithms for NP problems

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • With dynamic programming, we have four steps:
    • Show that there is an optimal substructure
    • Show the recursive relation that gives optimal solution
    • Find the value of the optimal solution (run the algorithm)
    • (optional) Find the combination that gives the optimal solution

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • With greedy algorithm, we add three more steps:
    • Show that there is an optimal substructure
    • Show the recursive relation that gives optimal solution
    • Show that if we make the greedy choice, only one subproblem remains
    • Show that it is safe to make the greedy choice
    • Find the value of the optimal solution (run the algorithm)
    • (optional) Find the combination that gives the optimal solution
    • Develop the recursive solution to an iterative one
      • (this step is similar to creating a bottom-up DP)

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • The two key ingredients are:
    • optimal substructure
      • optimal solution to the problem contains optimal solution to the subproblems
    • greedy-choice property
      • we can construct globally optimal solution by making locally optimal (greedy) choices
  • Biggest difference between greedy and DP:
    • DP is bottom-up (even the top-down version), greedy is top-down

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • The first steps:
    • Show that there is an optimal substructure
    • Show the recursive relation that gives optimal solution
    • Show that if we make the greedy choice, only one subproblem remains
    • Show that it is safe to make the greedy choice
  • With dynamic programming, we solve many subproblems and compare which one is best using the recursive relation
  • In greedy, we don't solve all the subproblems, so you don't really have to give the recursive relation

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • With dynamic programming, we solve many subproblems and compare which one is best using the recursive relation
  • In greedy, we don't solve all the subproblems, so you don't really have to give the recursive relation

     
  • Remember that this is what recursive relation looks like

\[\text{minR}(n) = \begin{cases}\displaystyle\min_{i\ :\ b_i\le n}\left(1 +\text{minR}(n-b_i)\right) & \text{if $n > 0$}\\ \quad \quad \quad\quad 0 & \text{otherwise}\end{cases}\]

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • So alternatively, we can move away from the dynamic programming approach, and design a greedy algorithm using the following steps:
    • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
    • Show that the problem has an optimal substructure after we make the greedy choice
      • that is, after we make the greedy choice, the optimal solution to our problem contains the optimal solution to the subproblem
      • this is mostly formality, because if we don't have optimal substructure, then you can't recurse

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

  • So alternatively, we can move away from the dynamic programming approach, and design a greedy algorithm using the following steps:
    • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
    • Show that the problem has an optimal substructure after we make the greedy choice
    • Find the value of the optimal solution (run the algorithm)
    • Develop the recursive solution to an iterative one

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Given a graph, find the minimum spanning tree

4

1

3

2

3

2

5.1 - Greedy Algorithm

  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
  • Show that the problem has an optimal substructure after we make the greedy choice
  • Find the value of the optimal solution (run the algorithm)
  • Develop the recursive solution to an iterative one

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

  • We claim that we can always choose the cheapest edge connected to the tree that we have so far, without making a cycle
    • so there is only one subproblem to solve: find the minimum spanning tree after adding the node connected to this edge (i.e. the set of edges to make the MST with minimal weights) 
    • contrast with DP: we try every single edge

5.1 - Greedy Algorithm

  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

  • Proving the greedy-choice property:
    • suppose that both the purple and red node are not in the spanning tree yet, and the cheapest edge is the one going to the purple node

10

3

5.1 - Greedy Algorithm

  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

  • Proving the greedy-choice property:
    • suppose that both the purple and red node are not in the spanning tree yet, and the cheapest edge is the one going to the purple node
    • in the resulting MST, the red node must be connected to the purple node

10

3

???

5.1 - Greedy Algorithm

  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

  • Proving the greedy-choice property:
    • but then we can swap the two edges and still have a spanning tree, and the new one clearly has a lower weight
      • we have a contradiction!
      • so if you want to have the MST, you MUST pick the lower-weighted edge

3

???

5.1 - Greedy Algorithm

  • Show that the problem has an optimal substructure after we make the greedy choice

Example 1: Prim's Algorithm

  • Use cut-and-paste technique
    • we have the subproblem of finding the MST after picking the current node, that is, the set of edges that will make the MST
    • if the weight of this set of edges (from the subproblem) is not minimal (optimal), then we can find a 'better' set of edges with lower total weight

3

???

5.1 - Greedy Algorithm

  • If you have coins with different denominations \(\{c_1,c_2,\dots c_k\}\), what is the minimum number of coins required to represent a sum \(N\)?

Example 2: Coin Changing Problem

\[\text{minC}(n) = \begin{cases}\displaystyle\min_{i\ :\ b_i\le n}\left(1 +\text{minC}(n-c_i)\right) & \text{if $n > 0$}\\ \quad \quad \quad\quad 0 & \text{otherwise}\end{cases}\]

\(C = \{c_1,c_2,c_3,c_4\} =\{1,25,50,100\}\)

\(1+\text{minC}(n-100)\)

\(1+\text{minC}(n-50)\)

\(1+\text{minC}(n-25)\)

\(1+\text{minC}(n-1)\)

\(\text{minC}(n)\)

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

  • The greedy choice:
    • if \(c_\ell \le n < c_{\ell+1}\), then choose \(c_\ell\), or choose \(c_k\) if \(c_k \le n\)
    • if we do not take \(c_\ell\) (or \(c_k\)), then we need to replace this by smaller values:
      • if \(n \ge 100\) and we do not take 100, then we need to replace it with 2 50s
      • if \(50 \le n < 100\) and we do not take 50, then we need to replace it with 2 25s
      • if \(25 \le n < 50\) and we do not take 25, then we need to replace it with 25 1s
    • in all cases, we end up with a worse solution, therefore our greedy choice is the best optimal choice

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

  • Why is the proof so long and specific?
    • as in, I had to use the values 1, 25, 50, 100
  • Why can't we just prove it for any arbitrary \(\{c_1, c_2, c_3, \dots c_n\}\)



(by the way, I'm not going to prove optimal substructure here, because it's just the same cut-and-paste argument again)

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

\(C = \{c_1,c_2,c_3,c_4\} =\{1, 7, 10, 20\}\)

  • Use greedy algorithm to find the \(\text{minC}(22)\)
  • Use greedy algorithm to find the \(\text{minC}(15)\)
  • Use greedy algorithm to find the \(\text{minC}(14)\)
  • What conclusion can you draw?

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

\(C = \{c_1,c_2,c_3,c_4\} =\{1, 7, 10, 20\}\)

  • Use greedy algorithm to find the \(\text{minC}(15)\)
    • 15 = 10 + 1 + 1 + 1 + 1 + 1 (6 coins)
    • 15 = 7 + 7 + 1 (3 coins)
  • Use greedy algorithm to find the \(\text{minC}(14)\)
    • 14 = 10 + 1 + 1 + 1 + 1 (5 coins)
    • 14 = 7 + 7 (2 coins)
  • Moral of the story: you don't always have the greedy-choice property

5.1 - Greedy Algorithm

Summary

  • You need to justify why you can use greedy algorithm to solve an optimisation problem:
    • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
    • Show that the problem has an optimal substructure after we make the greedy choice
    • Find the value of the optimal solution (run the algorithm)
    • Develop the recursive solution to an iterative one