COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

5.1 - Greedy Algorithm

Greedy Algorithm

• What did we learn so far:
  • greedy algorithm doesn't go through all the subproblems
  • it chooses the locally optimal solution in the hope that it contributes to the globally optimal solution
  • .... and it may not work
  • what I didn't say: greedy algorithm is mostly for optimisation problems
     
• Chapter 16, in particular Section 16.2, pages 423-426, is a very good read (remember that I based DP and Greedy topics heavily on CLRS)

5.1 - Greedy Algorithm

Greedy Algorithm - Examples

• Prim's and Kruskal's algorithm (finding the minimum spanning tree)
• Dijkstra's algorithm (finding shortest path)
• The coin change problem (or rod making)
• Fractional knapsack
• Huffman encoding (Strings algorithm)

Non examples (i.e. problems you cannot solve using greedy algorithm)

• Knapsack
• Longest common subsequence
• Travelling salesman
• ... and a lot more

5.1 - Greedy Algorithm

Greedy Algorithm - Shortest Path

5.1 - Greedy Algorithm

10

15

3

A

F

C

D

• Find the shortest path from A to F

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

• Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

• Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

• Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

• Find the smallest set of vertices that covers all the edges

Greedy Algorithm - Minimum Vertex Cover

5.1 - Greedy Algorithm

• Even if greedy algorithm fails to give you the right answer, sometimes you still use it anyway because it runs so fast
  • maybe we can accept the answer that it produces is close enough
  • we use greedy algorithm in approximation algorithms for NP problems

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• With dynamic programming, we have four steps:
  • Show that there is an optimal substructure
  • Show the recursive relation that gives optimal solution
  • Find the value of the optimal solution (run the algorithm)
  • (optional) Find the combination that gives the optimal solution

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• With greedy algorithm, we add three more steps:
  • Show that there is an optimal substructure
  • Show the recursive relation that gives optimal solution
  • Show that if we make the greedy choice, only one subproblem remains
  • Show that it is safe to make the greedy choice
  • Find the value of the optimal solution (run the algorithm)
  • (optional) Find the combination that gives the optimal solution
  • Develop the recursive solution to an iterative one
    • (this step is similar to creating a bottom-up DP)

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• The two key ingredients are:
  • optimal substructure
    • optimal solution to the problem contains optimal solution to the subproblems
  • greedy-choice property
    • we can construct globally optimal solution by making locally optimal (greedy) choices
• Biggest difference between greedy and DP:
  • DP is bottom-up (even the top-down version), greedy is top-down

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• The first steps:
  • Show that there is an optimal substructure
  • Show the recursive relation that gives optimal solution
  • Show that if we make the greedy choice, only one subproblem remains
  • Show that it is safe to make the greedy choice
• With dynamic programming, we solve many subproblems and compare which one is best using the recursive relation
• In greedy, we don't solve all the subproblems, so you don't really have to give the recursive relation

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• With dynamic programming, we solve many subproblems and compare which one is best using the recursive relation
• In greedy, we don't solve all the subproblems, so you don't really have to give the recursive relation
  
   
• Remember that this is what recursive relation looks like

\[\text{minR}(n) = \begin{cases}\displaystyle\min_{i\ :\ b_i\le n}\left(1 +\text{minR}(n-b_i)\right) & \text{if $n > 0$}\\ \quad \quad \quad\quad 0 & \text{otherwise}\end{cases}\]

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• So alternatively, we can move away from the dynamic programming approach, and design a greedy algorithm using the following steps:
  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
  • Show that the problem has an optimal substructure after we make the greedy choice
    • that is, after we make the greedy choice, the optimal solution to our problem contains the optimal solution to the subproblem
    • this is mostly formality, because if we don't have optimal substructure, then you can't recurse

Elements of a Greedy Algorithm

5.1 - Greedy Algorithm

• So alternatively, we can move away from the dynamic programming approach, and design a greedy algorithm using the following steps:
  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
  • Show that the problem has an optimal substructure after we make the greedy choice
  • Find the value of the optimal solution (run the algorithm)
  • Develop the recursive solution to an iterative one

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

7

4

5

1

3

5

2

3

2

4

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Given a graph, find the minimum spanning tree

4

1

3

2

3

2

5.1 - Greedy Algorithm

• Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
• Show that the problem has an optimal substructure after we make the greedy choice
• Find the value of the optimal solution (run the algorithm)
• Develop the recursive solution to an iterative one

Example 1: Prim's Algorithm

5.1 - Greedy Algorithm

• Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

• We claim that we can always choose the cheapest edge connected to the tree that we have so far, without making a cycle
  • so there is only one subproblem to solve: find the minimum spanning tree after adding the node connected to this edge (i.e. the set of edges to make the MST with minimal weights) 
  • contrast with DP: we try every single edge

5.1 - Greedy Algorithm

• Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

• Proving the greedy-choice property:
  • suppose that both the purple and red node are not in the spanning tree yet, and the cheapest edge is the one going to the purple node

10

3

5.1 - Greedy Algorithm

• Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

• Proving the greedy-choice property:
  • suppose that both the purple and red node are not in the spanning tree yet, and the cheapest edge is the one going to the purple node
  • in the resulting MST, the red node must be connected to the purple node

10

3

???

5.1 - Greedy Algorithm

• Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve

Example 1: Prim's Algorithm

• Proving the greedy-choice property:
  • but then we can swap the two edges and still have a spanning tree, and the new one clearly has a lower weight
    • we have a contradiction!
    • so if you want to have the MST, you MUST pick the lower-weighted edge

3

???

5.1 - Greedy Algorithm

• Show that the problem has an optimal substructure after we make the greedy choice

Example 1: Prim's Algorithm

• Use cut-and-paste technique
  • we have the subproblem of finding the MST after picking the current node, that is, the set of edges that will make the MST
  • if the weight of this set of edges (from the subproblem) is not minimal (optimal), then we can find a 'better' set of edges with lower total weight

3

???

5.1 - Greedy Algorithm

• If you have coins with different denominations \(\{c_1,c_2,\dots c_k\}\), what is the minimum number of coins required to represent a sum \(N\)?

Example 2: Coin Changing Problem

\[\text{minC}(n) = \begin{cases}\displaystyle\min_{i\ :\ b_i\le n}\left(1 +\text{minC}(n-c_i)\right) & \text{if $n > 0$}\\ \quad \quad \quad\quad 0 & \text{otherwise}\end{cases}\]

\(C = \{c_1,c_2,c_3,c_4\} =\{1,25,50,100\}\)

\(1+\text{minC}(n-100)\)

\(1+\text{minC}(n-50)\)

\(1+\text{minC}(n-25)\)

\(1+\text{minC}(n-1)\)

\(\text{minC}(n)\)

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

\(C = \{c_1,c_2,c_3,c_4\} =\{1, 7, 10, 20\}\)

• Use greedy algorithm to find the \(\text{minC}(22)\)

• Use greedy algorithm to find the \(\text{minC}(15)\)

• Use greedy algorithm to find the \(\text{minC}(14)\)

• What conclusion can you draw?

5.1 - Greedy Algorithm

Example 2: Coin Changing Problem

\(C = \{c_1,c_2,c_3,c_4\} =\{1, 7, 10, 20\}\)

• Use greedy algorithm to find the \(\text{minC}(15)\)
  • 15 = 10 + 1 + 1 + 1 + 1 + 1 (6 coins)
  • 15 = 7 + 7 + 1 (3 coins)

• Use greedy algorithm to find the \(\text{minC}(14)\)
  • 14 = 10 + 1 + 1 + 1 + 1 (5 coins)
  • 14 = 7 + 7 (2 coins)

• Moral of the story: you don't always have the greedy-choice property

5.1 - Greedy Algorithm

Summary

• You need to justify why you can use greedy algorithm to solve an optimisation problem:
  • Show that it is safe to make a greedy choice such that our problem only has one subproblem to solve
  • Show that the problem has an optimal substructure after we make the greedy choice
  • Find the value of the optimal solution (run the algorithm)
  • Develop the recursive solution to an iterative one