COMP3010: Algorithm Theory and Design
Daniel Sutantyo, Department of Computing, Macquarie University
5.0 - Greedy Algorithm - Prelude
What have we done so far
- Complexity and Correctness
- Brute force algorithm
- Dynamic programming
5.0 - Greedy Algorithm - Prelude
What's next?
- Greedy algorithm (based on CLRS Chapter 16)
- Divide and conquer
- Divide and conquer
Original problem
Subproblem A
Subproblem B
Subproblem C
Subproblem D
Subproblem E
Subproblem F
Subproblem G
Subproblem H
Subproblem I
Subproblem J
Subproblem K
Subproblem M
Subproblem N
Subproblem O
Subproblem P
Subproblem L
Problem and Subproblems
5.0 - Greedy Algorithm - Prelude
- Dynamic programming
Problem and Subproblems
Original problem
Subproblem A
Subproblem B
Subproblem C
Subproblem D
Subproblem E
Subproblem F
Subproblem G
Subproblem H
Subproblem I
Subproblem J
Subproblem K
Subproblem M
Subproblem N
Subproblem O
Subproblem P
Subproblem L
5.0 - Greedy Algorithm - Prelude
- Greedy algorithm
- make the choice that looks best at the moment
- local optimal vs global optimal
- local optimal: the optimal choice, based on what you know at this time
- global optimal: the actual optimal choice (that leads to the optimal solution)
What is Greedy Algorithm?
10
15
3
A
F
C
D
3
14
5.0 - Greedy Algorithm - Prelude
Greedy vs DP vs Brute Force
- In terms of coding, greedy algorithm is probably the easiest to do
- which is why there aren't going to be any coding assignment based on greedy algorithm
- however, in terms of theory, greedy algorithm is probably the hardest one
- because you have to justify why it works
- it's easy to see that brute force works
- difficulty: did I generate all possible subproblems?
- it's easy to apply top-down dynamic programming
- difficulty: did I see any overlapping subproblems?
5.0 - Greedy Algorithm - Prelude
- Greedy does not solve all the subproblems
Original problem
Subproblem A
Subproblem B
Subproblem C
Subproblem D
Subproblem E
Subproblem F
Subproblem G
Subproblem H
Subproblem I
Subproblem J
Subproblem K
Subproblem M
Subproblem N
Subproblem O
Subproblem P
Subproblem L
Greedy vs DP vs Brute Force
5.0 - Greedy Algorithm - Prelude
Example: More Rod Making
We have a bunch of metallic rods with fixed lengths. Is it possible to produce a rod of certain length if we are allowed to join any two or more rods together, but we cannot cut any rod into two or more pieces.
- Input: The array \(B = \{b_1, b_2, \dots, b_n\}\) containing the lengths of the rods and an integer \(L \ge 0\)
- Output: True if there is a subarray \(\{b_{i_0}, b_{i_1}, \dots, b_{i_k}\}, 1 \le i_j \le n\) such that
\[\sum_{j=0}^k b_{i_j} = L\]
5.0 - Greedy Algorithm - Prelude
Example: More Rod Making
[ 50 , 2 , 18 , 11 , 9 , 23 , 5 , 10 , 30 , 6 , 17 ] L = 27
[ 2 , 18 , 11 , ... , 17 ] L = -23
[ 2 , 18 , 11 , ... , 17 ] L = 27
[18 , 11 , ... , 17 ] L = -25
pick 2
don't pick 2
...
...
...
...
...
...
[ 18 , 11, ... , 17] L = -23
[18 , 11, ... , 17 ] L = 25
[ 18 , 11 , ... , 17 ] L = 27
...
...
pick 50
pick 2
don't pick 50
don't pick 2
5.0 - Greedy Algorithm - Prelude
Example: More Rod Making
[ 50 , 30 , 23 , 18 , 17 , 11 , 10 , 9 , 6 , 5 , 2 ] L = 27
[ 18 , 17 , 11 , ... , 2 ] L = 4
[ 18 , 17 , 11 , ... , 2 ] L = 27
[ ] L = 2
pick 2
don't pick 2
[ ] L = 4
[9 , 6 , ... , 2 ] L = 9
[ 9 , 6 , ... , 2 ] L = 27
pick 23
pick 18
don't pick 23
don't pick 18
pick 9
but greedy algorithm doesn't try all subproblems!
5.0 - Greedy Algorithm - Prelude
Example: More Rod Making
We have an infinite number of metallic rods with fixed lengths. Is it possible to produce a rod of certain length if we are allowed to join any two or more rods together, but we cannot cut any rod into two or more pieces.
- Input: The array \(B = \{b_1, b_2, \dots, b_n\}\) containing the lengths of the rods and an integer \(L \ge 0\)
- Output: True if we can find constants \(c_i \ge 0\), \(1 \le i \le n\), such that
\[\sum_{i=1}^n c_i b_i = L\]
5.0 - Greedy Algorithm - Prelude
Example: More Rod Making
We have an infinite number of metallic rods with fixed lengths. Is it possible to produce a rod of certain length if we are allowed to join any two or more rods together, but we cannot cut any rod into two or more pieces.
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
5.0 - Greedy Algorithm - Prelude
Rod Making - Divide and Conquer
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- How do we perform divide and conquer?
- let's make this very simple:
- call the problem f
- e.g. f(265) = the minimum number of rods required to make a rod of length 265
- how do you break the problem down?
- let's make this very simple:
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- divide and conquer:
f(265)
f(132)
f(133)
- 50 + 11 + 50 + 11 + 50 + 12 + 50 + 11 (i.e. a lot of rods)
- (this sounds stupid, right?)
f(61)
f(61)
f(62)
f(61)
Rod Making - Divide and Conquer
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- brute force
- try every combination?
- are you going to try 1 + 1 + 1 + 1 + ... + 1 ?
Rod Making - Brute Force
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
Rod Making - Brute Force
f(265)
f(165)
f(65)
f(215)
f(240)
f(264)
f(115)
f(140)
f(164)
f(115)
f(165)
f(140)
f(190)
f(164)
f(214)
f(239)
f(263)
...
...
f(15)
f(15)
f(15)
f(115)
f(15)
pick 100
pick 50
pick 25
pick 1
- can you use branch and bound?
- are you going to try 1 + 1 + 1 + 1 + ... + 1?
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- are there overlapping subproblems?
Rod Making - Dynamic Programming
f(265)
f(165)
f(65)
f(215)
f(240)
f(264)
f(115)
f(140)
f(164)
f(115)
f(165)
f(140)
f(190)
f(164)
f(214)
f(239)
f(263)
...
...
f(15)
f(15)
f(15)
f(115)
f(15)
pick 100
pick 50
pick 25
pick 1
pick 100
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- We have overlapping subproblems
- What else do need to have?
- Show that there is an optimal substructure
- Show the recursive relation that gives optimal solution
- Find the VALUE of the optimal solution
- e.g. find the least number of rods that you need
- (optional) Find the COMBINATION that gives the optimal solution
- e.g. find the actual combination of rods that give you the optimal answer
Rod Making - Dynamic Programming
5.0 - Greedy Algorithm - Prelude
- Example: Given [ 100, 50, 25, 1 ]
- can we make 265?
- what is the minimum number of rods that we need?
- We have overlapping subproblems
- Show that there is an optimal substructure
- A problem exhibits optimal substructure if the optimal solution to the problem can be constructed using optimal solutions to the subproblems
- Maybe we should define the problem more formally
- Let minR(n) be the function that returns the minimum number of rods to construct a rod of length n
Rod Making - Dynamic Programming
5.0 - Greedy Algorithm - Prelude
- Let minR(n) be the function that returns the minimum number of rods to construct a rod of length n
Rod Making - Dynamic Programming
minR(n)
1+ minR(n-100)
1+minR(n-50)
1+minR(n-25)
1+minR(n-1)
pick 50
pick 25
pick 1
pick 100
5.0 - Greedy Algorithm - Prelude
Rod Making - Dynamic Programming
minR(n)
1+ minR(n-100)
1+minR(n-50)
1+minR(n-25)
1+minR(n-1)
pick 50
pick 25
pick 1
pick 100
- Suppose n = 265, and suppose that the optimal solution is 18 bars:
- 2 bars of length 100,
- 1 bar of length 50,
- 15 bars of length 1)
- which branch gives you this solution (what did you pick first)
5.0 - Greedy Algorithm - Prelude
Rod Making - Dynamic Programming
minR(n)
1+ minR(n-100)
1+minR(n-50)
1+minR(n-25)
1+minR(n-1)
pick 50
pick 25
pick 1
pick 100
- Suppose that our algorithm decides that the 2nd branch is the optimal path, i.e. the subproblem minR(215)
- we have to solve minR(215) to get the optimal solution to minR(265) (the original problem)
- the solution to minR(215) that we are using must be optimal as well, i.e. 17 is the optimal answer
- because if there is a better solution to minR(215), e.g 13, then the optimal solution to minR(265) would be 14, not 18
- a contradiction
5.0 - Greedy Algorithm - Prelude
Rod Making - Dynamic Programming
- To generalise:
- we have the optimal solution to minR(n) (say minR(n) = x)
- the optimal solution to minR(n) uses the optimal solution to minR(n-b)
- if the solution to minR(n-b) is NOT optimal
- then there is a better solution to minR(n-b), and this means there is a better solution to minR(n), say y with y < x
- so if we have the optimal solution to minR(n) then it must use the optimal solution to its subproblems
\(\text{minR}(n)\)
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
...
...
5.0 - Greedy Algorithm - Prelude
Rod Making - Dynamic Programming
\(\text{minR}(n)\)
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
...
...
5.0 - Greedy Algorithm - Prelude
- What about the recursive relation?
- what is the relationship between the problem and its subproblems, i.e. how we choose which subproblem do we choose as part of our answer?
Rod Making - Dynamic Programming
\(\text{minR}(n)\)
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
...
...
5.0 - Greedy Algorithm - Prelude
\[\text{minR}(n)\]
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
Rod Making - Dynamic Programming
\(\text{minR}(n)\)
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
...
...
5.0 - Greedy Algorithm - Prelude
\[\text{minR}(n) = \begin{cases}\displaystyle\min_{i\ :\ b_i\le n}\left(1 +\text{minR}(n-b_i)\right) & \text{if $n > 0$}\\ \quad \quad \quad\quad 0 & \text{otherwise}\end{cases}\]
- if you remember the code for rod cutting, this is going to be very similar
Rod Making - Dynamic Programming
\(\text{minR}(n)\)
\(1+\text{minR$(n-b_1)$}\)
\(1+\text{minR}(n-b_2)\)
\(1+\text{minR}(n-b_k)\)
...
...
...
5.0 - Greedy Algorithm - Prelude
- we spent so much time discussing how to solve this problem using divide and conquer, brute force, dynamic programming
- in all cases, we're going through the whole search tree, which is an overkill
- in reality, and hopefully can already see it, this problem is very simple
Rod Making - Greedy Algorithm
5.0 - Greedy Algorithm - Prelude
- Argument: the larger the rods that I use, the faster I can make a rod of length 265
- so pick 100
- then pick 100 again
- then pick 50, then finish it off with 1s
- We can argue that you can't do better (you already saw this, right?)
minR(265)
1+ minR(165)
1+minR(215)
1+minR(240)
1+minR(264)
pick 50
pick 25
pick 1
pick 100
Rod Making - Greedy Algorithm
5.0 - Greedy Algorithm - Prelude
- Can you always do this? Always pick the largest rod that you have?
Rod Making - Greedy Algorithm
5.0 - Greedy Algorithm - Prelude
- Can you always do this? Always pick the largest rod that you have?
- Example: Given [ 70, 50, 10, 1 ]
- can we make 100?
- what is the minimum number of rods that we need?
minR(100)
1+ minR(30)
1+minR(50)
1+minR(90)
1+minR(99)
pick 50
pick 10
pick 1
pick 70
Rod Making - Greedy Algorithm
5.0 - Greedy Algorithm - Prelude
- So can we use greedy or not?
- This is why greedy algorithm is the hardest of the bunch
- The algorithm is easy, and it's extremely efficient, but PROVING that it works is not trivial