COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

3.3 - Recursive Backtracking

3.3 - Recursive Backtracking

Recursive backtracking

  • Recursive backtracking (or just backtracking) is a technique to produce a solution by iterating through all possible configurations of the search space
    • i.e. what we have been doing in the previous video
  • At each node of the search tree we check if we have the solution:
    • if yes, great and we can stop
      •  unless we have to find more solutions
    • if not, then we can try to extend our solution
    • if we cannot go anywhere else, then we backtrack, going to a more primitive solution and starting again from there

Generating Combinations - 3-Letter Combinations

3.3 - Recursive Backtracking

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

d ?

"abd"

"ab"

"bcd"

"bc"

"d"

""

... ...

... ...

(and so on)

Generating Combinations - 3-Letter Combinations

3.3 - Recursive Backtracking

// generate combinations of length 'size' of the characters in the array 'characters'
public static void generate_combinations_of_size(char[] characters, int i, 
                                                 ArrayList<Character> result, int size) {
  // if i reaches the end of the array, we're done
  if (i >= characters.length) {
    // print it out if we have the required size
    if (result.size() == size) System.out.println(result);
  }
  else {
    // current element
    Character ch = characters[i];
    
    // recurse, include the i-th character
    result.add(ch);
    generate_combinations_of_size(characters,i+1,result,limit);
    
    // recurse, don't include the i-th character
    result.remove(ch);  
    generate_combinations_of_size(characters,i+1,result,limit);
    
  }
}

Generating Combinations - Rod Making

3.3 - Recursive Backtracking

public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if we get to the end of array but still can't make the sum, 
  // return false
  if (i >= b.length) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}
// recurse, include the i-th character
result.add(ch);
generate_combinations_of_size(characters,i+1,result,limit);
  
// recurse, don't include the i-th character
result.remove(ch);  
generate_combinations_of_size(characters,i+1,result,limit);

Generating Combinations

3.3 - Recursive Backtracking

Generating Combinations

3.3 - Recursive Backtracking

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

d ?

"abcd"

(and so on)

"abcde"

e ?

Generating Combinations

3.3 - Recursive Backtracking

Generating Combinations

3.3 - Recursive Backtracking

// generate combinations of length 'size' of the characters in the array 'characters'
public static void generate_combinations_of_size(char[] characters, int i, 
                                                 ArrayList<Character> result, int size) {
  // if i reaches the end of the array, we're done
  if (i >= characters.length) {
    // print it out if we have the required size
    if (result.size() == size) System.out.println(result);
  }
  else {
    // current element
    Character ch = characters[i];
    
    // recurse, include the i-th character
    result.add(ch);
    generate_combinations_of_size(characters,i+1,result,limit);
    
    // recurse, don't include the i-th character
    result.remove(ch);  
    generate_combinations_of_size(characters,i+1,result,limit);
    
  }
}

Generating Permutations

3.3 - Recursive Backtracking

// generate all permutations of the characters in the set h
public static void generate_permutation_using_set(HashSet<Character> h, 
                                                  ArrayList<Character> result) {
  // no character left in the set, so we are done
  if (h.isEmpty()) {
    System.out.println(result);
  }
	
  HashSet<Character> temp = new HashSet<Character>(h);
  for(Character c : temp) {
    // choose a character to add 
    h.remove(c); 
    result.add(c);
    generate_permutation_using_set(h,result);
    // backtracking
    result.remove(c); 
    h.add(c);
  }
}

Generating Permutations

3.3 - Recursive Backtracking

"a"

"b"

"z"

"c"

""

"d"

... ...

""

"a"

"b"

"y"

"c"

"d"

... ...

"b"

"c"

"z"

"d"

... ...

HashSet<Character> temp = new HashSet<Character>(h);
for(Character c : temp) {
  // choose a character to add 
  h.remove(c); 
  result.add(c);
  generate_permutation_using_set(h,result);
  // backtracking
  result.remove(c); 
  h.add(c);
}

Generating Permutations

3.3 - Recursive Backtracking

// generate all permutations of the array result
// start with call to generate_permutation(result,0,result.length-1)
// where result is the array that you want to permute
public static void generate_permutation(char[] result, int start, int end) {
  // print out the result
  if (start == end) 
    System.out.println(result);
  else {
    for(int i = start; i <= end; i++) {
      // swap the elements in start and i
      swap(result,start,i);
      generate_permutation(result,start+1,end);
      swap(result,start,i);
    }
  }
}

Generating Permutations

3.3 - Recursive Backtracking

  • Example: A B C D
    • A B C D
      • B C D
        • C D
          • D    print [ A B C D ]
        • D C
          • C     print [ A B D C ]
      • C B D
        • B D
          • D     print [ A C B D ]
        • D B
          • B     print [ A C D B ]
for(int i = start; i <= end; i++) {
  swap(result,start,i);
  generate_permutation(result,start+1,end);
  swap(result,start,i);
}

Generating Permutations

3.3 - Recursive Backtracking

for(int i = start; i <= end; i++) {
  swap(result,start,i);
  generate_permutation(result,start+1,end);
  swap(result,start,i);
}

ABCD

ABCD

BACD

DBCA

CBAD

ABCD

ACBD

ADCB

ABCD

ABDC

BACD

BCAD

BDCA

CBAD

CABD

CDAB

DBCA

DCBA

DACB

DACB

DABC

(swap A with A)

(swap A with B)

(swap A with C)

(swap A with D)

B ~ B

B ~ C

B ~ D

C ~ C

C ~ D

A ~ A

A ~ C

A ~ D

B ~ B

B ~ A

B ~ D

B ~ B

B ~ C

B ~ D

C ~ C

C ~ B

3.3 - Recursive Backtracking

Recursive backtracking

  • Recursive backtracking is essentially just a DFS traversal on the search space
  • Can you do BFS instead?

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

3.3 - Recursive Backtracking

Recursive backtracking

  • Recursive backtracking is essentially just a DFS traversal on the search space
  • Can you do BFS instead?
    • Remember that we want to generate all combinations of n bits, e.g. 000, 001, 010, 011, etc
    • We add a bit on every level

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

1

0

11

10

01

11

111

110

101

100

011

010

011

000

3.3 - Recursive Backtracking

Recursive backtracking

  • DFS:  a  \(\rightarrow\)  ab  \(\rightarrow\)  abc  \(\rightarrow\)  abcd  \(\rightarrow \dots \rightarrow\) (backtrack) \(\rightarrow\) ab
  • BFS :
    • generate a
    • generate ab, a, b
    • generate abc, ab, etc

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

1

0

11

10

01

11

111

110

101

100

011

010

011

000

3.3 - Recursive Backtracking

Recursive backtracking

  • BFS: you can just generate 111, 0111, 1101, 1011, 1110
    • i.e. generate bits of increasing length
    • generally DFS with recursion is just easier

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

1

0

11

10

01

11

111

110

101

100

011

010

011

000

3.3 - Recursive Backtracking

Branch and Bound

  • The code we had was inefficient:
// if i reaches the end of the array, we're done
if (i >= characters.length) {
  // print it out if we have the required size
  if (result.size() == size) System.out.println(result);
}

"a"

""

"ab"

"a"

"b"

""

"abc"

"ab"

"ac"

"a"

"bc"

"b"

"c"

a ?

b ?

c ?

""

""

d ?

"abcd"

3.3 - Recursive Backtracking

Branch and Bound

  • Quick fix:
// generate combinations of length 'size' of the characters in the array 'characters'
public static void generate_combinations_of_size(char[] characters, int i, 
                                                 ArrayList<Character> result, int size) {
  // if we already have more than 'size' characters in result, then we don't need
  // to go any further
  if (result.size() > size)
    return;
  // if i reaches the end of the array, we're done
  if (i >= characters.length) {
    // print it out if we have the required size
    if (result.size() == size) System.out.println(result);
  }
  else { ... }
}

3.3 - Recursive Backtracking

Branch and Bound

  • Four things to watch out for:
    • Have I got my answer?
      • Should I stop here? Should I move on
    • Can I go any further? Am I at the end?
    • Can I improve my answer? 
      • Add things, remove things?
    • Is there any point of improving my answer?

Example - Rod Making

3.3 - Recursive Backtracking

public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if we get to the end of array but still can't make the sum, 
  // return false
  if (i >= b.length) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}

Example - Rod Making

3.3 - Recursive Backtracking

[ 10 , 12 , 5 , 7 , 11 ]   L = 25

[ 12 , 5 , 7 , 11 ]   L = 15

[ 5 ,  7 , 11 ]    L = 3

pick 12

don't pick 12

...

...

...

...

...

...

[ 5 , 7 , 11 ]   L = 15

...

pick 10

pick 12

don't pick 10

don't pick 12

...

[ 12 , 5 , 7 , 11 ]   L = 25

[ 5 ,  7 , 11 ]    L = 13

[ 5 , 7 , 11 ]   L = 25

Example - Rod Making

3.3 - Recursive Backtracking

[ 50 , 2 , 18 , 11 , 9 , 23 , 5 , 10 , 30 , 6 , 17 ]   L = 27

[ 2 , 18 , 11 , ... , 17 ]   L = -23

[ 2 , 18 , 11 , ... , 17 ]   L = 27

[18 , 11 , ... , 17 ] L = -25

pick 2

don't pick 2

...

...

...

...

...

...

[ 18 , 11, ... , 17] L = -23

[18 , 11, ... , 17 ] L = 25

[ 18 , 11 , ... , 17 ] L = 27

...

...

pick 50

pick 2

don't pick 50

don't pick 2

Branch and Bound

3.3 - Recursive Backtracking

  • Some literature make a distinction between backtracking and branch-and-bound (or pruning, or early exit)
  • The idea is the same, you perform a DFS on the search tree, and at every node, you can make the decision to stop and backtrack
    • backtracking is a more general, you generally keep on going until it is no longer feasible (e.g. searching problem)
    • branch-and-bound adds an additional stopping constraint: if you have found a better solution (e.g. optimisation problem)

Branch and Bound

3.3 - Recursive Backtracking

  • With branch-and-bound method, we can apply a bounding function at every node to see if there is any point in doing further recursion
  • If there is no point in doing further recursions, then we can prune the search tree, reducing the number of cases that we need to evaluate
    • I have also seen pruning function and feasibility function being used instead of bounding function

Branch and Bound

3.3 - Recursive Backtracking

public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if we get to the end of array but still can't make the sum, 
  // return false
  if (i >= b.length) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}
public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if L is negative or we get to the end of the array
  // return false
  if (i >= b.length || L < 0) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}

Branch and Bound

3.3 - Recursive Backtracking

  • Be careful not to write an inefficient pruning function
    • our pruning function in the previous example is just an integer comparison, so it is O(1)






       
    • can you think of a bad pruning function?
public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if L is negative or we get to the end of the array
  // return false
  if (i >= b.length || L < 0) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}

Branch and Bound

3.3 - Recursive Backtracking

public static boolean solve(int[] b, int i, int L) {
  // if L == 0, we are done
  if (L == 0){
    return true;
  }
  // if L is negative or we get to the end of the array
  // return false
  if (i >= b.length || L < 0 || sum(b,i) < L) {
    return false;
  }
  // else keep on going with the remaining sum
  return solve(b,i+1,L-b[i]) || solve(b,i+1,L);
}

General Improvements

3.3 - Recursive Backtracking

[ 50 , 2 , 18 , 11 , 9 , 23 , 5 , 10 , 30 , 6 , 17 ]   L = 27

[ 2 , 18 , 11 , ... , 17 ]   L = -23

[ 2 , 18 , 11 , ... , 17 ]   L = 27

[18 , 11 , ... , 17 ] L = -25

pick 2

don't pick 2

[ 18 , 11, ... , 17] L = -23

[18 , 11, ... , 17 ] L = 25

[ 18 , 11 , ... , 17 ] L = 27

...

pick 50

pick 2

don't pick 50

don't pick 2

  • Can you make some more improvement?
    • this algorithm \(O(2^n)\), so there's probably not much more you can do about the code, but can change the data to make the pruning more efficient?

General Improvements

3.3 - Recursive Backtracking

  • If you sort the input in descending order, we can prune earlier:
    • [ 50, 2, 18, 11, 9, 23, 5, 10, 30, 6, 17 ]    L = 27
      • we are going to try a lot of combinations starting with 2
    • [ 50, 30, 23, 18, 17, 11, 10, 9, 6, 5, 2 ]    L = 27
      • we pretty much ignore the first 2 entries
      • we get our answer quite early 

General Improvements

3.3 - Recursive Backtracking

  • So should we always sort the data?
    • [ 50, 30, 23, 18, 17, 11, 10, 9, 6, 5, 2 ]    ​
    • [ 50, 2, 18, 11, 9, 23, 5, 10, 30, 6, 17 ]
  • L = 27, which input gives you the smaller search tree?
  • L = 12, which input gives you the smaller search tree?
     
  • We cannot be sure that sorting the input in descending order will always give the best performance
    • but it probably is a good idea
    • this is a form of heuristics

Heuristics

3.3 - Recursive Backtracking

  • "Heuristic search algorithms have an air of voodoo about them" - Skiena, page 247
  • A heuristic is a problem solving approach that is practical, but not guaranteed to be optimal (or even rational)
    • educated guess, rule of thumb, common sense, experience
    • the most fundamental form of heuristic is trial and error
    • you can read more about heuristics in Chapter 7 of Skiena

Heuristics

3.3 - Recursive Backtracking

  • How should you pick your next case?
    • random sampling
    • greedy method (pick best value for now) - sorting is a bit like this
    • prior knowledge or experience
    • do a quick local search
    • simulated annealing

General Improvements

3.3 - Recursive Backtracking

  • Work out what a good pruning function is
    • use heuristics
  • Use precomputation (will discuss in the workshop)
  • See if you can work backward (will also discuss in the workshop)
    • these are my input, what kind of search space do I have
    • this is the search space, what kind of input can produce this

Summary

3.3 - Recursive Backtracking

  • Recursive backtracking
    • Code using DFS approach
  • Branch and bound, Pruning function
  • Heuristics and other general improvements
  • Brute-force method can actually be hard