COMP3010: Algorithm Theory and Design

Daniel Sutantyo, Department of Computing, Macquarie University

2.4 Correctness of Iterative Algorithms

Prelude

2.4 - Correctness of Iterative Algorithms

  • We discussed loop invariant informally in the previous lecture
    • exponentiation
    • find minimum value
    • selection sort (not finished)
  • For this lecture
    • finishing selection sort
    • Horner's method
  • See also pg. 19 of CLRS for insertion sort analysis

Definition

2.4 - Correctness of Iterative Algorithms

  • A loop invariant is a statement about the loop that must satisfy the following properties:
    • Initialisation: the loop invariant is true before the first iteration of the loop
    • Maintenance: if the loop invariant is true at the start of one iteration of the loop, then it is also true at the start of the next iteration
    • Termination: when the loop terminates, the loop invariant gives a useful property that can be used to show that the algorithm is correct

Definition

2.4 - Correctness of Iterative Algorithms

  • In the previous lecture, I said that \(0 \le i < n\) is not the loop invariant
    • it's just not very useful (see termination property from previous page)
    • it's like saying binary search is \(O(n^2)\)
void selection_sort(int arr[]){
  for (int i = 0; i < n-1; i++){
    int min_index = i;
    for (int j = i+1; j < n; j++){
      if (arr[j] < arr[min_index])
        min_index = j;
    }
    swap(i,min_index);
  }
}

What is it for?

2.4 - Correctness of Iterative Algorithms

  • We use loop invariant to prove the correctness of an algorithm (or part of the algorithm that uses loops)
  • In Comp3010, the algorithms are usually small, it is very likely that there is only one major loop in the algorithm, so that is all you have to care about
  • However, remember that a loop invariant can only handle parts of the code that has a loop
    • in particular, you still have to prove that the algorithm terminates

Loop invariant and induction

2.4 - Correctness of Iterative Algorithms

  • You can see the similarities between induction and loop invariants:
    • ​we want to prove that a statement is correct
    • we start by showing that the statement is correct in the base case (initialisation property)
    • we show that if the statement is correct at the start of one iteration, then it is also correct at the end of that iteration (maintenance property)
  • The difference is, with induction, you often try to show that the statement is true for all positive numbers
    • normally you show a formula is correct for all n
    • with programming, you only need to show that the statement is correct at the loop's termination

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

void selection_sort(int arr[]){
  for (int i = 0; i < n-1; i++){
    int min_index = i;
    for (int j = i+1; j < n; j++){
      if (arr[j] < arr[min_index])
        min_index = j;
    }
    swap(i,min_index);
  }
}
  • Loop invariant: The array arr[0 .. i-1] is sorted
    • at beginning: The array arr[0 .. -1] is sorted
    • at start of iteration: The array arr[0 .. k-1] is sorted
      • at end of iteration: The array arr[0 .. k] is sorted
    • at the end: The array arr[0 .. n-2] is sorted

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

void selection_sort(int arr[]){
  for (int i = 0; i < n-1; i++){
    // find the index of the smallest element in arr[i .. n-1]
    int min_index = find_minimum(arr,i);
    swap(i,min_index);
  }
}
  • Loop invariant: The array arr[0 .. i-1] is sorted
    • this is partially correct
    • maybe a better language:
      • the subarray arr[0 .. i-1] is in sorted order
    • the problem is that this only tells us that arr[0 .. n-2] is sorted on termination, so is there something we can say about arr[n-1]?

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

void selection_sort(int arr[]){
  for (int i = 0; i < n-1; i++){
    // find the index of the smallest element in arr[i .. n-1]
    int min_index = find_minimum(arr,i);
    swap(i,min_index);
  }
}

2

3

5

7

11

8

2

3

5

7

8

11

the last element is always the largest one!

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

void selection_sort(int arr[]){
  for (int i = 0; i < n-1; i++){
    // find the index of the smallest element in arr[i .. n-1]
    int min_index = find_minimum(arr,i);
    swap(i,min_index);
  }
}
  • Let's add to the loop invariant:
    • the subarray arr[0 .. i-1] is in sorted order
      • AND
    • all values in arr[0 .. i-1] are smaller than or equal to the values in arr[i .. n-1]

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

The subarray arr[0 .. i-1] is in sorted order

AND

all the values in arr[0 .. i-1] are smaller than or equal to the values in arr[i..n-1]

  • Initialisation:
    • at the start of the loop, before the first iteration, i = 0, there are no elements in the subarray arr[0 .. -1]
    • so the loop invariant is trivially true

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

The subarray arr[0 .. i-1] is in sorted order AND all the values in arr[0 .. i-1] are smaller than or equal to the values in arr[i..n-1]

  • Maintenance: (say i = k)
    • suppose that at start of iteration, arr[0 .. k-1] is sorted and the values in arr[0..k-1] are all smaller than or equal to values in arr[k .. n-1]
    • in the loop:​
      • ​we find index of smallest element in arr[k .. n-1] and swap it with arr[k]
    • at the end of the iteration:
      • arr[k] is now smaller than or equal to all the values in arr[k+1 .. n-1] but it was greater than or equal to all the values in arr[0 .. k-1]
      • so arr[0 .. k] is sorted and all elements in arr[0 .. k] are smaller than or equal to the values in arr[k+1 .. n-1]
      • i gets incremented to k+1, so the loop invariant is still true

Example: Selection Sort

2.4 - Correctness of Iterative Algorithms

The subarray arr[0 .. i-1] is in sorted order AND all the values in arr[0 .. i-1] are smaller than or equal to the values in arr[i..n-1]

  • Termination: (i = n-1)
    • The loop invariant now states
      • The subarray arr[0 .. n-2] is in sorted order AND arr[n-1] is greater than or equal to all the values in arr[0 .. n-2]
    • In other words the array is sorted
  • Notice again how this is just induction, all we do is show that the statement is correct for all valid i
  • You still need to show why the loop terminates!
    • ​actually, just do this at the start, say that i starts at 0, and in the loop, the only time we modify i is to increase it, so it will eventually hit n-1

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

horner(x,i):
 if (i == n) 
   return c_n 
 else 
   return c_i + x * evaluate(x,i+1)

\[ f(x) = c_nx^n + c_{n-1}x^{n-1} + \cdots +c_1x + c_0 \]

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[ f(x) = c_nx^n + c_{n-1}x^{n-1} + \cdots +c_1x + c_0 \]

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y

\[\text{horner}(x,i) = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

\[\text{horner}(x,0) = c_nx^{n} + c_{n-1}x^{n-1} + \cdots + c_{1}x + c_0\]

\[\text{horner}(x,n) = c_nx^{n}\]

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[\text{horner}(x,i) = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

\(i = n\)

\[c_nx^{n-n} + c_{n-1}x^{n-1-n} + \cdots + c_{n+1}x + c_n\]

\(i = n-1\)

\[c_nx^{n-n+1} + c_{n-1}x^{n-1-n+1} + \cdots + c_{n}x + c_{n-1}\]

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

  • Initialisation:
    • at the start of the loop, i = n, so the loop invariant states
// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y

\[y = c_n\]

but, it's not ... at the start of the loop, \(y = 0\)

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

  • Initialisation:
    • at the start of the loop, i = n, so the loop invariant states
// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y

\[y = c_n\]

but, it's not ... at the start of the loop, \(y = 0\)

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

  • Termination:
    • at the of the loop, i = -1, so the loop invariant states
// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y

\[y = c_nx^{n+1} + c_{n-1}x^{n} + \cdots + c_{0}x + c_{-1}\]

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y
// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n+1; i > 0; i--)
    y = c[i-1] + x * y;
  return y

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = 0\]

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n+1; i > 0; i--)
    y = c[i-1] + x * y;
  return y

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

  • Initialisation:
    • at the start of the loop, i = n+1, so the loop invariant states

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

\[y = c_nx^{n-k} + c_{n-1}x^{n-1-k} + \cdots + c_{k+1}x + c_k\]

  • Maintenance:
    • at the start of the iteration, i = k, so we have

then we multiply \(y\) by \(x\) and add \(c_{k-1}\)

\[y = c_nx^{n-k+1} + c_{n-1}x^{n-k} + \cdots + c_{k+1}x + c_kx + c_{k-1}\]

which is

\[y = c_nx^{n-(k-1)} + c_{n-1}x^{n-1-(k-1)} + \cdots + c_{k}x + c_{k-1}\]

this is just the loop invariant with \(i = k-1\)

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

so it is preserved

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

  • Termination:
    • at termination \(i = 0\), so we have
// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n+1; i > 0; i--)
    y = c[i-1] + x * y;
  return y

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

\[y = c_nx^{n} + c_{n-1}x^{n-1} + \cdots + c_{1}x + c_0\]

Example: Horner's method

2.4 - Correctness of Iterative Algorithms

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n+1; i > 0; i--)
    y = c[i-1] + x * y;
  return y

\[y = c_nx^{n-i} + c_{n-1}x^{n-1-i} + \cdots + c_{i+1}x + c_i\]

\[y = c_nx^{n-(i+1)} + c_{n-1}x^{n-1-(i+1)} + \cdots + c_{i+2}x + c_{i+1}\]

L2 :

L1 :

// assuming the coefficients are in c[0..n]
int horner(int x, int[] c):
  int y = 0, n = c.length-1;
  for (int i = n; i >= 0; i--)
    y = c[i] + x * y;
  return y