Data Structures
Lecture 1 - An Introduction
Algorithms
INPUT
OUTPUT
Algorithms
INPUT
OUTPUT
[ 2, 3, 4, 1 ]
quicksort()
[ 1, 2, 3, 4 ]
-
We can view algorithms as a tool for solving computational problems.
SORTING PROBLEM
INPUT A list of n elements
SORTING PROBLEM
INPUT A list of n elements
SORTING PROBLEM
OUTPUT A permutation of the list
such that
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 7, 2, 5, 11 ]
[ 2, 5, 7, 11 ]
,
)
(
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 7, 2, 5, 11 ]
[ 2, 5, 7, 11 ]
,
)
(
[ 44, 27, 31, 12, 29 ]
[ 12, 27, 29, 31, 44 ]
,
)
(
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 7, 2, 5, 11 ]
[ 7, 2, 5, 11 ]
,
)
(
[ 44, 27, 31, 12, 29 ]
[ 12, 27, 29, 31, 44 ]
,
)
(
.
.
.
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 7, 2, 5, 11 ]
[ 7, 2, 5, 11 ]
,
)
(
[ 44, 27, 31, 12, 29 ]
[ 12, 27, 29, 31, 44 ]
,
)
(
.
.
.
[ 2, 3, 4, 1 ]
[ 1, 2, 3, 4 ]
,
)
(
[ 7, 2, 5, 11 ]
[ 7, 2, 5, 11 ]
,
)
(
[ 44, 27, 31, 12, 29 ]
[ 12, 27, 29, 31, 44 ]
,
)
(
.
.
.
-
A computational problem can be viewed as a set of problem instances.
-
When does an algorithm solve a computational problem?
-
When does an algorithm solve a computational problem?
-
The algorithm must halt on every input.
-
When does an algorithm solve a computational problem?
-
The algorithm must halt on every input.
-
The algorithm must produce the correct output for every input.
Data Structures
Data structures are a way to...
Data Structures
Data structures are a way to...
Store
Data Structures
Data structures are a way to...
Store
Organize
Data Structures
Data structures are a way to...
Store
Organize
Modify
Data Structures
Data structures are a way to...
Store
Organize
Modify
Access
Data Structures
Data structures are a way to...
Store
Organize
Modify
Access
...data!
Data Structures
Algorithms
5
7
8
9
6
5
7
8
9
6
Check the first entry is sorted with respect to itself
5
7
8
9
6
Check the first two entries are sorted with respect to each other
5
7
8
9
6
Compare the last entry to the first entry...
5
7
8
9
6
Since 7 < 9 we don't need to do anything!
5
7
8
9
6
Now we check the first three elements are sorted!
5
7
8
9
6
We compare the second and third elements...
5
7
8
9
6
Since 5 < 9 we need to rearrange...
7
8
9
6
So let's remove 5 and store it...
key = 5
7
8
6
And move 9 into the space left behind...
key = 5
9
7
8
6
Next we compare the first element with our key...
key = 5
9
7
8
6
Since 7 > 5 we need to rearrange...
key = 5
9
8
6
So let's put 7 in the empty space!
key = 5
9
7
8
6
Since there are no elements left to compare to, we know the key is the smallest element and place it in the remaining space.
key = 5
9
7
5
8
6
Since there are no elements left to compare to, we know the key is the smallest element and place it in the remaining space.
key = 5
9
7
5
8
6
Now the first three elements are sorted!
key = 5
9
7
5
8
6
Now let's sort the first four elements!
key = 5
9
7
5
8
6
Just in case we need it later, we might as well set the key to 6 straight away!
key = 6
9
7
5
8
6
Let's compare elements in reverse order again...
key = 6
9
7
5
8
Since 6 < 9 we need to remove 6...
key = 6
9
7
5
8
... and put 9 in the empty space.
key = 6
9
7
5
8
Then we can compare the next element with the key...
key = 6
9
7
5
8
Since 7 > 6 we need to...
key = 6
9
7
5
8
... put 7 in the empty space.
key = 6
9
7
5
8
Now we compare the key with the first element.
key = 6
9
7
5
8
But 6 > 5, so we don't need to move 5, and can place 6 into the empty space!
key = 6
9
7
5
8
But 6 > 5, so we don't need to move 5, and can place 6 into the empty space!
key = 6
9
7
5
6
8
Now the first 5 elements are sorted!
key = 6
9
7
5
6
8
All that remains is the sixth element!
key = 6
9
7
5
6
8
Let's store it in the key straight away...
key = 8
9
7
5
6
8
Now let's check the key against the fifth element.
key = 8
9
7
5
6
8
Since 8 < 9 we have to...
key = 8
9
7
5
6
...delete 8...
key = 8
9
7
5
6
...and put 9 in the empty space.
key = 8
9
7
5
6
Now we compare the key to the next element.
key = 8
9
7
5
6
But 7 < 8 so...
key = 8
9
7
5
6
We can just insert the key into the empty space!
key = 8
9
7
5
6
8
Now the list is fully sorted!
key = 8
9
7
5
6
8
Now the list is fully sorted!
key = 8
9
7
5
6
8
-
How do we take the ideas we used to sort this list to design an algorithm that works on all lists?
INSERTION SORT(A)
for
j = 0
to
A.length
key = A[ j ]
i = j - 1
while
i > -1
and
A[ i ] > key
A[ i + 1 ] = A[ i ]
i = i - 1
A[ i + 1 ] = A[ key ]
-
How can we be sure that insertion sort always returns the correct output?
-
We can use loop invariants!
-
Which is basically just induction...
BASE CASE
In the first iteration of the loop we need to check if the first element of the list is sorted. This is trivially true.
CLAIM
On the jth loop, the first j+1 elements of the list are sorted.
STEP CASE
It is easy to see that the newly considered element is placed in the position of the smallest element that is greater than the new element.
STEP CASE
It is easy to see that the key is placed in the position of the smallest element amongst the j - 1 first elements that is greater than the key.
It is also easy to see that all elements greater than the key have their index increased by one, so the key is correctly sorted with respect to its larger elements.
The new element is correctly sorted with respect to its smaller elements as it replaced a larger element and the original j - 1 elements were correctly sorted at the beginning of the loop.
STEP CASE
Now we only need to check the original j -1 elements are correctly sorted. Pick an arbitrary element which was not the key.
Assume the index of the element was increased during the loop. This means it was greater than the key.
Since the index of any element can be increased by at most one, this means the element is still sorted with respect to its smaller elements.
STEP CASE
Likewise, if the index of the element was increased, so was the index of its greater elements. So, the chosen element is still correctly sorted with respect to its larger elements.
Thus we conclude that any element that had its index increased is still sorted correctly.
If an element did not have its index increased, then neither did its smaller elements, so the element is still sorted with respect to its smaller elements.
STEP CASE
Since the index of an existing element cannot be decreased, the element must still be sorted with respect to its greater elements.
Thus any element whose index did not change is still correctly sorted.
Since we have covered all cases the step case is now complete!
Conclusion
We have shown by induction that at the end of every loop j , the first j elements of the list are sorted.
Since the loop runs for n steps, where n is the length of the list, we conclude that the list must be fully sorted once it terminates!
Hence insertion sort solves the sorting problem!
A Better Way?
-
Insertion sort is inspired by how humans sort things one at a time.
-
Can we do better than this?
-
Many algorithms are design according to the principle of divide and conquer.
Divide and Conquer
Divide the problem into subproblems
Conquer the subproblem by solving it recursively
Combine the subproblems into the solution of the original problem
9
6
5
7
8
4
Divide the problem into subproblems until we can't anymore!
9
6
5
7
8
4
6
5
7
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Conquer each subproblem
9
6
5
7
8
4
6
5
7
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Conquer each subproblem
9
6
5
7
8
4
6
5
7
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Merge the solutions
9
6
5
7
8
4
6
5
7
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Conquer
9
7
5
6
8
4
7
5
6
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Merge
9
6
5
7
8
4
7
5
6
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Conquer
9
6
5
7
8
4
7
5
6
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Merge
9
6
5
7
8
4
7
5
6
9
8
4
5
7
6
7
5
9
8
4
4
8
9
Conquer
9
6
5
7
8
4
7
5
6
8
4
9
5
7
6
7
5
9
8
4
8
9
Merge
8
6
4
5
7
9
7
5
6
8
4
9
5
7
6
7
5
9
8
4
8
9
Merge
MERGESORT(A, start, end)
if start < end
middle = floor((start + end) / 2)
MERGESORT(A, start, middle)
MERGESORT(A, middle + 1, end)
MERGE(A, middle + 1, end)
Divide and conquer algorithms are naturally recursive!
Next Time
-
How do we merge?
-
How can we compare mergesort and insertion sort?