Types of Complexity

Time Complexity

Analysis of how long an algorithm will take to complete.

Space Complexity

Analysis of how much memory (space) an algorithm will use in order to complete.

Both are Important

But we will only cover time

Time Complexity

Number of Operations

• Abstract away hardware specific features
• Formally, we look at number of "operations"
• Mostly just constants
• Worst case complexity

Complexity of Linear Search

def search(value, numbers):
    """
    Linear search for a value in an array of numbers.
    Returns the index of the value or -1 if it does not exist.
    """
    for index, num in enumerate(numbers):  # A comparison
        if num == value:  # A comparison
            return index
        
    return -1

Binary Search?

def binary_search(value, array):
    """Return the index of the value in the array, or -1 if it does not exist."""
    start = 0
    end = len(array) - 1

    # How many times do we do this?
    while start <= end:  # A comparison
        middle = int((start + end) / 2)
        if array[middle] == value:  # A comparison
            return middle
        elif value < array[middle]:  # A comparison
            end = middle - 1
        else:
            start = middle + 1
            
    return -1

Binary Search Complexity

We divide the list in half each time. Assume that the number of items in the list is a power of two. That is,
n = 2^k and k = log n.

How many items are in the list at each iteration?

Average Case Complexity

Worst case complexity is rather pessimistic. Sometimes we want to know on average how our algorithm performs.

Difficult to calculate for most algorithms.

Linear Search (Average)

How would we calculate the average case complexity for linear search?

We have the series:

2 + 4 + 6 + ... + 2n = 2(1 + 2 + 3 + ... + n)

2 (n ( n + 1 ) / 2 ) / n = n + 1

􏰁ϴ​(n)

Sorting Algorithms

Bubble Sort Complexity

def bubble_sort(array):
    # How many comparisons do we make?

    for i in range(0, len(array) - 1):
        for j in range(0, len(array) - i - 1):
            if array[j] > array[j + 1]:  # A comparison
                array[j], array[j + 1] = array[j + 1], array[j]
                
    return array

Insertion Sort Complexity

def insertion_sort(array):
    # How many comparisons do we make?

    for i in range(1, len(array)):
        value = array[i]
        position = i
        for j in range(i, -1, -1):
            if array[j - 1] <= value:  # A comparison
                break
                
            array[j] = array[j - 1]
            
        array[j] = value
                
    return array

BogoSort

I have a sorting algorithm for you. How about we just generate ALL of the permutations of the array and then check if it's sorted?

What's the complexity of this algorithm?

Matrices

Matrix Multiplication

How many operations are performed when multiplying a a x b matrix by a b x c matrix?

Matrix Chain Multiplication

We know matrix multiplication is not commutative, but it is associative. Given matrices A₁, A₂, and A₃, we want to find A₁A₂A₃, but with the fewest number of operations.

Is A₁(A₂A₃) or (A₁A₂)A₃ better?

Algorithmic Paradigms

We covered one!

Greedy algorithms is an example

Brute Force Algorithm

When we "brute force" a problem, we try to solve it in the most straightforward way possible.

Many of the algorithms we covered are "brute force", in that we don't do anything special.

They are less efficient, but sometimes necessary.

Other Paradigms

• Dynamic programming
• Divide and conquer

Terminology

Complexity Terms

A constant time algorithm is O(1)

A logarithmic algorithm is O(log n)

A linear algorithm is O(n)

A linearithmic algorithm is O(n log n)

A polynomial algorithm is O(n^b)

O(bⁿ) is exponential

O(n!) is factorial

Tractability

https://xkcd.com/287/

http://xkcd.com/1425/

Tractable Problems

Problems that can be solved in worst case polynomial time are tractable.

Exponential and factorial algorithms cannot be solved in a reasonable amount of time for large inputs. These are intractable problems.

P and NP

Tractable problems belong to a class of problems called P (polynomial time).

Intractable problems that can be verified in polynomial time belong to class NP (nondeterministic polynomial time).

NP Complete

For a problem to be NP-Complete, it must belong to NP and reduce down to an algorithm that is also NP-Complete.

How do you find the first NP-Complete algorithm? Take a grad-level course.

3 - Satisfiability

P vs. NP

Are problems that are NP actually in class P? If so, then that means any problem can be solved in polynomial time. This has huge implications for cryptography and solving intractable problems.

The current thought is P is not equal to NP, but there is no proof.