Asymptotic Complexity

Agenda

• Prerequisites
• Why is the Analysis of Algorithms important?
• What's the goal of Asymptotic Complexity?
• What are we really measuring?
• RAM computational model
• Machine Operations as a Function
• Asymptotic Complexity
• Asymptotic Notation (Θ, O, Ω)

Prerequisites

• Basic Programming
• Basic Algebra
• Mathematical Functions
• Graphing
• Logarithms

Why is the Analysis of Algorithms Important?

• Foundational concept for computation
• Design performant algorithms
• Understand language primitives
• Develop an intuition for scale
• Identify intractable problems
• Learn patterns and techniques from the "greatest hits" of programming
• Impetus for my book (https://hideoushumpbackfreak.com/algorithms/)

Litmus Test

If you think you don't need it, or you know it and never use it, you most likely need to study it.

What's the Goal of Asymptotic Complexity?

• Performance-centric analysis (sort of...)
• Why not measure clock time?
• Machine variation
• Input size must be a factor
• Concisely express how an algorithm will scale without regard to a particular physical machine
• Simple means for comparing algorithms
• "[Asymptotic Complexity] significantly simplifies calculations because it allows us to be sloppy – but in a satisfactorily controlled way" - Donald Knuth
• Theoretical framework for advanced algorithm analysis

What are we measuring?

• Approximate number of Machine Operations
• It's not practical to measure the actual number of machine operations
• Programming language variations
• Machine variations
• Parallelism
• Radom Access Machine (RAM) Model
• Not to be confused with Random Access Memory

Random Access Machine (RAM)

• Theoretical computing machine
• Algorithm Comparisons independent of a physical machine
• Runs pseudo code
• Assumptions
• Sequential execution of instructions
• Fixed word sizes
• Standard integer and floating-point numbers
• ​Operations​ (loosely defined)
• Arithmetic (add, subtract, multiply, divide, remainder, floor, ceiling)
• Data Movement (load, store, copy)
• Control (conditional branch, unconditional branch, subroutine call, return)

RAM Operations

1 data operation

1 control operation X 3 iterations

1 arithmetic and 1 data operation X 3 iterations

1 control operation

Total = 12 operations

If we are being pedantic, this is how to count operations...

1 control operation

j ← 0

for i∈{1,2,3} do
j←i × 2
end for

return j

RAM Operations

• An operation is roughly equivalent to an executed line of pseudo code
• It's acceptable to take reasonable liberties

$$d \gets \sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2}$$

1 operation

1 operation X 3 iterations

1 operation X 3 iterations

1 operation

ignore

1 operation

How do machine operations relate to scale?

j ← 0

for i∈{1,2,3} do
j←i × 2
end for

return j

Total = 8 operations

Machine Operations as a $$f(x)$$

• When considering scale, growth rate in relation to input size is what's important
• Mathematical function describing the relationship between operations and input size
• Input Size
• Could be anything: number of bits, a scalar value,  etc...
• Typically number of items:  elements in array, nodes in graph, etc...
• Conventions
• Input size = $$n$$
• Growth rate function = $$T(n)$$

$$T(n)$$ Example

function square(A)
for i ← 0 to |A| −1 do
A[i] = A[i] x A[i]
endfor
end function

1 operation per item

1 operation per item

Total = 2 operations per item

$$T(n) = 2n$$

More Complex $$T(n)$$

• Not all algorithms can be expressed by a single $$T(n)$$
• Upper Bound
• Lower Bound
function findDuplicates(A)
D←∅
for i←0 to |A| −1 do
for j←0 to |A| −1 do
if i≠j and A[i] = A[j] then
D←D ∪ {A[i]}
end if
end for
end for
return D
end function

1 operation

$$n$$ operations

$$n^2$$ operations ($$n$$ per item)

$$n^2$$ operations ($$n$$ per item)

Between 0 and $$n^2$$ operations

1 operation

T(n)=1+n+n^2+n^2+n^2+1 \\ T(n)=3n^2+n+2

Upper Bound

T(n)=1+n+n^2+n^2+0+1 \\ T(n)=2n^2+n+2

Lower Bound

Comparing $$T(n)$$

Under reasonable assumptions, square scales much better than findDuplicates

Asymptotic Complexity

Review: The goal is to concisely express how an algorithm will scale​ (growth rate)

Loosely consider the asymptotic complexity to be the "shape" of $$T(n)$$

Deriving Asymptotic Complexity

Remove all constants and non-leading terms

T(n)=3n^2+n+2

Constants

T(n)=n^2

The "shape" does not change

6 Basic Complexity Classes

Name Growth Rate Example
Constant Hash Table (Look up)
Logarithmic Binary Tree (Search)
Exponential Traveling Salesman (Dynamic Programming)
Factorial Traveling Salesman (Brute Force)
T(n) = 1
T(n) = \log_2{n}
T(n) = n
T(n) = n^2
T(n) = 2^n
T(n) = n!

What does Asymptotic Complexity Really Say?

• Larger asymptotic complexity values equate to longer run times given that the input size is sufficiently large.
• Consider two theoretical algorithms
• $$\sigma - T(n) = n^2$$ - asymptotically quadratic
• $$\alpha - T(n) = 50n$$ - asymptotically linear

Question: Does the linear algorithm always execute fewer machine operations?

What does Asymptotic Complexity Really Say?

Answer: No, for values of $$n$$ lower than 50, $$\sigma$$ has fewer operations that $$\alpha$$

Asymptotic Complexity

One Last Time:

The goal is to concisely express how an algorithm will scale​ (growth rate)

Asymptotic Notation

• Known as Landau Symbols outside of CS
• Express asymptotic concepts concisely
• $$\Theta$$ - Big Theta
• $$O$$ - Big O
• $$\Omega$$ - Big Omega

$$\Theta$$ - Big Theta

• Asymptotic Tight Bound
• Asymptotic complexity is the same regardless of input
• It’s not possible to specify $$\Theta$$ for all algorithms
• Recall findDuplicates
• Lower Bound $$T(n) = 3n^2+n+2$$
• Upper Bound $$T(n) = 2n^2+n+2$$
• ​Asymptotic complexity for upper and lower is $$T(n) = n^2$$
• findDuplicates = $$\Theta(n^2)$$

$$O$$ - Big O

• Asymptotic Upper Bound
• Worst Case Scenario
• O stands for Order of or more precisely the German word Ordnung
• $$\Theta$$ is a stronger statement than $$O$$
• if findDuplicates = $$\Theta(n^2)$$ then findDuplicates = $$O(n^2)$$
• Big O makes no indication of the best case scenario

$$\Omega$$ - Big Omega

• Asymptotic Lower Bound
• Best Case Scenario
• $$\Theta$$ is a stronger statement than $$\Omega$$
• if findDuplicates = $$\Theta(n^2)$$ then findDuplicates = $$\Omega(n^2)$$
• Big $$\Omega$$ makes no indication of the worst case scenario

Example Analysis

function bubbleSort(A)
sorted ← false
index ← |A| − 2
while sorted is false do
sorted←true
for i ← to index do
if A[i] > A[i + 1] then
swap A[i] and A[i + 1]
sorted ← false
end if
end for
index ← index - 1
end while
end function

Lower $$2n + 5$$               $$\Omega(n)$$

Upper $$4n^2 + 3n + 3$$    $$O(n^2)$$

* Continues to loop until the for loop on line 6 makes a complete revolution without triggering the if statement on line 7. The extra operation is the exit condition (sorted is true)
** The number of iterations is reduced by 1 for each iteration of the outer loop by virtue of line 12. So it’s technically $$\sum_{i=1}^{n} i$$ ; however, this is a bit of minutia that’s best left rounded up to $$n^2$$.

Thank You!

• If you liked this content, let me know and I'll make more
• If you hated this content, let me know and I'll do better