COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

1.1 Problems and Algorithms

Topics

1.1 - Problems and Algorithms

• Problems
• Algorithms
• Why study algorithms

Algorithm

1.1 - Problems and Algorithms

• Definition: 
  • a procedure to accomplish a specific task
  • Wikipedia: finite sequence of well-defined, computer-implementable instructions, typically to solve a class of problems or to perform a computation
  • CLRS: well-defined computational procedure that takes some value as input and produces some value as output
• Exact definitions are not that important, it's more important that you understand the concepts

Algorithm

1.1 - Problems and Algorithms

• Algorithm is used to solve problems (or specifically, computational problem)
• What is a computational problem?
  • Given a set of inputs and a set of outputs, a computational problem specifies the desired relationship between the two sets
• Example:
  • Given an array \(A\), find the largest value in \(A\)
  • Given an integer \(n\), find the nontrivial prime factors of \(n\)
  • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest way to travel from A to B

Computational Problem

1.1 - Problems and Algorithms

• Why even talk about what a problem is? We are in 3rd year!
  • a different point of view (maybe?)
• Problem: Given an array \(A\), find the largest value in \(A\)

[ 5, 1, 7, 2, 6, 9, 11 ]

11

Computational Problem

1.1 - Problems and Algorithms

[ 5, 1, 7, 2, 6, 9, 11 ]

11

Computational Problem

1.1 - Problems and Algorithms

• A computational problem should have a well-defined input, including any constraints, plus a well-defined output and how it relates to the input
  • bad: Given an array \(A\), find the largest value in \(A\)
  • better:
    • ​Given an array of integers \(A\), find the largest value in \(A\)
    • input: an array of integers
    • output: the largest integer in \(A\)
• The first step in solving a problem is to understand the problem, and this requires you to understand what the inputs are and how the output is related
• Don't start solving a problem until you understand it

Computational Problem

1.1 - Problems and Algorithms

• Given a positive integer \(n\), find the nontrivial prime factors of \(n\)
  • input: a positive integer \(n\)
  • output: the nontrivial prime factors of \(n\)

Computational Problem

1.1 - Problems and Algorithms

• Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest way of travelling between city A and city B
  • input:
    • a map with \(n\) cities
    • the cost of travelling between any two cities
    • the cities A and B
  • output:
    • the cheapest way of travelling between city A and city B

Computational Problem

1.1 - Problems and Algorithms

• So when given a problem, you should think about
  • the set of inputs (what kind of inputs you are getting)
  • the set of outputs (what kind of output you need)
  • how do you link them

Not Computational Problem

1.1 - Problems and Algorithms

• These are not computational problems:
  • Find the best place in Australia
    • input and output: ??? what is a place? a city? a beach? a bar?
    • what does 'best' mean? 
  • Given a set of triangles, sort them
    • input: set of triangles
    • output: the same set of triangles in different order
    • problem: how do you sort them?

Computational Problem

1.1 - Problems and Algorithms

2

3

...

15

16

...

[2]

[3]

...

[3,5]

...

input

output

• Given a positive integer \(n\), find the nontrivial prime factors of \(n\)
  • input: a positive integer \(n\)
  • output: the nontrivial prime factors of \(n\)

Computational Problem

1.1 - Problems and Algorithms

[ 5, 1, 7, 2, 6, 9, 11 ]

11

[ 1, 2, 5, 6, 7, 9, 11 ]

[ 11, 9, 7, 6, 5, 2, 1 ]

[ 1, 1, 1, 1, 1, 1, 11 ]

. . .

[ 1 ]

1

. . .

[ 1, 1 ]

[ 1, 1, 1]

• ​Given an array of integers \(A\), find the largest value in \(A\)
• input: an array of integers
• output: the largest integer in \(A\)

Computational Problem

1.1 - Problems and Algorithms

• The hope is that you can now see a problem as a set of inputs and outputs and the desired relationship between them
• A particular input to a problem is referred to as an instance of a problem

Wikipedia: a computational problem can be viewed as an infinite collection of instances together with a possibly empty, set of solutions for every instance

https://en.wikipedia.org/wiki/Computational_problem

• An algorithm gives you that relationship, i.e it maps the inputs to the correct output
• An algorithm is correct if for every instance it halts with the correct output

Computational Problem

1.1 - Problems and Algorithms

• I like thinking about the input first because it gives me some idea about the complexity of the problem (how many possible inputs are there)
• It can also help me optimise my solution
  • can I ignore certain type of inputs?
  • will I get small inputs or large inputs? 
• Obviously the type of input also tells me what data structure I can use
• See Chapter 10 of Skiena (it's only 3-4 pages)

Types of Computational Problem

1.1 - Problems and Algorithms

• Decision problem: the output to every instance of input is either a yes or a no
  • Given a positive integer \(n\), is it a prime number?
  • Given a map with \(n\) cities and the cost of travelling between each city, can you travel from city A to city B for less than \(x\)?
• Search problem: find one or more output that satisfies the required relation to the input
  • Given a positive integer \(n\), find its nontrivial prime factors
  • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find all possible paths between city A and city B

Types of Computational Problem

1.1 - Problems and Algorithms

• Counting problem: find the number of solutions to a given search problem
  • Given a positive integer \(n\), how many nontrivial prime factors does it have?
  • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the number of possible paths between City A and City B
• Optimisation problem: find the best possible solution among the set of all possible solutions to a search problem​
  • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest cost of travelling between City A and City B

Why Study Algorithms?

1.1 - Problems and Algorithms

• Obvious solutions are not always tractable
  • tractable: can be solved in polynomial time, i.e. 'easy'
• Tractable solutions can be improved
• Recognise hard problems
  • prevents us from wasting time looking for an efficient solution

Obvious solutions are not always tractable

1.1 - Problems and Algorithms

• Brute-force solution:
  • Compute every single possible paths
• What is the number of possible paths from A to B?

no intermediate cities

1 intermediate city

\((n-2)\) choices

\(\vdots\)

A

A

B

B

\(\vdots\)

A

B

2 intermediate cities

\((n-2)(n-3)\) choices

1 choice

Obvious solutions are not always tractable

1.1 - Problems and Algorithms

$$ 1 + (n-2) + (n-3)(n-2) + (n-4)(n-3)(n-2) + \cdots + \binom{n-2}{k}k! $$

• Total number of paths:

(via 1 city)

(via 2 cities)

(via 3 cities)

(via \(k\) cities)

• For large \(n\), it is impossible to check all paths, e.g. \(50!\approx 3.04 \times 10^{64}\)

Tractable solutions can be improved

1.1 - Problems and Algorithms

• Given a list of \(n\) integers, find the largest and smallest integers among them
• What is the minimum number of comparisons that you need to do? 

• \((n-1)\) comparisons, why?

• How many comparisons do you need to find the largest and the smallest integers?

Tractable solutions can be improved

1.1 - Problems and Algorithms

• Given a list of \(n\) integers, find the largest and smallest integers among them

• Our first solution is likely to be \((2n-2)\) comparisons, i.e. just go through each numbers and record the maximum and minimum so far

13  

63

44

22

93

21

72

67

• We need \((2 * 8 - 2)\) = 14 comparisons in the above example

• Our first solution is likely to be \((2n-2)\) comparisons, i.e. just go through each numbers and record the maximum and minimum so far

Tractable solutions can be improved

1.1 - Problems and Algorithms

• Given a list of \(n\) integers, find the largest and smallest integers among them

• Another method: (assume \(n\) is even) pair up the numbers by doing \(n/2\) comparisons

13  

63

44

22

93

21

72

67

44

13

22

72

21

63

93

67

4 comparisons

3 comparisons

3 comparisons

• In total, the number of comparisons you need is roughly: $$\frac{n}{2} + \left(\frac{n}{2}-1\right) + \left(\frac{n}{2}-1\right) = \left(\frac{3n}{2} - 2\right)$$

Recognise hard problems

1.1 - Problems and Algorithms

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

How do you study algorithms?

1.1 - Problems and Algorithms

• We start by going through the theoretical foundations
  • so far we have defined a few terms which we will need for later parts (problems, intractability)
  • later we will learn more tools (asymptotic notations, loop invariants)
• We go through the most common problem solving strategies (divide-and-conquer, dynamic programming)
• You learn by examples, and most importantly, by writing your own solutions

How do you study algorithms?

1.1 - Problems and Algorithms

• What makes a good chef?
  • do you copy (or adapt) other people recipes?
  • do you memorise recipes?
  • do you only study recipes?
    • do you study the ingredients?
    • do you study your utensils?
  • can you be a good chef if you only read and memorise recipes?
  • can you be a good chef if you don't cook?