COMP3010: Algorithm Theory and Design

Daniel Sutantyo,  Department of Computing, Macquarie University

1.1 Problems and Algorithms

Topics

1.1 - Problems and Algorithms

  • Problems
  • Algorithms
  • Why study algorithms

Algorithm

1.1 - Problems and Algorithms

  • Definition: 
    • a procedure to accomplish a specific task
    • Wikipedia: finite sequence of well-defined, computer-implementable instructions, typically to solve a class of problems or to perform a computation
    • CLRS: well-defined computational procedure that takes some value as input and produces some value as output
  • Exact definitions are not that important, it's more important that you understand the concepts

Algorithm

1.1 - Problems and Algorithms

  • Algorithm is used to solve problems (or specifically, computational problem)
  • What is a computational problem?
    • Given a set of inputs and a set of outputs, a computational problem specifies the desired relationship between the two sets
  • Example:
    • Given an array \(A\), find the largest value in \(A\)
    • Given an integer \(n\), find the nontrivial prime factors of \(n\)
    • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest way to travel from A to B

Computational Problem

1.1 - Problems and Algorithms

  • Why even talk about what a problem is? We are in 3rd year!
    • a different point of view (maybe?)
  • Problem: Given an array \(A\), find the largest value in \(A\)

[ 5, 1, 7, 2, 6, 9, 11 ]

11

Computational Problem

1.1 - Problems and Algorithms

[ 11, 9, 7, 6, 5, 2, 1 ]

[ 1, 2, 5, 6, 7, 9, 11 ]

[ ]

[ 1, 1, 1, 1, 1, 1 ]

[ 5.6, 4.1, 2.7, 2.1, 16.0, 9.1]

[ 5, 1, 7, 2, 6, 9, 11 ]

11

["chicken","beef","lamb"]

[null,null,null,null]

Computational Problem

1.1 - Problems and Algorithms

  • A computational problem should have a well-defined input, including any constraints, plus a well-defined output and how it relates to the input
    • bad: Given an array \(A\), find the largest value in \(A\)
    • better:
      • ​Given an array of integers \(A\), find the largest value in \(A\)
      • input: an array of integers
      • output: the largest integer in \(A\)
  • The first step in solving a problem is to understand the problem, and this requires you to understand what the inputs are and how the output is related
  • Don't start solving a problem until you understand it

Computational Problem

1.1 - Problems and Algorithms

  • Given a positive integer \(n\), find the nontrivial prime factors of \(n\)
    • input: a positive integer \(n\)
    • output: the nontrivial prime factors of \(n\)

Computational Problem

1.1 - Problems and Algorithms

  • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest way of travelling between city A and city B
    • input:
      • a map with \(n\) cities
      • the cost of travelling between any two cities
      • the cities A and B
    • output:
      • the cheapest way of travelling between city A and city B

Computational Problem

1.1 - Problems and Algorithms

  • So when given a problem, you should think about
    • the set of inputs (what kind of inputs you are getting)
    • the set of outputs (what kind of output you need)
    • how do you link them

Not Computational Problem

1.1 - Problems and Algorithms

  • These are not computational problems:
    • Find the best place in Australia
      • input and output: ??? what is a place? a city? a beach? a bar?
      • what does 'best' mean? 
    • Given a set of triangles, sort them
      • input: set of triangles
      • output: the same set of triangles in different order
      • problem: how do you sort them?

Computational Problem

1.1 - Problems and Algorithms

2

3

...

15

16

...

[2]

[3]

...

[3,5]

...

input

output

  • Given a positive integer \(n\), find the nontrivial prime factors of \(n\)
    • input: a positive integer \(n\)
    • output: the nontrivial prime factors of \(n\)

Computational Problem

1.1 - Problems and Algorithms

[ 5, 1, 7, 2, 6, 9, 11 ]

11

[ 1, 2, 5, 6, 7, 9, 11 ]

[ 11, 9, 7, 6, 5, 2, 1 ]

[ 1, 1, 1, 1, 1, 1, 11 ]

. . .

[ 1 ]

1

. . .

[ 1, 1 ]

[ 1, 1, 1]

  • ​Given an array of integers \(A\), find the largest value in \(A\)
  • input: an array of integers
  • output: the largest integer in \(A\)

[ ]

???

Computational Problem

1.1 - Problems and Algorithms

  • The hope is that you can now see a problem as a set of inputs and outputs and the desired relationship between them
  • A particular input to a problem is referred to as an instance of a problem

Wikipedia: a computational problem can be viewed as an infinite collection of instances together with a possibly empty, set of solutions for every instance

https://en.wikipedia.org/wiki/Computational_problem

  • An algorithm gives you that relationship, i.e it maps the inputs to the correct output
  • An algorithm is correct if for every instance it halts with the correct output

Computational Problem

1.1 - Problems and Algorithms

  • I like thinking about the input first because it gives me some idea about the complexity of the problem (how many possible inputs are there)
  • It can also help me optimise my solution
    • can I ignore certain type of inputs?
    • will I get small inputs or large inputs? 
  • Obviously the type of input also tells me what data structure I can use
  • See Chapter 10 of Skiena (it's only 3-4 pages)

Types of Computational Problem

1.1 - Problems and Algorithms

  • Decision problem: the output to every instance of input is either a yes or a no
    • Given a positive integer \(n\), is it a prime number?
    • Given a map with \(n\) cities and the cost of travelling between each city, can you travel from city A to city B for less than \(x\)?
  • Search problem: find one or more output that satisfies the required relation to the input
    • Given a positive integer \(n\), find its nontrivial prime factors
    • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find all possible paths between city A and city B

Types of Computational Problem

1.1 - Problems and Algorithms

  • Counting problem: find the number of solutions to a given search problem
    • Given a positive integer \(n\), how many nontrivial prime factors does it have?
    • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the number of possible paths between City A and City B
  • Optimisation problem: find the best possible solution among the set of all possible solutions to a search problem
    • Given a map with \(n\) cities and the cost of travelling between each pair of cities, find the cheapest cost of travelling between City A and City B

Why Study Algorithms?

1.1 - Problems and Algorithms

  • Obvious solutions are not always tractable
    • tractable: can be solved in polynomial time, i.e. 'easy'
  • Tractable solutions can be improved
  • Recognise hard problems
    • prevents us from wasting time looking for an efficient solution

Obvious solutions are not always tractable

1.1 - Problems and Algorithms

  • Brute-force solution:
    • Compute every single possible paths
  • What is the number of possible paths from A to B?

no intermediate cities

1 intermediate city

\((n-2)\) choices

\(\vdots\)

A

A

B

B

\(\vdots\)

A

B

2 intermediate cities

\((n-2)(n-3)\) choices

1 choice

Obvious solutions are not always tractable

1.1 - Problems and Algorithms

$$ 1 + (n-2) + (n-3)(n-2) + (n-4)(n-3)(n-2) + \cdots + \binom{n-2}{k}k! $$

  • Total number of paths:

(via 1 city)

(via 2 cities)

(via 3 cities)

(via \(k\) cities)

  • For large \(n\), it is impossible to check all paths, e.g. \(50!\approx 3.04 \times 10^{64}\)

50! = 30,414,093,201,713,378,043,612,608,166,064,768,844,377,641,568,960,512,000,000,000,000

Tractable solutions can be improved

1.1 - Problems and Algorithms

  • Given a list of \(n\) integers, find the largest and smallest integers among them
  • What is the minimum number of comparisons that you need to do? 
  • \((n-1)\) comparisons, why?
  • How many comparisons do you need to find the largest and the smallest integers?

Tractable solutions can be improved

1.1 - Problems and Algorithms

  • Given a list of \(n\) integers, find the largest and smallest integers among them
  • Our first solution is likely to be \((2n-2)\) comparisons, i.e. just go through each numbers and record the maximum and minimum so far

13  

63

44

22

93

21

72

67

  • We need \((2 * 8 - 2)\) = 14 comparisons in the above example
  • Our first solution is likely to be \((2n-2)\) comparisons, i.e. just go through each numbers and record the maximum and minimum so far

Tractable solutions can be improved

1.1 - Problems and Algorithms

  • Given a list of \(n\) integers, find the largest and smallest integers among them
  • Another method: (assume \(n\) is even) pair up the numbers by doing \(n/2\) comparisons

13  

63

44

22

93

21

72

67

44

13

22

72

21

63

93

67

4 comparisons

3 comparisons

3 comparisons

  • In total, the number of comparisons you need is roughly: $$\frac{n}{2} + \left(\frac{n}{2}-1\right) + \left(\frac{n}{2}-1\right) = \left(\frac{3n}{2} - 2\right)$$

Recognise hard problems

1.1 - Problems and Algorithms

  • In later weeks you are going to learn about intractable problems, that is, problems that have no known polynomial (i.e efficient) solution
  • If you have to solve a problem, and you can see that it is intractable, then you should not waste time trying to find an efficient solution for it!

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

Recognise hard problems

1.1 - Problems and Algorithms

Cartoon by Stefan Szeider, available at https://www.ac.tuwien.ac.at/people/szeider/cartoon/

How do you study algorithms?

1.1 - Problems and Algorithms

  • We start by going through the theoretical foundations
    • so far we have defined a few terms which we will need for later parts (problems, intractability)
    • later we will learn more tools (asymptotic notations, loop invariants)
  • We go through the most common problem solving strategies (divide-and-conquer, dynamic programming)
  • You learn by examples, and most importantly, by writing your own solutions

How do you study algorithms?

1.1 - Problems and Algorithms

  • What makes a good chef?
    • do you copy (or adapt) other people recipes?
    • do you memorise recipes?
    • do you only study recipes?
      • do you study the ingredients?
      • do you study your utensils?
    • can you be a good chef if you only read and memorise recipes?
    • can you be a good chef if you don't cook?