COMP333

Algorithm Theory and Design

Daniel Sutantyo

Department of Computing

Macquarie University

Lecture slides adapted from lectures given by Frank Cassez, Mark Dras, and Bernard Mans

Summary

Algorithm complexity (running time, recursion tree)
Algorithm correctness (induction, loop invariants)
Problem solving methods:
- exhaustive search
- dynamic programming
- greedy method
- divide-and-conquer
- algorithms involving strings
- probabilistic methods
- algorithms involving graphs

Graph Algorithms

Maximum Flow Problem
Shortest Path Algorithms

Maximum Flow Problem

overview

source

sink

suppose that you want to move materials from one location to another location
each vertex represents a location
each edge represents the connections between two locations

Maximum Flow Problem

overview

the source vertex produces material and the sink vertex consumes them
the weight of each edge signifies the amount of material that can flow through it

source

sink

in the maximum-flow problem, we want to compute the greatest rate at which we can move materials from the source vertex to the sink vertex
real life applications:
- goods distribution, water pipes, road networks
- airline scheduling

Maximum Flow Problem

overview

source

sink

12/16

12/12

4/13

4/14

4/4

12/20

0/7

0/9

0/4

source

sink

Maximum Flow Problem

example of a flow

Maximum Flow Problem

more examples

11/16

12/12

8/13

11/14

4/4

15/20

7/7

4/9

1/4

source

sink

12/16

12/12

4/13

4/14

4/4

12/20

0/7

0/9

0/4

source

sink

Maximum Flow Problem

definitions

A flow network $G = \langle V,E \rangle$ is a directed graph where:
- each edge $(u,v) \in E$ has a non-negative capacity $c(u,v)$
- if $(u,v) \in E$, then $(v,u)\not\in E$, i.e. if you have an edge going in one direction, then you cannot have another edge in the reverse direction
- there is a source vertex $s$ and a sink vertex $t$
Assumptions:
- the graph is connected and each node lies in the path between the source and the sink vertices
- $|E| \ge |V|-1$

(because each vertex has degree at least 1)

Maximum Flow Problem

definitions

a flow in $G$ is a function$f(u,v) : V \times V \rightarrow \mathbb R$ for $u,v \in V$ that satisfies the following two properties:
- capacity constraint: for all $u,v \in V$,
  - $0\le f(u,v) \le c(u,v)$, i.e. the flow through an edge cannot exceed the capacity of that edge
- flow conservation: for all $u \in V \setminus \{s,t\}$
  - i.e. except for the source and sink vertices, all inflows to a vertex must equal the outflows from that vertex

\[\sum_{v\in V}f(v,u) = \sum_{v\in V} f(u,v)\]

Maximum Flow Problem

definitions

The value of a flow is defined by
- i.e. it is the total flow out of the source
- in the maximum-flow problem, we want to find a flow of maximum value
For reasons we will see later, it is convenient to instead have the definition

\[|f| = \sum_{v \in V} f(s,v) \]

\[|f| = \sum_{v \in V} f(s,v) - \sum_{v \in V} f(v,s)\]

Maximum Flow Problem

definitions

11/16

12/12

8/13

11/14

4/4

15/20

7/7

4/9

1/4

source

sink

11/16

11/12

8/13

11/14

4/4

15/20

7/7

3/9

0/4

source

sink

this is a flow

this is a different flow

Maximum Flow Problem

definitions

11/16

11/12

8/13

11/14

4/4

15/20

7/7

3/9

0/4

source

sink

$f(s,a) = 11$

$f(s,c) = 8$

$f(c,a) = 0$

$f(a,b) = 11$

$f(b,c) = 3$

$f(c,d) = 11$

$f(d,b) = 7$

$f(b,t) = 15$

$f(d,t) = 4$

the value of this flow is $19$ (i.e $|f| = 19$ )

Maximum Flow Problem

definitions

11/16

12/12

4/13

7/14

0/4

15/20

7/7

4/9

1/4

source

sink

1/16

1/12

1/13

1/14

1/4

1/20

0/7

0/9

0/4

source

sink

this is also a flow

this is yet another flow

Maximum Flow Problem

definitions

1/16

1/12

1/13

1/14

1/4

1/20

0/7

0/9

0/4

source

sink

$f(s,a) = 1$

$f(s,c) = 1$

$f(c,a) = 0$

$f(a,b) = 1$

$f(b,c) = 0$

$f(c,d) = 1$

$f(d,b) = 0$

$f(b,t) = 1$

$f(d,t) = 1$

the value of this flow is $19$ (i.e $|f| = 2$ )

Maximum Flow Problem

definitions

Summary of the definitions:
- a flow in $G$ is a function $f(u,v) : V \times V \rightarrow \mathbb R$, $u,v \in V$
  - satisfying capacity constraint and flow conservation constraint
- a flow network has many flows (if it is nontrivial network), and each flow has its own value $|f|$
- in the Maximum Flow Problem, we want to find the flow with the maximum value

Ford-Fulkerson Method

Method:
- start by setting $f(u,v)$ to $0$ for all $u,v \in V$
- while there is an augmenting path $p$ in the residual network $G_f$:
  - augment flow $f$ along $p$
We begin by defining what residual networks and augmenting paths are

Ford-Fulkerson Method

residual networks

Residual network $G_f$:
- intuitively it is the edges of $G$ that still have some capacity left (i.e. the ones that can take additional flow).
- (not so intuitively) $G_f$ may contain an edge that is not in $G$:
  - if you want to reduce the flow from $u$ to $v$, you place an edge from $v$ to $u$ with capacity equalling to the current flow from $u$ to $v$ (we will see an example in the next slide)
  - remember, in $G$, you are not allowed to have a reverse edge (i.e. if you have $(u,v)$, then you cannot also have $(v,u)$

Ford-Fulkerson Method

residual networks

11/16

12/12

8/13

11/14

4/4

15/20

7/7

4/9

1/4

$G$:

$G_f$:

Ford-Fulkerson Method

formal definitions for residual networks

Define the residual capacity $c_f(u,v)$ by

\[c_f(u,v)=\begin{cases} c(u,v)-f(u,v)& \text{if $(u,v)\in E$,} \\ f(v,u) & \text{if $(v,u)\in E$,}\\ 0 & \text{otherwise} \end{cases} \]

Define the residual network of $G$ induced by $f$ as $G_f = (V,E_f)$ where
- $E_f = \{(u,v) \in V \times V : c_f(u,v) > 0\}$
- notice that a residual network is basically a flow network that is allowed to have both $(u,v)$ and $(v,u)$

Ford-Fulkerson Method

augmenting path

an augmenting path $p$ is a simple path from $s$ to $t$ in the residual network $G_f$

$G_f$:

if you treat $G_f$ as a flow diagram, then you can increase the flow through each edge by 4 without violating any constraint

Ford-Fulkerson Method

augmenting path

so you can treat an augmenting path as a flow in $G_f$

4/4

4/5

Ford-Fulkerson Method

augmenting path

formally, we can define a flow $f_p$ in $G_f$ with the following property:

4/4

4/5

\[f_p(u,v) = \begin{cases}\min\{c_f(u,v)\} & \text{if $(u,v)$ is on $p$,}\\ 0 & \text{otherwise}\end{cases}\]

in other words, $f_p$ is a flow where for each edge $f_p(u,v)$ is the maximum residual capacity of the edges in the path

Ford-Fulkerson Method

augmenting path

If $f$ is a flow in $G$ and $f_p$ is a flow in $G_f$, then define the function $(f \uparrow f_p) : V \times V \rightarrow \mathbb R$ as

\[(f \uparrow f_p)(u,v) = \begin{cases} f(u,v) + f_p(u,v) - f_p(v,u) &\text{if $(u,v) \in E$,} \\ 0 & \text{otherwise.}\end{cases}\]

we call the function $(f \uparrow f_p)$ as the augmentation of flow $f$ by $f_p$
basically we increase the flow on $(u,v)$ by $f_p(u,v)$, but we decrease it by $f_p(v,u)$ (we call this a cancellation)
the function $(f\uparrow f_p)(u,v)$ is a flow in $G$ with value

$|f\uparrow f_p| = |f| + |f_p| > |f|$

(Lemma 26.1, Corollary 26.3 of CLRS, page 717, 720)

Ford-Fulkerson Method

Method:
- start by setting $f(u,v)$ to $0$ for all $u,v \in V$
- while there is an augmenting path $p$ in the residual network $G_f$:
  - augment flow $f$ along $p$

Ford-Fulkerson Method

the algorithm

Ford-Fulkerson Algorithm for a flow network $G = \langle V,E\rangle$:
- set $f(u,v) = 0$ for all $(u,v)$ in $E$
- create the residual network $G_f$
- while there exists a simple path $p$ from source to sink in $G_f$:
  - set $c_f(p) = \min\{c_f(u,v) : (u,v) \in p\}$
  - for each edge $(u,v)$ in $p$:
    - if $(u,v) \in E$:
      - $(u,v) = (u,v) + c_f(p)$
    - else
      - $(v,u) = (v,u) - c_f(p)$

Ford-Fulkerson Method

example run

$G:$

$G_f:$

Ford-Fulkerson Method

example run

12/16

12/12

0/13

0/14

0/4

12/20

0/7

0/9

0/4

$G:$

$G_f:$

Ford-Fulkerson Method

example run

12/16

12/12

4/13

4/14

4/4

12/20

0/7

0/9

0/4

$G:$

$G_f:$

Ford-Fulkerson Method

example run

12/16

12/12

11/13

11/14

4/4

19/20

7/7

0/9

0/4

$G:$

$G_f:$

Ford-Fulkerson Method

another example

if you remember in the algorithm, you may have to reduce your current flow:
- for each edge $(u,v)$ in $p$:
  - if $(u,v) \in E$:
    - $(u,v) = (u,v) + c_f(p)$
  - else
    - $(v,u) = (v,u) - c_f(p)$
it did not happen in the previous example, so here is another one that has this behaviour

Ford-Fulkerson Method

another example

4/16

8/12

4/13

4/14

4/4

4/20

0/7

4/9

4/4

$G$:

$G_f$:

Ford-Fulkerson Method

another example

4/16

8/12

4/13

4/14

4/4

4/20

0/7

4/9

4/4

$G$:

8/16

8/12

4/13

4/14

4/4

8/20

0/7

0/9

0/4

$G$:

Ford-Fulkerson Method

complexity analysis

The main operation in the algorithm is finding the augmenting path:
- while there is an augmenting path $p$ in the residual network $G_f$:
Suppose that $f^*$ is the maximum flow.
Each time we augment, we increase our current flow by at least 1, so if the maximum flow value is $|f^*|$, then the loop runs $|f^*|$ times
Each time we do an update, we have to do at most $O(|E|)$ operations, hence the complexity is $O(|E||f^*|)$

Ford-Fulkerson Method

complexity analysis

The Edmond-Karps algorithm uses BFS to find the augmenting path $p$, and it can be proven that this variant runs in $O(VE^2)$ (the proof is in CLRS pg. 727-729)

Ford-Fulkerson Method

correctness

The proof for the correctness of the algorithm is detailed in CLRS, Theorem 26.6, pg 723.
It is called the Max-flow min-cut theorem

Shortest Path Problem

definitions

Let $G = \langle V,E \rangle$ be a weighted, directed graph where:
- $V$ is the set of vertices in $G$, and
- $E$ is the set of edges in $G$
Let $w(u,v)$ be the weight of the edge connecting vertex $u$ and vertex $v$
Let $\{x_1 = u, x_2, x_3, \dots, x_{k-1}, x_k = v\}$ be the shortest path between vertex $u$ and vertex $v$
For now we will only find the weight of this path, not the actual path itself (i.e. the sum of the weights of all the edges in the path)

Shortest Path Problem

decomposition into subproblems

ShortestPath($u$,$v$)

ShortestPath($y_1$,$v$)

ShortestPath($y_2$,$v$)

ShortestPath($y_\ell$,$v$)

$\dots$

(where the vertices $y_i$, $1 \le i \le \ell$, are the other vertices in the graph)

Shortest Path Problem

decomposition into subproblems

ShortestPath($u$,$v$)

$w(u,y_1)$ + ShortestPath($y_1$,$v$)

$\dots$

$w(u,y_2)$ + ShortestPath($y_2$,$v$)

$w(u,y_\ell)$ + ShortestPath($y\ell$,$v$)

\[\text{ShortestPath}(u,v) = \begin{cases} \displaystyle \min_{1 \le i \le \ell} \left(w(u,y_i) + \text{ShortestPath}(y_i,v)\right) & \text{if $u \ne v$} \\ 0 &\text{if $u = v$} \end{cases}\]

Shortest Path Problem

optimal substructure

Let $\{u, x_2, x_3, \dots, x_{k-1}, v\}$ be the shortest path between vertex $u$ and vertex $v$
We have $\text{ShortestPath}(u,v) = w(u,x_2) + \text{ShortestPath}(x_2,v)$

ShortestPath($u$,$v$)

$w(u,y_1)$ + ShortestPath($y_1$,$v$)

$\dots$

$w(u,y_2)$ + ShortestPath($y_2$,$v$)

$w(u,y_\ell)$ + ShortestPath($y\ell$,$v$)

Shortest Path Problem

optimal substructure

Suppose that $\text{ShortestPath}(u,v)$ is optimal, i.e. it is the weight of the shortest path between $u$ and $v$
We have $\text{ShortestPath}(u,v) = w(u,x_2) + \text{ShortestPath}(x_2,v)$
- let $\text{ShortestPath}(x_2,v) = W$
If $\text{ShortestPath}(x_2,v)$ is not optimal, then there is another path between $x_2$ and $v$ that has a lower weight, say $W^* < W$
But this means that the shortest path between $u$ and $v$ has weight $w(u,x_2) + W^* < w(u,x_2) + W$, which contradicts our initial assumption
Therefore $\text{ShortestPath}(x_2,v)$ must also be optimal
See Lemma 24.1 (pg 645) of CLRS if you need a more detailed explanation

Shortest Path Problem

negative values

can the graph contains edges with negative weights?

-5

-4

Shortest Path Problem

negative values

can the graph contains edges with negative weights?

-5

-4

if the graph contains any negative-weight cycles, then the shortest-path is not well defined because we can always find a path that is shorter

Shortest Path Problem

cycles

can the shortest path contains cycles?

-5

-4

Shortest Path Problem

cycles

can the shortest path contains cycles?

-5

-4

suppose the shortest path $\{u,x_1,x_2,\dots,x_{k-1},v\}$ contains a cycle $\{x_i,x_{i+1},\dots,x_j\}$: can we find a shorter path?

Shortest Path Problem

cycles

so let us assume:
- the graph does not contain negative cycles
- the shortest path contains no cycles
- assume no self-cycles (or alternatively, assume the shortest distance from a vertex to itself is zero)

Shortest Path Problem

relaxation

in finding the shortest path, we start with an upper bound for the distances between one vertex to another
- e.g. with Dijkstra's algorithm, you start with the distance between the starting vertex and all other vertices using just one edge (which may be $\infty$ if there is no edge between the two)
on each subsequent step, we include more edges to try and reduce this upper bound, we call this process a relaxation

Shortest Path Problem

relaxation

More formally:
- let $d(v)$ be the distance between the starting vertex $s$ and $v$
- let $d(u)$ be the distance between the starting vertex $s$ and $u$
- to perform a relaxation step on edge $(u,v)$:
  - if $d(v) > d(u) + w(u,v)$:
    - then set $d(v) = d(u) + w(u,v)$
- note that relaxing an edge $(u,v)$ means using that edge to see if you can find a shorter path

Shortest Path Problem

relaxation

in the above example:
- at start, $d(a) = 10$ and $d(b) = 4$
- upon relaxing the edge$(b,a$), we obtain $d(a) = 5$

Bellman-Ford Algorithm

Bellman-Ford algorithm is a single-source shortest-path algorithm which can handle graph with negative edge weights
Some notation:
- Given a graph $G\langle V,E\rangle$, let $s \in V$ be the starting vertex (i.e. we will find the shortest distance from $s$ to every other vertices in $G$)
- For $v \in V$ let $\delta(v)$ be the shortest distance so far from $s$ to $v$
  - $\delta(s) = 0$

Bellman-Ford Algorithm

the algorithm

set $\delta(s) = 0$ and set $\delta(v) = \infty$ for all $v \in V \setminus\{s\}$
for $i = 1$ to $|V|-1$:
- for each edge $(u,v) \in E$, relax $(u,v$)
for each edge $(u,v)\in E$
- if $\delta(v) > \delta(u) + w(u,v)$
  - return false
return true

Bellman-Ford Algorithm

the algorithm

The algorithm is actually very simple: relax every edge $|V|-1$ times
The running time is $O(|V||E|)$
- this reduces to $O(|V|+|E|$ for directed acyclic graph (see CLRS 24.2)
Compare this to Dijkstra's $O(|V|^2)$ (which can be optimised to $O(|E| + |V|\log |V|)$

Bellman-Ford Algorithm

example

-5

-4

$\infty$

$0$

$(s,e)$

$(b,e)$

$(s,b)$

$(b,d)$

$(c,b)$

$(d,c)$

$(e,d)$

$\delta(e) = -4$

$\delta(b) = 8$

no change

$\delta(d) = 17$

no change

$\delta(c) = 12$

$\delta(d) = -3$

Bellman-Ford Algorithm

example

-5

-4

$8$

$-3$

$12$

$-4$

$0$

$(s,e)$

$(b,e)$

$(s,b)$

$(b,d)$

$(c,b)$

$(d,c)$

$(e,d)$

no change

$\delta(c) = -8$

no change

Bellman-Ford Algorithm

example

-5

-4

$8$

$-3$

$-8$

$-4$

$0$

$(s,e)$

$(b,e)$

$(s,b)$

$(b,d)$

$(c,b)$

$(d,c)$

$(e,d)$

no change

$\delta(b) = -1$

no change

Bellman-Ford Algorithm

example

-5

-4

$-1$

$-3$

$-8$

$-4$

$0$

$(s,e)$

$(b,e)$

$(s,b)$

$(b,d)$

$(c,b)$

$(d,c)$

$(e,d)$

no change

$\delta(b) = -1$

no change

Bellman-Ford Algorithm

if the example is not enough we can go through another one in the workshop (let me know)
proof of correctness is in CLRS pg. 652-654, and again, we can go through this in the workshop

Let $G\langle V,E \rangle$ be a weighted directed graph which does not contain any negative cycle
Let $p$ be the shortest path from vertex $i$ to vertex $j$
Recall that the shortest path problem has the optimal substructure property
- i.e. all subpaths of the shortest path are also shortest paths

All Pairs Shortest Path

ShortestPath($i$,$j$)

$w(i,y_1)$ + ShortestPath($y_1$,$j$)

$\dots$

$w(i,y_2)$ + ShortestPath($y_2$,$j$)

$w(i,y_\ell)$ + ShortestPath($y\ell$,$j$)

All Pairs Shortest Path

Let $p$ be the shortest path from vertex $i$ to vertex $j$
Suppose that $p$ contains at most $m$ edges
- we can decompose $p$ into $p^\prime$ and $(k,j)$ where $p^\prime$ is the shortest path from $i$ to $k$ (since the shortest path problem exhibits optimal substructure)
- $p^\prime$ contains at most $(m-1)$ edges

All Pairs Shortest Path

ShortestPath($i$,$j$) with at most $m$ edges

ShortestPath$(i,y_1) + w(y_1,j)$

$\dots$

ShortestPath$(i,y_2) + w(y_2,j)$

ShortestPath$(i,y_\ell) + w(y_\ell,j)$

ShortestPath($i$,$j$)

$w(i,y_1)$ + ShortestPath($y_1$,$j$)

$\dots$

$w(i,y_2)$ + ShortestPath($y_2$,$j$)

$w(i,y_\ell)$ + ShortestPath($y\ell$,$j$)

with at most $(m-1)$ edges

Let $l_{ij}^{(m)}$ be the weight of the shortest path from vertex $i$ to vertex $j$ with at most $m$ edges
We have:

All Pairs Shortest Path

recursive relation

\[ l_{ij}^{(0)} = \begin{cases}0 & \text{if $i = j$},\\ \infty & \text{otherwise.}\end{cases} \]

i.e. when $m = 0$, $l_{ij}^{(0)}$ is the weight of the shortest path between vertex $i$ and vertex $j$ with no edges

For $m \ge 1$, what is the recursive relation?

All Pairs Shortest Path

recursive relation

\[ l_{ij}^{(1)} = \begin{cases}0 & \text{if $i = j$},\\ ? & \text{otherwise.}\end{cases} \]

\[ l_{ij}^{(1)} = \begin{cases}0 & \text{if $i = j$},\\ w(i,j) & \text{otherwise.}\end{cases} \]

\[ l_{ij}^{(2)} =\ ? \]

For $m \ge 1$, what is the recursive relation?

All Pairs Shortest Path

recursive relation

\[ l_{ij}^{(2)} =\ ? \]

\[ l_{ij}^{(2)} =\min_{1 \le k \le n} \left(l_{ik}^{(1)} + w(k,j)\right) \]

note: $w(j,j) = 0$

For $m \ge 1$, what is the recursive relation?

All Pairs Shortest Path

recursive relation

$l_{ij}^{(m)} = $

\[\min_{1 \le k \le n} \left( l_{ik}^{(m-1)} + w(k,j)\right)\]

if there are $n$ vertices, then the shortest path should contain at most $n-1$ edges
- remember, we assume that there are no negative cycles

Approach:
- we compute a series of matrices $L^{(1)}$, $L^{(2)}$, $\dots$, $L^{(n-1)}$, containing $l_{ij}^{(1)}$, $l_{ij}^{(2)}$, $\dots$, $l_{ij}^{(n-1)}$ for all pairs of vertices $(i,j)$
  - i.e. this series of matrices record the weights of the shortest paths between $i$ and $j$ as we consider more and more edges
What is $L^{(1)}$?
- $L^{(1)}$ is the adjacency matrix of the graph $G$

All Pairs Shortest Path

matrix multiplication algorithm

Matrix multiplication approach

example

-5

\[L^{(1)} = \left(\begin{matrix} 0 & 8 & \infty & \infty & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & \infty & \infty \\ \infty & \infty & -5 & 0 & \infty \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

-4

\[L^{(4)} = \left(\begin{matrix} 0 & -1 & -8 & -3 & -4 \\ \infty & 0 & -1 & 4 & 3 \\ \infty & 7 & 0 & 11 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & 3 & -4 & 1 & 0\end{matrix}\right)\]

Matrix multiplication approach

example

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & ? & ? & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

Matrix multiplication approach

example

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & ? & ? & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

start with $L_{12}^{(2)} = l_{12}^{(2)}$
\[l_{12}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,2)\right)\]
- $l_{11}^{(1)} + w(1,2) = 0 + 8 = 8$
- $l_{12}^{(1)} + w(2,2) = 8 + 0 = 8$
- $l_{13}^{(1)} + w(3,2) = \infty + 7 = \infty$
- $l_{14}^{(1)} + w(4,2) = \infty + \infty = \infty$
- $l_{15}^{(1)} + w(5,2) = -4 + \infty = \infty$

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & ? & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

Matrix multiplication approach

example

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & ? & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

compute $L_{13}^{(2)} = l_{13}^{(2)}$
\[l_{13}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,3)\right)\]
- $l_{11}^{(1)} + w(1,3) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,3) = 8 + \infty = \infty$
- $l_{13}^{(1)} + w(3,3) = \infty + 0 = \infty$
- $l_{14}^{(1)} + w(4,3) = \infty + -5 = \infty$
- $l_{15}^{(1)} + w(5,3) = -4 + \infty = \infty$

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

Matrix multiplication approach

example

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & ? & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & ? \\ ? & 0 & ? & ? & ? \\ ? & ? & 0 & ? & ? \\ ? & ? & ? & 0 & ? \\ ? & ? & ? & ? & 0\end{matrix}\right)\]

Matrix multiplication approach

example

ShortestPath($i$,$j$) with at most $m$ edges

ShortestPath$(i,y_1) + w(y_1,j)$

$\dots$

ShortestPath$(i,y_2) + w(y_2,j)$

ShortestPath$(i,y_\ell) + w(y_\ell,j)$

with at most $(m-1)$ edges

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

Matrix multiplication approach

example

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & -4 \\ \infty & 0 & 4 & 4 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & \infty \\ \infty & \infty & -4 & 1 & 0\end{matrix}\right)\]

\[L^{(3)} = \left(\begin{matrix} 0 & 8 & -8 & -3 & -4 \\ \infty & 0 & -1 & 4 & 3 \\ \infty & 7 & 0 & 11 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & 3 & -4 & 1 & 0\end{matrix}\right)\]

\[L^{(4)} = \left(\begin{matrix} 0 & -1 & -8 & -3 & -4 \\ \infty & 0 & -1 & 4 & 3 \\ \infty & 7 & 0 & 11 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & 3 & -4 & 1 & 0\end{matrix}\right)\]

Matrix multiplication approach

algorithm

// Let L = L^(i) and W be the adjacency matrix
// Compute Lp = L^(i+1)

public static int[][] extend(int[][] L, int [][] W) {
  int INF = Integer.MAX_VALUE
  int n = L.length;
  int[][] Lp = new int[n][n];
		
  for (int i = 0; i < n; i++) {
    for (int j = 0; j < n; j++) {
    
      // set Lp[i][j] to infinity at start
      Lp[i][j] = INF;
      for (int k = 0; k < n; k++) {
        // if either W[k][j] or L[i][k] is infinity,
        // then Lp[i][j] should be infinity as well
        if (W[k][j] != INF && L[i][k] != INF) 
          Lp[i][j] = Math.min(Lp[i][j], L[i][k]+W[k][j]);
      }
    
    }
  }		
  return Lp;
}

Matrix multiplication approach

complexity

-5

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

-4

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & -4 \\ \infty & 0 & 4 & 4 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & \infty \\ \infty & \infty & -4 & 1 & 0\end{matrix}\right)\]

Matrix multiplication approach

complexity

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

\[W = \left(\begin{matrix} 0 & 8 & \infty & \infty & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & \infty & \infty \\ \infty & \infty & -5 & 0 & \infty \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & -4 \\ \infty & 0 & 4 & 4 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & \infty \\ \infty & \infty & -4 & 1 & 0\end{matrix}\right)\]

Matrix multiplication approach

complexity

\[L^{(2)}_{14} = \left(\begin{matrix}0 & 8 &\infty & \infty & -4\end{matrix}\right)\left(\begin{matrix}\infty \\ 9 \\ \infty \\ 0 \\ 1\end{matrix}\right)\]

how do you compute $W \times L^{(1)}$?

\[L^{(2)}_{14} = (0\times \infty) + (8 \times 9) + (\infty \times \infty) + (\infty * 0) + (-4 * 1)\]

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & -4 \\ \infty & 0 & 4 & 4 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & \infty \\ \infty & \infty & -4 & 1 & 0\end{matrix}\right)\]

Matrix multiplication approach

complexity

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

\[L^{(2)}_{14} = (0\times \infty) + (8 \times 9) + (\infty \times \infty) + (\infty * 0) + (-4 * 1)\]

do you see the similarity?

Matrix multiplication approach

complexity

compute $L_{14}^{(2)} = l_{14}^{(2)}$
\[l_{14}^{(2)} = \min_{1 \le k \le n} \left( l_{1k}^{(1)} + w(k,4)\right)\]
- $l_{11}^{(1)} + w(1,4) = 0 + \infty = \infty$
- $l_{12}^{(1)} + w(2,4) = 8 + 9 = 17$
- $l_{13}^{(1)} + w(3,4) = \infty + \infty = \infty$
- $l_{14}^{(1)} + w(4,4) = \infty + 0 = \infty$
- $l_{15}^{(1)} + w(5,4) = -4 + 1 = -3$

\[L^{(2)}_{14} = (0\times \infty) + (8 \times 9) + (\infty \times \infty) + (\infty * 0) + (-4 * 1)\]

the algorithm has the same complexity as matrix multiplication, except that we replace some operations:
- multiplication is replaced by addition
- addition is replaced by the $\text{min}$ operation

Matrix multiplication approach

complexity

remember that we assumed that our graph does not have any negative-weight cycles plus the shortest path should not have any cycle
so the shortest path between vertices $i$ and $j$ (assuming it exists) is simple and contain at most $(|V|-1)$ edges
thus $L^{(|V|-1)}$ should give us the shortest-path weights for our graph

Matrix multiplication approach

complexity

since computing $L^{(i+1)}$ from $L^{i}$ and $W$ is equivalent to performing a matrix multiplication, which costs $O(n^3)$ (of an $n \times n$ matrix), then the complexity of the matrix multiplication approach is $O(|V|^4)$
however, this 'matrix multiplication' procedure is also associative, so actually:
- $L^{(1)} = W$
- $L^{(2)} = W^2 = W \times W$
- $L^{(4)} = W^4 = W^2 \times W^2$
- $L^{(8)} = W^8 = W^4 \times W^4$

Matrix multiplication approach

complexity

in other words, we can employ repeated squarings
e.g. to compute $L^{(19)}$, we compute:
- $L^{(1)} \rightarrow L^{(2)} \rightarrow L^{(4)} \rightarrow L^{(8)} \rightarrow L^{(16)} \rightarrow L^{(32)}$
thus the complexity of this algorithm is actually $O(|V|^3\log |V|)$
can we push this further?
- use Strassen's algorithm: $O(|V|^{2.807}\log |V|)$
- use other fast matrix multiplication tricks

Floyd-Warshall algorithm

let $p$ be the shortest path from vertex $i$ to vertex $j$ as before
in the previous method, we decompose $p$ into $p^\prime$ and $(k,j)$ where $p^\prime$ is the shortest path from $i$ to $k$
- we consider the shortest path with just one edge, then increase it to two edges, three edges, and so on
let us now use a different approach to decompose the problem

ShortestPath($i$,$j$) with at most $m$ edges

ShortestPath$(i,y_1) + w(y_1,j)$

$\dots$

ShortestPath$(i,y_2) + w(y_2,j)$

ShortestPath$(i,y_\ell) + w(y_\ell,j)$

with at most $(m-1)$ edges

Floyd-Warshall algorithm

let $V = \{1,2,3\dots,n\}$
take a subset $\{1,2,3,\dots,k\}$ for some $k < n$
let $p$ be the shortest path from $i$ to $j$ with intermediate vertices only in $\{1,2,3,\dots,k\}$
- e.g. $i = 6$, $j = 8$, $V^* = \{1,2,3\}$

$p : 6 \rightarrow 1 \rightarrow 3 \rightarrow 2 \rightarrow 8$

Floyd-Warshall algorithm

the intermediate vertices of $p$ are drawn from $\{1,2,3,\dots,k\}$
consider the shortest path $p^\prime$ from $i$ to $j$ with all intermediate vertices in the set $\{1,2,3,\dots,k-1\}$
- there are two possibilities
  - $k$ is not an intermediate vertex of $p$, so $p$ = $p^\prime$
  - $k$ is an intermediate vertex of $p$
    - this means we can decompose $p$ into two parts:
      - shortest path from $i$ to $k$ with vertices in $\{1,2,\dots,k-1\}$
      - shortest path from $k$ to $j$ with intermediate vertices in $\{1,2,\dots, k-1\}$

Floyd-Warshall algorithm

case 1: $k$ is not an intermediate vertex of $p$, so $p$ = $p^\prime$

$p : 6 \rightarrow 1 \rightarrow 2 \rightarrow 8$

$p^\prime : 6 \rightarrow 1 \rightarrow 2 \rightarrow 8$

Floyd-Warshall algorithm

$p : 6 \rightarrow 1 \rightarrow 3\rightarrow 2 \rightarrow 8$

$p^\prime : 6 \rightarrow 1 \rightarrow 2 \rightarrow 8$

case 2: $k$ is an intermediate vertex of $p$
- this means we can decompose $p$ into two parts:
  - shortest path from $i$ to $k$ with vertices in $\{1,2,\dots,k-1\}$
  - shortest path from $k$ to $j$ with intermediate vertices in $\{1,2,\dots, k-1\}$

Floyd-Warshall algorithm

define $d_{ij}^{(k)}$ to be the weight of a shortest path from vertex $i$ to vertex $j$ for which all intermediate vertices are in the set $\{1,2,\dots,k\}$
what is $d_{ij}^{(0)}$?

it is the weight of the shortest path from $i$ to $j$ with no intermediate vertices (since our vertices start at $1$)
so $d^{(0)}_{ij} = w(i,j)$

Floyd-Warshall algorithm

what about $d_{ij}^{(k)}$?
remember that if $p$ is the shortest path using only vertices $\in \{1,2,\dots,k\}$
- we decompose it by taking out $k$, hence:

\[ d_{ij}^{(k)} = \min\left(d_{ij}^{(k-1)},d_{ik}^{(k-1)} + d_{kj}^{(k-1)}\right)\]

($k$ is not in the shortest path)

($k$ is in the shortest path, so it is the sum of shortest path from $i$ to $k$ and $k$ to $j$)

Floyd-Warshall algorithm

what about $d_{ij}^{(k)}$?
remember that if $p$ is the shortest path using only vertices $\in \{1,2,\dots,k\}$
- we decompose it by taking out $k$, hence:

\[ d_{ij}^{(k)} = \min\left(d_{ij}^{(k-1)},d_{ik}^{(k-1)} + d_{kj}^{(k-1)}\right)\]

($k$ is not in the shortest path)

($k$ is in the shortest path, so it is the sum of shortest path from $i$ to $k$ and $k$ to $j$)

Floyd-Warshall algorithm

as before we compute a series of matrices $D^{(k)}_{ij}$ for $k = \{1,2,\dots,n\}$ where $n$ is the number of vertices
we start with $D^{(0)}_{ij} = W$, the adjacency matrix

Floyd-Warshall algorithm

example

-5

\[D^{(0)} = \left(\begin{matrix} 0 & 8 & \infty & \infty & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & \infty & \infty \\ \infty & \infty & -5 & 0 & \infty \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

-4

the first row in $D^{(1)}$ should be the same (why?)
in fact all the rows should remain the same because no other vertices can go to $1$!

\[D^{(1)} = \left(\begin{matrix} 0 & 8 & \infty & \infty & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & \infty & \infty \\ \infty & \infty & -5 & 0 & \infty \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

Floyd-Warshall algorithm

example

-5

-4

first row:
- column 1 unchanged
- column 2 unchanged (why?)
- column 3
  - can 1 go to 3 via 2?
- column 4
  - can 1 go to 4 via 2?
- column 5 unchanged (why?)

\[D^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & 17 & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & \infty & -5 & 0 & \infty \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

Floyd-Warshall algorithm

example

-5

-4

\[D^{(3)} = \left(\begin{matrix} 0 & 8 & \infty & 17 & -4 \\ \infty & 0 & \infty & 9 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & \infty & \infty & 1 & 0\end{matrix}\right)\]

Floyd-Warshall algorithm

example

-5

-4

\[D^{(5)} = \left(\begin{matrix} 0 & -1 & -8 & -3 & -4 \\ \infty & 0 & -1 & 4 & 3 \\ \infty & 7 & 0 & 11 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & 3 & -4 & 1 & 0\end{matrix}\right)\]

\[D^{(4)} = \left(\begin{matrix} 0 & 8 & 12 & 17 & -4 \\ \infty & 0 & 4 & 9 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & 5 \\ \infty & 3 & -4 & 1 & 0\end{matrix}\right)\]

Floyd-Warshall algorithm

// Let W be the adjacency matrix

void floydWarshall(int W[][]) { 
  int INF = Integer.MAX_VALUE;
  int n = W.length; // assume W is a square 2D array with
  int D[][] = new int[n][n]; 
        
  // set D^(0) = W
  for (int i = 0; i < n; i++) 
    for (int j = 0; j < n; j++) 
      D[i][j] = W[i][j]; 
    
  for (k = 0; k < n; k++){ 
    for (i = 0; i < n; i++){ 
      for (j = 0; j < n; j++){
        // if either D[i][k] or D[k][j] is infinity, 
        // then D[i][j] should not be changed
        if (D[i][k] != INF && D[k][j] != INF){
          if (D[i][k] + D[k][j] < D[i][j]) 
            D[i][j] = D[i][k] + D[k][j]; 
        }
      }
    }
  }
}

Floyd-Warshall algorithm

complexity of the algorithm is $O(|V|^3)$ (a triple loop)
- line 20 should be a $O(1)$ operation

Floyd-Warshall algorithm

comparison with matrix multiplication method

both methods start with the adjacency matrix
in the matrix multiplication method, we start with the shortest path between $i$ and $j$ using just one edge
- we then incrementally included more and more edges
- we end after we have included $|V|-1$ edges
in Floyd-Warshal algorithm, we start with the shortest path between $i$ and $j$ with no intermediate vertices
- we then use vertex 1 as an intermediary, then vertex 2, then vertex 3, and so on
- we end after we have used every vertex as an intermediary

Floyd-Warshall algorithm

comparison with matrix multiplication method

-5

-4

\[L^{(2)} = \left(\begin{matrix} 0 & 8 & \infty & -3 & -4 \\ \infty & 0 & 4 & 4 & 3 \\ \infty & 7 & 0 & 16 & 10 \\ \infty & 2 & -5 & 0 & \infty \\ \infty & \infty & -4 & 1 & 0\end{matrix}\right)\]

make sure you understand the difference!

COMP333 Algorithm Theory and Design - W9 2019 - Graph Algorithms

By Daniel Sutantyo

COMP333 Algorithm Theory and Design - W9 2019 - Graph Algorithms

Lecture notes for Week 9 of COMP333, 2019, Macquarie University