Prim's algorithm

Finish the minimum spanning tree: grow one tree by the cheapest edge leaving it

← OUTLINE

Kruskal, and the union-find behind it

• Kruskal: sort the edges by weight and walk them lightest first. Take an edge if its endpoints are in different pieces, skip it if it would close a cycle; stop at |V| − 1 edges.
• Here: 1, 2, 3, 4 taken; B–D (5) skipped (B and D are already joined through A); then 6 and 7. Weight 23.
• Union-find answers “same piece?”: find follows parent pointers to a root, union links two roots. With path compression and union by rank each call is nearly O(1), so the sort dominates: O(E log E).

KRUSKAL ON THE WEEK 7 GRAPH. TEAL: TAKEN. MAGENTA DASHED: B–D, THE FIRST EDGE SKIPPED.

Connect every town at the lowest total cost

• That is a minimum spanning tree: the cheapest set of edges that leaves everything connected.
• Prim's answer: start anywhere (Salem), grab the cheapest road leaving your network, repeat. Six roads, 372 miles. The 160-mile Portland–Bend run is never chosen.
• The same algorithm lays out power lines and water mains, and, run backwards, clusters data points (single-linkage clustering).

APPROXIMATE ROAD MILES. PDX PORTLAND, HRV HOOD RIVER, SLM SALEM, CRV CORVALLIS, NWP NEWPORT, EUG EUGENE, BND BEND. TEAL: PRIM FROM SALEM.

The algorithm

1. Pick any start vertex s. The tree is {s}.
2. Among all edges with one endpoint in the tree and one outside, take the lightest. Add it and its outside endpoint to the tree.
3. Repeat until every vertex is in the tree.

Walkthrough 1 of 2: start at A

• Tree {A}. Crossing edges: A–B (4), A–D (2). Take A–D (2).
• Tree {A, D}. Crossing: A–B (4), B–D (5), D–F (7), D–E (9). Take A–B (4).
• Tree {A, B, D}. Crossing: B–E (3), D–F (7), B–C (8), D–E (9). Take B–E (3).
• B–D (5) silently stopped being a crossing edge when B joined: both endpoints are inside now.
• Next: the crossing edges now include E–G (1).

TEAL VERTICES: IN THE TREE. AMBER: THE LIGHTEST EDGE LEAVING IT.

Walkthrough 2 of 2: the same tree, a different order

• Take E–G (1), then C–E (6), then D–F (7). Six edges, seven vertices, done.
• Prim's order: 2, 4, 3, 1, 6, 7. Kruskal's order on the same graph: 1, 2, 3, 4, 6, 7.
• Same six edges, total weight 23. With distinct weights there is only one MST, so both must find it.
• Prim's tree is connected at every step. Kruskal's forest is not.

Try it: Prim, one heap pop at a time

Try it yourself: https://courses.lpcordova.phd/tools/weighted-graphs.html?algo=prim&graph=lecture&start=A&controls=0

Prim in Python, and its cost

 1  import heapq
 2  
 3  def prim(adj: dict[str, list[tuple[str, int]]], s: str) -> 
list[tuple[str, str, int]]:
 4      in_tree = {s}
 5      tree: list[tuple[str, str, int]] = []
 6      pq = [(w, s, v) for v, w in adj[s]]       # (weight, from, to)
 7      heapq.heapify(pq)
 8      while pq and len(in_tree) < len(adj):
 9          w, u, v = heapq.heappop(pq)           # lightest candidate
10          if v in in_tree:
11              continue                     # no longer crosses the cut
12          in_tree.add(v)
13          tree.append((u, v, w))
14          for x, wx in adj[v]:
15              if x not in in_tree:
16                  heapq.heappush(pq, (wx, v, x))
17      return tree

• The heap holds candidate edges. Stale ones (both endpoints inside) are discarded when popped.
• Each edge is pushed at most twice and popped at most twice.
• Heap size is O(E), so each push or pop is O(log E) = O(log V).
• Total: O(E log V), the same as Kruskal.
• Kruskal or Prim? A data structure choice, not a correctness one: edge list and union-find, or adjacency lists and a heap.

Weighted shortest paths

When fewest edges is no longer lightest; negative edges and negative cycles

← OUTLINE

Portland to Bend on one charge

• Every map app solves this: a single-source shortest path, weighted in minutes, miles, dollars, or kWh.
• Fewest segments is wrong (PDX ⁠→⁠ HRV ⁠→⁠ BND, 50 kWh). Lightest: PDX ⁠→⁠ GCP ⁠→⁠ WSP ⁠→⁠ BND, 18 − 2 + 14 = 30 kWh.
• A negative cycle would be a loop that gains charge forever. Reality forbids it, so a model with one is wrong, and the algorithm must say so.

ILLUSTRATIVE NET kWh. PDX PORTLAND, GCP GOVERNMENT CAMP, HRV HOOD RIVER, WSP WARM SPRINGS, BND BEND. TEAL: THE LIGHTEST ROUTE.

Weighted, directed graphs

• Each edge (u, v) now has a direction and a weight w(u, v): a distance, a time, a cost.
• The weight of a path is the sum of its edge weights.
• Two ways from S to A here: the direct edge costs 5. Going S ⁠→⁠ B ⁠→⁠ A costs 2 + 1 = 3.
• Fewest edges is not the same as lightest. Monday's BFS counts hops; with weights, that gives the wrong answer.

THE RUNNING EXAMPLE. ONE EDGE HAS A NEGATIVE WEIGHT.

The problem, and what we return

• Input: a weighted directed graph and a source vertex s. Output: δ(s, v) for every v.
• As in BFS, store parent[v], the vertex before v on a shortest path; the parents form a shortest-path tree.
• A negative edge is fine. A negative cycle (weights summing below zero) is not: every lap makes the path cheaper, so no shortest path exists. A correct algorithm must detect it.

TEAL: THE SHORTEST-PATH TREE FROM S, WITH δ(S, v) AT EACH VERTEX.

Bellman-Ford

Relax every edge |V| − 1 times

← OUTLINE

Relaxation: one operation, repeated

• Keep an estimate d[v] for every vertex: the weight of the best path to v found so far. Start with d[s] = 0, everything else ∞.
• Relaxing an edge (u, v) asks one question: is going through u better than what I have for v?

1  def relax(u: str, v: str, w: int, d: dict[str, float], parent: dict[str, str]) -> None:
2      if d[u] + w < d[v]:
3          d[v] = d[u] + w
4          parent[v] = u

Bellman-Ford: the algorithm

1. Set d[s] = 0 and every other estimate to ∞.
2. Repeat |V| − 1 times: relax every edge in the graph, in any fixed order.
3. Make one more pass. If any edge can still be relaxed, report a negative cycle.

Walkthrough 1 of 2: rounds 1 and 2

Edge order every round: C⁠→⁠D, A⁠→⁠C, B⁠→⁠C, B⁠→⁠D, B⁠→⁠A, S⁠→⁠A, S⁠→⁠B.

• Round 1. The first five edges have a tail at ∞: nothing to extend. S ⁠→⁠ A: d[A] = 5. S ⁠→⁠ B: d[B] = 2.
• Round 2. A ⁠→⁠ C: d[C] = 5 + 3 = 8. B ⁠→⁠ D: d[D] = 2 + 6 = 8. B ⁠→⁠ A: 2 + 1 = 3 beats 5, so d[A] = 3.
• d[C] = 8 was computed from the old d[A] = 5. It is now stale, and the next round will fix it.

AFTER ROUND 2. MAGENTA: ESTIMATES THAT CHANGED THIS ROUND.

Walkthrough 2 of 2: rounds 3, 4, and the check

• Round 3. C ⁠→⁠ D: d[D] = 8 − 4 = 4. A ⁠→⁠ C: 3 + 3 = 6 beats 8, so d[C] = 6. Now d[D] is stale.
• Round 4 (round |V| − 1). C ⁠→⁠ D: 6 − 4 = 2 beats 4, so d[D] = 2. Every estimate now equals δ.
• Check pass. No edge can be relaxed: no negative cycle.
• The shortest path to D, S ⁠→⁠ B ⁠→⁠ A ⁠→⁠ C ⁠→⁠ D, has four edges and took four rounds. After round k, every vertex whose shortest path has at most k edges is correct; a simple path has at most |V| − 1 edges, which is why |V| − 1 rounds are enough.

Try it: Bellman-Ford, one relaxation at a time

Try it yourself: https://courses.lpcordova.phd/tools/weighted-graphs.html?algo=bellman&graph=bellman&start=S

Bellman-Ford in Python

 1  def bellman_ford(vertices: list[str], edges: list[tuple[str, str, int]],
s: str) -> tuple[dict[str, float], dict[str, str | None]]:
 2      d: dict[str, float] = {v: float('inf') for v in vertices}
 3      parent: dict[str, str | None] = {v: None for v in vertices}
 4      d[s] = 0
 5      for _ in range(len(vertices) - 1):
 6          for u, v, w in edges:
 7              if d[u] + w < d[v]:               # relax
 8                  d[v] = d[u] + w
 9                  parent[v] = u
10      for u, v, w in edges:                     # check pass
11          if d[u] + w < d[v]:
12              raise ValueError('negative cycle')
13      return d, parent

• Two nested loops: O(V · E).
• Early exit: if a whole round changes nothing, the answers are final; stop.
• On a dense graph this is O(V³). With no negative weights, Dijkstra (next section) brings it to O(E log V) with the same heap Prim used.

Dijkstra

No negative edges? Relax in greedy order with a heap: O(E log V)

← OUTLINE

Salem to everywhere, by road

• Relax in a smarter order: settle the closest unsettled town next; a heap keyed by d[v] hands it over.
• Pop order from Salem: Corvallis 36, Portland 47, Eugene 64, Newport 89, Hood River 109, Bend 132. Each town pops once, final.
• Prim's fiber tree reached Bend through Corvallis and Eugene (210 miles of driving); the shortest-path tree takes Salem–Bend directly, 132. Same graph, two trees.

ROAD MILES FROM SALEM. TEAL: DIJKSTRA'S SHORTEST-PATH TREE. PRIM'S TREE USED CRV–EUG AND EUG–BND INSTEAD.

Dijkstra: the algorithm

1. d[s] = 0, everything else ∞; put s in a min-heap keyed by d.
2. Pop the vertex u with the smallest d. Skip it if already settled; otherwise it is settled and d[u] is final.
3. Relax every edge (u, v); whenever d[v] improves, push (d[v], v). Repeat until the heap is empty.

Try it: Dijkstra, one settled town at a time

Try it yourself: https://courses.lpcordova.phd/tools/weighted-graphs.html?algo=dijkstra&graph=oregon&start=SLM

Dijkstra in Python, and the one thing it cannot do

 1  Adj = dict[str, list[tuple[str, int]]]
 2  Dist, Parent = dict[str, float], dict[str, str | None]
 3  
 4  def dijkstra(adj: Adj, s: str) -> tuple[Dist, Parent]:
 5      d: Dist = {v: float('inf') for v in adj}
 6      parent: Parent = {v: None for v in adj}
 7      d[s] = 0
 8      pq: list[tuple[float, str]] = [(0, s)]
 9      settled: set[str] = set()
10      while pq:
11          du, u = heapq.heappop(pq)
12          if u in settled:                      # stale entry
13              continue
14          settled.add(u)
15          for v, w in adj[u]:
16              if du + w < d[v]:                 # relax
17                  d[v] = du + w
18                  parent[v] = u
19                  heapq.heappush(pq, (d[v], v))
20      return d, parent

• Stale heap entries are skipped, not deleted: no decrease-key, same O(E log V).
• Negative edges break it. Sandy ⁠→⁠ Zigzag 1, Sandy ⁠→⁠ Government Camp 3, Government Camp ⁠→⁠ Zigzag −5 (downhill), Zigzag ⁠→⁠ Rhododendron 1. Rhododendron settles at 2; the true answer is −1. Settled must mean final.
• Negative weights: Bellman-Ford, O(V · E). None: Dijkstra, O(E log V). Unweighted: BFS, O(V + E).

Activity: Two Trees

Prim and Dijkstra on one graph, Dijkstra on another, and one question about a negative edge

← OUTLINE

Two Trees

• For Prim, list the crossing edges each time; the lightest one is the answer, and the list is where mistakes hide.
• For Dijkstra, one row per pop: the vertex, its final distance, and the estimates after relaxing its edges. Settled vertices are never touched again.

Debrief 1 of 3: Prim from S

• Tree {S}: crossing S–A (6), S–B (2). Take S–B (2).
• Tree {S, B}: crossing S–A (6), A–B (5), B–C (7), B–D (11). Take A–B (5).
• Tree {S, A, B}: A–C (4), B–C (7), B–D (11). Take A–C (4).
• Tree {S, A, B, C}: C–E (10), B–D (11), C–D (12). Take C–E (10).
• Tree {S, A, B, C, E}: D–E (9), B–D (11), C–D (12). Take D–E (9). Total 30.

THE MST FROM S: S-B, B-A, A-C, C-E, E-D, WEIGHT 30.

Debrief 2 of 3: Dijkstra from S, and the two trees

• Pop order S, B, A, C, D, E. Shortest paths: A = 6 (direct), B = 2, C = 9 (S–B–C), D = 13 (S–B–D), E = 19 (S–B–C–E).
• Inside the MST: A = 7, C = 11, E = 21, D = 30 (S–B–A–C–E–D) against a true distance of 13. The two trees share only S–B and C–E.
• The MST minimizes the total weight of the edges it keeps; a shortest-path tree minimizes each vertex's distance from s. Different objectives, different trees. The heap compares w(u, v) for Prim and d[u] + w(u, v) for Dijkstra.

THE SHORTEST-PATH TREE FROM S, WITH δ(S, v). COMPARE WITH THE MST.

Debrief 3 of 3: Dijkstra on the shuttle

• Pop order S, Q, P, R, T. Best route S⁠→⁠Q⁠→⁠T, 9 kWh; parents Q←S, P←Q, R←P, T←Q. d[P] dropped from 7 to 4 when Q was popped: a new (4, P) went into the heap and the old (7, P) was popped later and skipped.
• (f) R⁠→⁠T measured as −4. Dijkstra is no longer guaranteed: it settles a vertex on the assumption that no later path can be shorter, and a negative edge breaks that. Run Bellman-Ford.
• (g) With the −4, the best route is S⁠→⁠Q⁠→⁠P⁠→⁠R⁠→⁠T = 3 + 1 + 2 − 4 = 2. A negative cycle would be a loop that gains charge forever, so the data would be wrong; Bellman-Ford's check pass reports it.

THE SHORTEST-PATH TREE FROM S WITH ALL LEGS NON-NEGATIVE. WITH R ⁠→⁠ T = −4 THE ROUTE TO T CHANGES.

Wrap-up

What to keep, and what comes next

← OUTLINE

The one-slide version

• Prim: one growing tree, always the lightest edge leaving it, a heap of candidate edges. O(E log V). Correct by the cut property, like Kruskal.
• Weighted shortest paths: lightest path, not fewest edges. Negative edges fine, negative cycles make the problem undefined.
• Relaxation: try to improve d[v] through (u, v). Estimates are never too low.
• Bellman-Ford: relax every edge |V| − 1 times, then one check pass. O(V · E), negative weights allowed, detects negative cycles.
• An MST is not a shortest-path tree. Different objective, different tree, even on the same graph.
• Dijkstra: no negative weights, so relax in order of distance with a heap: each vertex settles once. O(E log V), the same heap as Prim, a different key.
• Next: dynamic programming. Bellman-Ford was secretly our first DP: d after round k is the best path using at most k edges.

References

1. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, 4th ed. MIT Press, 2022. Section 21.2 (Prim's algorithm), Chapter 22 introduction, Section 22.1 (Bellman-Ford), and Section 22.3 (Dijkstra).
2. E. W. Dijkstra, “A note on two problems in connexion with graphs,” Numerische Mathematik, vol. 1, pp. 269–271, 1959.
3. R. C. Prim, “Shortest connection networks and some generalizations,” Bell System Technical Journal, vol. 36, no. 6, pp. 1389–1401, 1957.
4. R. Bellman, “On a routing problem,” Quarterly of Applied Mathematics, vol. 16, no. 1, pp. 87–90, 1958.
5. L. R. Ford Jr., “Network flow theory,” RAND Corporation, Paper P-923, 1956.