CS110 Lecture 12: More Multithreading Examples - Ice Cream Store

Principles of Computer Systems

Winter 2021

Stanford University

Computer Science Department

Instructors: Chris Gregg and

Nick Troccoli

PDF of this presentation

CS110 Topic 3: How can we have concurrency within a single process?

Learning About Threads

Introduction to Threads

Threads and Mutexes

Condition Variables and Semaphores

Multithreading Patterns

Lecture 9

Lecture 9-10

Lecture 10

Lecture 11

Multithreading Examples

Lecture 12

Learning Goals

Review all of the multithreading techniques we've learned to coordinate threads
Understand how we can apply multiple techniques in a larger multithreading program

Lecture Plan

Reviewing Multithreading Techniques
Visiting the Ice Cream Store

Lecture Plan

Reviewing Multithreading Techniques
- Mutex: Binary Lock
- Condition Variable: Generalized Wait
- Semaphore: Permits and Thread Coordination
Visiting the Ice Cream Store

Mutex

A lock with one holder at a time that lets us enforce mutual exclusion
Enables multithreading pattern #1: binary lock - we have multiple threads, but we want only one thread at a time to be able to execute some code (e.g. modifying shared data structure).
Optionally paired with a lock_guard that locks when created, and unlocks when destroyed.

Condition Variable

Allows thread communication by letting threads wait or notify other waiting threads.
Enables multithreading pattern #2: generalized wait - we want a thread to go to sleep until it's notified by another thread that some condition is true.
Note: a condition variable provides a more general form of waiting not limited to a permits count (e.g. useful where a wait condition is more complex than a counter being > 0).

Semaphore

Combines a condition variable, mutex and int to track a count of something. Threads can increment this count or wait for the count to be > 0.
Enables multithreading pattern #3: permits - model a finite, discrete number of available resources. We initialize the semaphore with the initial resource count. If a thread needs the resource, it calls wait on the semaphore. Threads may also signal to indicate more resources are available.
Enables multithreading pattern #4: binary thread coordination - we want thread A to proceed only once thread B completes some work. We initialize the semaphore to 0; thread A waits on it, and thread B signals when done. The semaphore thus records the status of one event (0 = not-yet-completed or completed-and-checked and 1 = completed-but-not-yet-checked). This is like a general version of thread::join().

Semaphore

Enables multithreading pattern #5: general thread coordination - Thread A waits for something to happen n times before proceeding. Have a semaphore initialized to -n + 1; thread A waits on it, and other threads signal. Thus, we can imagine that there are missing permits that must be returned for A to advance. Or, initialize to 0; thread A waits n times on it, and other threads signal. Thus, there are no permits initially, and A must have n permits to advance. This is like a general version of thread::join().

Multithreading Patterns

Binary lock (mutex) - e.g. dining philosophers' forks
Generalized wait (condition variable) - e.g. waiting for complex condition
Permits (semaphore) - e.g. dining philosophers permits for eating
Binary coordination (semaphore) - e.g. writer telling reader there is new content
Generalized coordination (semaphore) - e.g. thread waits for N others to finish a task
Layered Construction (combo) - combine multiple patterns

Layered Construction

Layered Construction (combo) - combine multiple patterns

Examples:

dining philosophers: mutexes for forks, semaphore for permits
worker threads operate on a shared data structure protected by a mutex. When they're done, they signal a generalized coordination semaphore, and a final thread waits for all workers to signal before continuing.
threads A and B compete to be the first to finish work and signal C to continue via a binary coordination semaphore. To ensure the winner's info is recorded, the winning thread acquires a mutex, updates a shared variable with their information, and then signals C to continue.

Lecture Plan

Reviewing Multithreading Techniques
Visiting the Ice Cream Store

Visiting The Ice Cream Store

Now, let's use our multithreading knowledge to understand an in-depth multithreading program simulating an ice cream store!
There are customers, clerks, a manager and a cashier, coordinating in various ways.

An icon for a "CS110 Ice cream store", saying "CS110 Ice Cream, Inc." and "Multiple threads of flavor!"

ice-cream-store.cc

Visiting The Ice Cream Store

Each customer wants to order some number of ice cream cones.
A customer spawns a new clerk to make each ice cream cone.
A clerk makes a single cone, and must have it approved by the manager.
The single manager approves or rejects cones made by clerks.
Once a customer's order is made, they must get in line with the cashier to check out.
The cashier helps customers check out in the order in which they got on line.

Ice Cream Store: scaffolding

static mutex rgenLock;
static RandomGenerator rgen;

static unsigned int getNumCones() {
  lock_guard<mutex> rgenLockGuard(rgenLock);
  return rgen.getNextInt(kMinConeOrder, kMaxConeOrder);
}

static unsigned int getBrowseTimeMS() {
  lock_guard<mutex> rgenLockGuard(rgenLock);
  return rgen.getNextInt(kMinBrowseTimeMS, kMaxBrowseTimeMS);
}

static unsigned int getPrepTimeMS() {
  lock_guard<mutex> rgenLockGuard(rgenLock);
  return rgen.getNextInt(kMinPrepTimeMS, kMaxPrepTimeMS);
}

static unsigned int getInspectionTimeMS() {
  lock_guard<mutex> rgenLockGuard(rgenLock);
  return rgen.getNextInt(kMinInspectionTimeMS, kMaxInspectionTimeMS);
}

static bool getInspectionOutcome() {
  lock_guard<mutex> rgenLockGuard(rgenLock);
  return rgen.getNextBool(kConeApprovalProbability);
}

To model a "real" ice cream store, we want to randomize different occurrences throughout the program. We use these functions to do that.

Ice Cream Store: scaffolding

To model a "real" ice cream store, we want to randomize different occurrences throughout the program. We use these functions to do that.

static void browse() {
  cout << oslock << "Customer starts to kill time." << endl << osunlock;
  unsigned int browseTimeMS = getBrowseTimeMS();
  sleep_for(browseTimeMS);
  cout << oslock << "Customer just killed " << double(browseTimeMS) / 1000
       << " seconds." << endl << osunlock;
}

static void makeCone(unsigned int coneID, unsigned int customerID) {
  cout << oslock << "    Clerk starts to make ice cream cone #" << coneID 
       << " for customer #" << customerID << "." << endl << osunlock;
  unsigned int prepTimeMS = getPrepTimeMS();
  sleep_for(prepTimeMS);
  cout << oslock << "    Clerk just spent " << double(prepTimeMS) / 1000 
       << " seconds making ice cream cone #" << coneID 
       << " for customer #" << customerID << "." << endl << osunlock;
}

Ice Cream Store: main

int main(int argc, const char *argv[]) {

  // Make an array of customer threads, and add up how many cones they order
  int totalConesOrdered = 0;
  thread customers[kNumCustomers];
  for (size_t i = 0; i < kNumCustomers; i++) {
    int numConesWanted = getNumCones();
    customers[i] = thread(customer, i, numConesWanted);
    totalConesOrdered += numConesWanted;
  }

  /* Make the manager and cashier threads to approve cones / checkout customers.
   * Tell the manager how many cones will be ordered in total. */
  thread managerThread(manager, totalConesOrdered);  
  thread cashierThread(cashier);

  // Join all the threads
  for (thread& customer: customers) customer.join();
  cashierThread.join();
  managerThread.join();

  return 0;
}

In main, we spawn all of the customers, the manager (telling it the total number of cones ordered), and the cashier. Why not clerks? Each customer spawns its own clerks.

Then, we wait for the threads to finish.

Ice Cream Store: customer

A customer does the following:

spawns a clerk for each cone
browses and waits for clerks to finish
gets its number in checkout line
tells cashier we are ready to check out
waits for cashier to ring us up

"gets its number in checkout line" - global counter, needs a binary lock

"tells cashier we are ready to check out" - one generalized coordination semaphore

"waits for cashier to ring us up" - binary coordination semaphore per customer

Ice Cream Store: customer

struct checkout {
  checkout(): nextPlaceInLine(0) {}
  atomic<unsigned int> nextPlaceInLine;
  semaphore waitingCustomers;
  semaphore customers[kNumCustomers];
} checkout;

Global struct shared by all customers and the cashier.

nextPlaceInLine is a counter that is automatically atomic for ++!
waitingCustomers is a generalized coordination semaphore that the cashier waits on
customers stores a binary coordination semaphore per customer, customers wait on them

Ice Cream Store: customer

static void customer(unsigned int id, unsigned int numConesWanted) {
  // Make a vector of clerk threads, one per cone to be ordered
  vector<thread> clerks(numConesWanted);
  for (unsigned int i = 0; i < clerks.size(); i++) {
    clerks[i] = thread(clerk, i, id);
  }

  browse();
  for (thread& clerk: clerks) clerk.join();

  // Now we are ready to check out.  Get our unique place in line.
  int place = checkout.nextPlaceInLine++;
  cout << oslock << "Customer " << id << " assumes position #" << place 
       << " at the checkout counter." << endl << osunlock;

  checkout.waitingCustomers.signal();

  checkout.customers[place].wait();
  cout << oslock << "Customer " << id 
       << " has checked out and leaves the ice cream store." 
       << endl << osunlock;
}

A customer does the following:

1) spawns a clerk for each cone

2) browses and waits for clerks

3) gets its place in checkout line

4) tells cashier it's there

5) waits for cashier to ring it up

struct checkout {
  checkout(): nextPlaceInLine(0) {}
  atomic<unsigned int> nextPlaceInLine;
  semaphore waitingCustomers;
  semaphore customers[kNumCustomers];
} checkout;

Ice Cream Store: clerk

A clerk does the following:

makes a cone
attempts to get exclusive access to the manager
tells the manager it needs approval
waits for the manager to decide whether to approve or reject
checks the manager's decision
forfeits exclusive access to the manager
if our cone was rejected, go to step 1

"attempts to get exclusive access to the manager" - binary lock

"tells the manager it needs approval" - binary coordination semaphore

"waits for the manager to decide..." - binary coordination semaphore

Ice Cream Store: clerk

struct {
  mutex available;
  semaphore requested;
  semaphore finished;
  bool passed;
} inspection;

Global struct shared by all clerks and the manager.

available is a lock that a clerk must hold in order to interact with the manager.
requested is a binary coordination semaphore that the manager waits on
finished is a binary coordination semaphore that a clerk waits on
passed stores the result of the most recent inspection - only for lock-holder.

Ice Cream Store: clerk

static void clerk(unsigned int coneID, unsigned int customerID) {
  bool success = false;
  while (!success) {
    makeCone(coneID, customerID);

    // We must be the only one requesting approval
    inspection.available.lock();

    // Let the manager know we are requesting approval
    inspection.requested.signal();

    // Wait for the manager to finish
    inspection.finished.wait();

    /* If the manager is finished, it has put its approval 
     * decision into "passed" */
    success = inspection.passed;

    // We're done requesting approval
    inspection.available.unlock();
  }
}

A clerk does the following:

makes a cone (do this first!)
gets exclusive manager access
tells the manager it needs approval
waits for the manager to decide
checks the manager's decision
forfeits manager access
if rejected, go to step 1

struct {
  mutex available;
  semaphore requested;
  semaphore finished;
  bool passed;
} inspection;

Ice Cream Store: manager

"waits for a clerk's cone to inspect" - binary coordination semaphore

"tells the clerk that we are done" - binary coordination semaphore

The single manager does the following while there are more cones needed:

waits for a clerk to request an inspection
inspects the cone and records decision to approve or not
tells the clerk that it is done
updates its cone counts
if more cones needed, go to step 1

Ice Cream Store: manager

struct {
  mutex available;
  semaphore requested;
  semaphore finished;
  bool passed;
} inspection;

Global struct shared by all clerks and the manager.

available is a lock that a clerk must hold in order to interact with the manager.
requested is a binary coordination semaphore that the manager waits on
finished is a binary coordination semaphore that a clerk waits on
passed stores the result of the most recent inspection - only for lock-holder.

Ice Cream Store: manager

static void manager(unsigned int numConesNeeded) {
  unsigned int numConesAttempted = 0;
  unsigned int numConesApproved = 0;

  while (numConesApproved < numConesNeeded) {
    // Wait for someone to request an inspection
    inspection.requested.wait();

    inspectCone();

    // Let them know we have finished inspecting
    inspection.finished.signal();

    // Update our counters
    numConesAttempted++;
    if (inspection.passed) numConesApproved++;
  }
  
  cout << oslock << "  Manager inspected a total of " 
       << numConesAttempted 
       << " ice cream cones before approving a total of " 
       << numConesNeeded 
       << "." << endl << "  Manager leaves the ice cream store." 
       << endl << osunlock;
}

The manager does the following while there are more cones needed:

waits for a clerk's cone to inspect
inspects the cone and records decision to approve or not.
tells the clerk that it is done.
updates its cone counts
if more cones needed, go to 1

struct {
  mutex available;
  semaphore requested;
  semaphore finished;
  bool passed;
} inspection;

Ice Cream Store: manager

static void inspectCone() {
  cout << oslock << "  Manager is presented with an ice cream cone." 
       << endl << osunlock;

  // Sleep for to simulate inspecting the cone
  unsigned int inspectionTimeMS = getInspectionTimeMS();
  sleep_for(inspectionTimeMS);

  // Generate a decision and put it into the `passed` field
  // No locks needed - we are only one accessing right now
  inspection.passed = getInspectionOutcome();
  string verb = inspection.passed ? "APPROVED" : "REJECTED";
  cout << oslock << "  Manager spent " 
       << double(inspectionTimeMS) / 1000
       << " seconds analyzing presented ice cream cone and " 
       << verb << " it." 
       << endl << osunlock;
}

The manager does the following while there are more cones needed:

waits for a clerk's cone to inspect
inspects the cone and records decision to approve or not.
tells the clerk that it is done.
updates its cone counts
if more cones needed, go to 1

struct {
  mutex available;
  semaphore requested;
  semaphore finished;
  bool passed;
} inspection;

Ice Cream Store: cashier

"waits for a customer to be ready to check out" - generalized coordination semaphore

"tells the i-th customer that it has checked out" - binary coordination semaphore per customer

The single cashier does the following while there are more customers to ring up:

waits for a customer to be ready to check out
tells the i-th customer that it has checked out
if more customers to ring up, go to step 1

Ice Cream Store: cashier

struct checkout {
  checkout(): nextPlaceInLine(0) {}
  atomic<unsigned int> nextPlaceInLine;
  semaphore waitingCustomers;
  semaphore customers[kNumCustomers];
} checkout;

Global struct shared by all customers and the cashier.

nextPlaceInLine is a counter that is automatically atomic for ++!
waitingCustomers is a generalized coordination semaphore that the cashier waits on
customers stores a binary coordination semaphore per customer, customers wait on them

Ice Cream Store: cashier

static void cashier() {
  cout << oslock 
       << "      Cashier is ready to help customers check out." 
       << endl << osunlock;

  // We check out all customers
  for (unsigned int i = 0; i < kNumCustomers; i++) {
    // Wait for someone to let us know they are ready to check out
    checkout.waitingCustomers.wait();
    cout << oslock << "      Cashier rings up customer " << i << "." 
         << endl << osunlock;

    // Let the ith customer know that they can leave.
    checkout.customers[i].signal();
  }

  cout << oslock << "      Cashier is all done and can go home." 
       << endl << osunlock;
}

The cashier does the following while there are more customers to ring up:

waits for a customer to be ready to check out
tells the i-th customer that it has checked out
if more customers to ring up, go to step 1

struct checkout {
  checkout(): nextPlaceInLine(0) {}
  atomic<unsigned int> nextPlaceInLine;
  semaphore waitingCustomers;
  semaphore customers[kNumCustomers];
} checkout;

Takeaways

There is a lot going on in this program!
Managing all of the threads, locking, waiting, etc., takes planning and foresight.
This isn't the only way to model the ice cream store
- How would you modify the model?
- What would we have to do if we wanted more than one manager?
- Could we create multiple clerks in main, as well? (sure)
This example prepares us for the next idea: ThreadPool.
- It does take time to spin up a thread, so if we have the threads already waiting, we can use them quickly. This is similar to farm, except that now, instead of processes, we have threads.

Recap

Reviewing Multithreading Techniques
- Mutex: Binary Lock
- Condition Variable: Generalized Wait
- Semaphore: Permits and Thread Coordination
Visiting the Ice Cream Store

Next time: Networking