CS111: Operating Systems Principles, Section 1

Spring 2022
Jerry Cain

Section 1: fork, execvp, and Redirection

Section 1: fork, execvp, and Redirection

  • In C, a file can be opened for reading and/or writing using open system call, and you can set the permissions at that time, as well. The open function comes with the following signatures:
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);

  • The first argument names the file you'd like to interact with, e.g. "sh111.cc"

  • The second argument is a bitwise or'ed collection of flags that specifies how you'd like to interact with the file. The argument must include exactly one of the following:

    • O_RDONLY: read only

    • O_WRONLY: write only

    • O_RDWR: read and write (this one won't come up in Project 1)

  • Include one or more of the following when opening a file for writing:
    • O_CREAT: create the file if it doesn't exist
    • O_EXCL: mandate the file be created from scratch
    • O_TRUNC: clear out file contents if it already exists
  • When creating a new file, you include a third argument to specify what the new file's permissions should be (often 0644 for "rw-r--r--")
  • Successful calls to open return file descriptors, which are integer identifiers used with other I/O-related functions to read from and write to files.

Section 1: fork, execvp, and Redirection

Section 1: fork, execvp, and Redirection

  • Here's a more complete list of the I/O-specific functions provided by the operating system.  In fact, the FILE * abstraction you're familiar with from C and the stream abstractions you're familiar with from C++ are all implemented in terms of these functions!





     
  • The read function reads in up to len characters from the file session identified by fd and plants what gets read into the supplied buffer. The return value is the number of bytes actually read, or 0 if the end of the file has been reached.
  • The write function assumes len meaningful bytes reside in the supplied buffer, and writes those characters out to the file accessible through fd.  You can assume that the return value already equal len, as it will be for reasonably small values of len.
  • close does what you'd expect.  It shuts down the file session and frees the associated descriptor up so it can be reused again.
  • You won't need to use read and write for your Project 1, though you'll certainly use open, close, and a few others we'll soon discuss.
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);

ssize_t read(int fd, char buffer[], size_t len);
ssize_t write(int fd, char buffer[], size_t len);

int close(int fd);
  • The tee program that ships with Linux copies everything from standard input to standard output, making zero or more extra copies in the named files supplied as user program arguments.
  • For example, if the file contains 27 bytes—the 26 letters of the English alphabet followed by a newline character—then the following would print the alphabet to standard output and to three files named one.txt, two.txt, and three.txt.

  • If the file vowels.txt contains the five vowels and the newline character, and tee is invoked as follows, one.txt would be rewritten to contain only the English vowels.

$ cat vowels.txt | ./tee one.txt
aeiou
$ cat one.txt 
aeiou
$ cat alphabet.txt | tee one.txt two.txt three.txt
abcdefghijklmnopqrstuvwxyz
$ cat one.txt 
abcdefghijklmnopqrstuvwxyz
$ cat two.txt
abcdefghijklmnopqrstuvwxyz
$ diff one.txt two.txt
$ diff one.txt three.txt
$

Section 1: fork, execvp, and Redirection

  • We'll work through an implementation of our tee program during the review session, but that implementation is on the next slide.

Implementing t to emulate tee

int main(int argc, char *argv[]) {
  int fds[argc];
  fds[0] = STDOUT_FILENO;
  for (size_t i = 1; i < argc; i++)
    fds[i] = open(argv[i], O_WRONLY | O_CREAT | O_TRUNC, 0644);

  char buffer[2048];
  while (true) {
    ssize_t numRead = read(STDIN_FILENO, buffer, sizeof(buffer));
    if (numRead == 0) break;
    for (size_t i = 0; i < argc; i++) write(fds[i], buffer, numRead);
  }

  for (size_t i = 1; i < argc; i++) close(fds[i]);
  return 0;
}
  • Note that argc incidentally equals the number of descriptors we need to write to. That's why we declare an int array (or rather, a descriptor array) of length argc.
  • STDIN_FILENO is a built-in constant for the number 0, which is the descriptor normally linked to standard input. STDOUT_FILENO is a constant for the number 1, which is the default descriptor bound to standard output.

Section 1: fork, execvp, and Redirection

  • Creating new processes: fork
    • Here's a simple program that knows how to spawn new processes. It uses system calls named fork, getpid, and getppid. The full program can be viewed right here.






       
    • Here's the output of two consecutive runs of the above program.
int main(int argc, char *argv[]) {
   std::cout << "Greetings from process " << getpid() 
             << " (with parent " << getppid() << ")" << std::endl;
   pid_t pid = fork();
   assert(pid >= 0);
   std::cout << "Bye-bye from process " << getpid() 
             << " (with parent " << getppid() << ")" << std::endl;
   return 0;
}
myth60$ ./basic-fork 
Greetings from process 29686! (parent 29351)
Bye-bye from process 29686! (parent 29351)
Bye-bye from process 29687! (parent 29686)
myth60$ ./basic-fork 
Greetings from process 29688! (parent 29351)
Bye-bye from process 29688! (parent 29351)
Bye-bye from process 29689! (parent 29688)
  • fork is called once, but it returns twice.
    • getpid and getppid return the process id of the caller and the process id of the caller's parent, respectively.
    • As a result, the output of our program is the output of two processes.
      • We should expect to see a single greeting but two separate bye-byes.
      • Each bye-bye is printed to the console by two different processes. The operating system decides whether the child or the parent gets to print its bye-bye first.
    • fork knows how to clone the calling process, synthesize a nearly identical copy of it, and schedule the copy to run as if it’s been running all along.
      • Think of it as a form of process mitosis, where one process becomes twins.
      • All segments (stack, data, code, etc) are faithfully replicated.
      • All open descriptors are replicated, and these copies are donated to the clone.

Section 1: fork, execvp, and Redirection

  • Reasonable question: Why does the child process insert its text into the same terminal that the parent does? And how?
    • Answer: The parent process’s file descriptors are cloned on fork.
    • Among those descriptors? File descriptors 0, 1, and 2.  The parent and child each print to their own file descriptor 1, and both are linked to the same terminal.

Section 1: fork, execvp, and Redirection

  • Next idea: Synchronizing between parent and child using waitpid
    • Synchronization between parent and child can be managed using yet another system call called waitpid. It can be used to temporarily block a process until that process terminates or stops.

       
    • The first argument supplies the pid of the child process that should complete before waitpid can return, or -1 if you'd like to wait for any one of many child processes.
    • The second argument supplies the address of an integer where process termination information can be placed (or we can pass NULL if we don't need the information).
    • The third argument is a collection of bitwise-or'ed flags we'll study later. For the moment, we'll just go with 0 as the required parameter value, which means that waitpid should only return when the process with the given pid exits.
    • The return value is the pid of the process that successfully exited, or -1 if waitpid fails (perhaps because the pid is invalid, or you passed in other bogus arguments).
pid_t waitpid(pid_t pid, int *status, int options);

Section 1: fork, execvp, and Redirection

  • Third example: Synchronizing between parent and child using waitpid
    • Consider the following program, which is more representative of how fork really gets used in practice (full program, with error checking, is right here).  
    • The parent process correctly waits for the child to complete using waitpid.
int main(int argc, char *argv[]) {
    std::cout << "Before." << std::endl;
    pid_t pid = fork();
    std::cout << "After." << std::endl;
    if (pid == 0) {
        std::cout << "I'm taking CS111!" << std::endl;
        return 111;
    }
    
    int status;
    waitpid(pid, &status, 0);
    assert(WIFEXITED(status) && WEXITSTATUS(status) == 111);
    std::cout << "Student completed CS111 and aced it!" 
              << std::endl;
    return 0;
}
  • The parent lifts child exit information out of the waitpid call, and uses the WIFEXITED macro to examine some high-order bits of its argument to confirm the process exited normally, and it uses the WEXITSTATUS macro to extract the lower eight bits of its argument to produce the child return value (111 as expected).
  • The waitpid call also donates child process-oriented resources back to the system.
  • Rounding out the trio: execvp

     

    • execvp effectively cannibalizes a process to run a different program from scratch.

      • ​path is relative or absolute pathname of the executable to be invoked.
      • argv is the argument vector that should be funneled through to the new executable's main function.
      • path and argv[0]generally end up being the same exact string.
      • If execvp fails to cannibalize the process and install a new executable image within it, it returns -1 to express failure.
      • If execvp succeeds, it 😱 never returns 😱 (to the original executable, anyway)
    • execvp has many variants (execle, execlp, and so forth. Type man execvp to see all of them).
int execvp(const char *path, char *argv[]);

Section 1: fork, execvp, and Redirection

  • Let's leverage fork, waitpid, and execvp to implement an executable called timeout, invoked like this:

     
    • timeout launches the provided command with all of the arguments that follow and allows it to run for up to n seconds before terminating it.
    • If the process running command finishes before time is up, timeout itself returns the exit code of that process without waiting any additional time.
    • If the process running command doesn’t finish before time is up, timeout kills it and returns an exit code of 124.
    • You can programmatically
      terminate a process by
      calling kill(pid, SIGKILL).
      (though you won't need
      to use kill for Project 1).













 

myth62:~$ ./timeout <n> <command> [<arg1> [<argv2 [...]]]
myth62:~$ ./timeout 5 sleep 3
myth62:~$ echo $? # this prints return value of last command
0
myth62:~$ ./timeout 5 sleep 10
myth62:~$ echo $?
124
myth62:~$ ./timeout 1 factor 1234 2345 3456
1234: 2 617
2345: 5 7 67
3456: 2 2 2 2 2 2 2 3 3 3
myth62:~$ echo $?
0
myth62:~$ ./timeout 0 factor 3125250912230709951372256510
myth62:~$ echo $?
124
myth62:~$

Section 1: fork, execvp, and Redirection

  • Check this out!
    ​










     
    • Overarching idea: spawn two processes.
      • The first process self-cannibalizes itself to run the intended command.
      • The second brute-force sleeps for the supplied number of seconds.
      • We rely on waitpid to identify which of the two processes finished first and then just terminate the other. Clever!
      • Provided the command of interest ran to completion, we return its exit status.  If time expires, we return 124.
int main(int argc, char *argv[]) {
    pid_t timed = fork();
    if (timed == 0) { execvp(argv[2], argv + 2); exit(0); }
  
    pid_t timer = fork();
    if (timer == 0) { sleep(atoi(argv[1])); return 0; }
  
    int status;
    pid_t gold = waitpid(-1, &status, 0);
    pid_t silver = gold == timed ? timer : timed;
    kill(silver, SIGKILL);
    waitpid(silver, NULL, 0);
    if (gold == timed) {
        return WEXITSTATUS(status);
    } else {
        return 124;
    }
}

Section 1: fork, execvp, and Redirection

int pipe(int fds[]);
  • Introducing the pipe system call.
    • The pipe system call takes an uninitialized array of two integers—we'll call it fds—and populates it with two file descriptors such that everything written to fds[1] can be read from fds[0].
    • Here's the prototype:

       
    • pipe is particularly useful for allowing parent processes to communicate with spawned child processes.
      • Recall that the file descriptor table of the parent is cloned across fork boundaries and preserved by execvp calls.
      • That means resources referenced by the parent's pipe endpoints are also referenced by the child's copies of them. Neat!

Section 1: fork, execvp, and Redirection

  • How does pipe work?
    • To illustrate how pipe works and how messages can be passed from one process to a second, let's consider the following program (available for play right here):
int main(int argc, char *argv[]) {
    int fds[2];
    pipe(fds);
    pid_t pid = fork();
    if (pid == 0) {
        close(fds[1]);  // close is the fclose of descriptors
        char buffer[6];
        read(fds[0], buffer, sizeof(buffer)); // read is the scanf of descriptors
        std::cout << "Read the following from the pid " << getpid() << ": \"" 
                  << buffer << "\"." << std::endl;
        close(fds[0]);
        return 0;
    }
    close(fds[0]);
    std::cout << "Printing \"hello\" from pid " << getpid() << "." << std::endl;
    write(fds[1], "hello", 6); // write is the printf of descriptors
    close(fds[1]);    
    waitpid(pid, NULL, 0);
    return 0;
}

Section 1: fork, execvp, and Redirection

  • How do pipe and fork work together in this example?
    • The base address of a small integer array called fds is shared with the call to pipe.
    • pipe allocates two descriptors, setting the first to read from a resource and the second to write to that same resource.  Think of this resource as an unnamed file that only the OS knows about.
    • pipe then plants copies of those two descriptors into indices 0 and 1 of the supplied array before it returns.
    • The fork call creates a child process, which itself inherits a shallow copy of the parent's fds array.
      • Immediately after the fork call, anything printed to fds[1] is readable from the parent's fds[0] and the child's fds[0].
      • Both the parent and child are capable of publishing text to the same resource via their copies of fds[1].
    • The parent closes fds[0] before it writes to anything to fds[1] to emphasize the fact that the parent has no need to read anything from the pipe.
    • The child closes fds[1] before it reads from fds[0] to be clear it has zero interest in printing anything to the pipe.

Section 1: fork, execvp, and Redirection

  • On fork, a child process inherits a properly wired file descriptor 1—you know, stdout—that's linked to the terminal.
  • If the child process calls execvp, then the freshly installed executable still directs anything published to descriptor 1 to your screen.
  • Descriptors are duplicated across fork boundaries, allowing pipes to bridge multiple processes. These bridges between processes enable an ability to pass arbitrary data from one process to a second.
     
  • Pipes are typically used to allow one process to pass data to a second. However, a process may want to rewrite a particular descriptor—say, 0, 1, or 2—so that its bound not to the keyboard or the terminal, but rather to one of the pipe endpoints.
  • This is where the dup2 system call comes into play, and this is its prototype.
     
     
  • dup2 rewires the target descriptor so that it references the same I/O that source does.  So, after dup2 returns, source and target are bound to the same resource. If target was referencing something else prior to the call, it is first closed before any rewiring is done.
  • The return value is simply the value of target, provided dup2 manages the rewiring without drama. Otherwise, it returns -1 like any unhappy system call would.

 

int dup2(int source, int target);

Section 1: fork, execvp, and Redirection

Section 1: fork, execvp, and Redirection

  • Here's a sample call to dup2 and close that comes up quite a bit in practice:

     
  • Here, the child's FILENO_STDIN—
    presumably open and linked to the
    screen—is closed, thereby detaching it from
    the terminal it's typically attached to.
    FILENO_STDIN is then bound to the same
    esource that fds[0] is, which, as per our
    illustration, is the read end of the pipe.
  • The subsequent close call shuts down the
    read-end descriptor allocated by the original
    ipe call, leaving the child's descriptor 0 as
    the only descriptor through which material
    residing in this pipe can be read.
  • This has the dramatic effect that all reads
    from file descriptor 0—a number which is
    effectively hardcoded into all scanf, getline, and getc calls—pull in bytes from a pipe instead of the terminal. The fact that the 0 descriptor was rewired under the hood is completely hidden from the implementation of all functions that read from it.
dup2(fds[0], STDIN_FILENO); // STDIN_FILENO is a #define constant for 0
close(fds[0]);
  • The Unix pipeline creates two or more concurrent processes—we'll call them siblings—so they're chained together via their standard input and output streams.
  • More specifically, the standard output of one process is forwarded on to and consumed as the standard input of the next process in the sequence.
     
  • Some examples:










     
  • We're going to focus on pipelines of length 2 right now, and we'll leave pipelines of arbitrary length to Project 1.  Protip: don't fake your understanding of this function.  If you truly understand the implementation to come, you are gold. 
myth51:~$ cat /usr/include/tar.h | wc
    112     600    3786
myth51:~$ echo -e "pear\ngrape\npeach\napricot\nbanana\napple" | sort | grep ap
apple
apricot
grape
myth51:~$ time sleep 5 | sleep 10
real	0m10.004s
user	0m0.006s
sys	0m0.001s
myth55:~$ curl -sL "http://cs111.stanford.edu/odyssey.txt" | sed 's/[^a-zA-Z ]/ /g' | 
> tr 'A-Z ' 'a-z\n' | grep [a-z] | sort -u | 
> comm -23 - <(sort /usr/share/dict/words) | less
myth55:~$

Section 1: fork, execvp, and Redirection

  • Let's implement a binary pipeline function that codes to the following interface:

     
  • pipeline accepts two argument vectors and, assuming both vectors are valid, spawns off twin processes with the added bonus that the standard output of the first is directed to the standard input of the second. 
    • For simplicity, we'll assume all pipeline calls are well-formed and work as expected.  argv1 and argv2 are each valid, NULL-terminated argument vectors, and pids is the base address of an array of length two.
    • We'll also assume all calls to pipe, dup2, close, execvp, and so forth succeed so that you needn't do any error checking whatsoever.
    • pipeline should return without waiting for either of the child processes to finish, and the pids of the two processes are dropped into pids[0] and pids[1].
    • We'll be sure the two processes running in parallel, so that the following takes about 10 seconds instead of 20.

       
void pipeline(char *argv1[], char *argv2[], pid_t pids[]);
pid_t pids[2];
pipeline({"sleep", "10", NULL}, {"sleep", "10", NULL}, pids);

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • The key difference between this function and prior ones is that two child processes are created—true twinsies—to run executables identified by argv1 and argv2.
    • The fork and execvp work is more straightforward than the descriptor work. Note, however, that the child pids are dropped in the two slots of the supplied pids array.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO); // second arg can be 1 instead of constant
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO); // second arg can be 0 instead of constant
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • The pipe call must come before the fork calls, because the pipe endpoints need to exist at the time the child processes come into being.  The ordering here is crucial, else the child processes won't inherit endpoint descriptors related to the pipe.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO);
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO);
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • The first child process inherits clones of the two pipe endpoints. The first child doesn't need fds[0] for anything meaningful, so it's closed right away.  fds[1] is also closed, but only after dup2 uses it to rewire the child's standard output to reference the write end of the pipe.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO);
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO);
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • Once the first child has been spawned, pipeline can close fds[1], since it's of no importance to the second child and doesn't need to be open for the second fork call.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO);
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO);
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • The second child's standard input is rewired to reference the read end of the pipe.  Once dup2 and fds[0] have been used to finagle that rewiring, fds[0] can be closed.  Note that fds[1] wasn't even inherited across the fork boundary.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO);
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO);
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

  • Here's the implementation of my own pipeline function:













     
    • Finally, we close fds[0] in the primary process.
    • Note that there were a total of five close calls across three processes.  This is the cost of creating a pipe ahead of multiple fork calls. There's simply no other programmatic way to enable unidirectional communication channels between sibling processes unless this type of thing is done.
void pipeline(char *argv1[], char *argv2[], pid_t pids[]) {
  int fds[2];
  pipe(fds);
  
  pids[0] = fork();
  if (pids[0] == 0) {
    close(fds[0]);
    dup2(fds[1], STDOUT_FILENO);
    close(fds[1]);
    execvp(argv1[0], argv1);
  }
  close(fds[1]); // was only relevant to first child, so close before second fork

  pids[1] = fork();  
  if (pids[1] == 0) {
    dup2(fds[0], STDIN_FILENO);
    close(fds[0]);
    execvp(argv2[0], argv2);
  }
  close(fds[0]);
}

Section 1: fork, execvp, and Redirection

Section 1: fork, execvp, and Redirection

  • In C, a file can be created opened for reading and/or writing using open system call, and you can set the permissions at that time, as well. The open function comes with the following signatures:
int open(const char *pathname, int flags);
int open(const char *pathname, int flags, mode_t mode);

  • There are many flags (see man 2 open for a list of them), and they can be bitwise or'd together. You must, however, include exactly one of the following flags:

    • O_RDONLY: read only

    • O_WRONLY: write only

    • O_RDWR: read and write (this one won't come up in Project 1)

  • Include one or more of the following when opening a file for writing:
    • O_CREAT: create the file if it doesn't exist
    • O_EXCL: mandate the file be created from scratch, return -1 if file already exists
    • O_TRUNC: clear out file contents if it already exists
  • When (and only when) you're creating a new file using O_CREAT do you include a third argument.  This third argument is used to specific what the new file's permission set should be (often 0644 for "rw-r--r--")
  • Successful calls to open return a file descriptor which can be passed to read, write, close, and dup2.  dup2 is key when supporting file redirection.

Copy of Project 1 Discussion

By Jerry Cain

Made with Slides.com

Copy of Project 1 Discussion

  • 506

More from Jerry Cain