CS110: Principles of Computer Systems
Autumn 2021
Jerry Cain
PDF
Lecture 25: Slow System Calls, Nonblocking I/O
- Fast system calls are those that return immediately, where immediately means they just need the processor and other local resources to get their work done.
- By this definition, there's no hard limit on the time they're allowed to take.
- Even if a system call were to take 60s to complete, I'd consider it to be fast if all 60 seconds were spent executing code—i.e. nothing blocked on external resources.
- Slow system calls are those that wait for an indefinite stretch of time for something outside the process to finish (e.g.
waitpid
), for something to become available (e.g.read
from a client socket that's not seen any data recently), or for some external event (e.g. network connection request from client viaaccept
.)- Some consider calls to
read
are considered fast—at least in practice—if they're reading from a local file, because there aren't really any external resources to preventread
from doing its work. Yes, some hardware needs to be accessed, but because that hardware is grafted into the machine, we can say with some confidence that the data being read from the local file will be available within a certain amount of time. -
write
calls are slow if data is being written to a socket and previously published data has congested internal buffers and not been pushed off the machine yet.
- Some consider calls to
- From the point of view of the kernel, a fast system call can complete without blocking or waiting and fully execute without needing to yield and reschedule the process.
Lecture 25: Slow System Calls, Nonblocking I/O
- Slow system calls are the ones capable of blocking a thread of execution indefinitely, rendering that thread inert until the system call returns.
- In some cases, we relied on signals and asynchronous signals handlers to lift calls to
waitpid
out of the normal flow of execution, and we've also relied onWNOHANG
to ensure thatwaitpid
never actually blocks.- When we use WNOHANG, our waitpid call is nonblocking! We just didn't call it that back when we learned it.
- When we use WNOHANG, our waitpid call is nonblocking! We just didn't call it that back when we learned it.
- We've relied on multithreading to get calls to
read
andwrite
off the main thread, because they're blocking.- Threading doesn't make the calls to
read
andwrite
any faster, but it does parallelize the stall times and free up the main thread so it can do other things. - You're familiar with these ideas because of
aggregate
andproxy
.
- Threading doesn't make the calls to
- When implementing servers and networked clients,
accept
,read
, and even write are slow.- Remember that read and write are embedded without our iosockstream methods.
- In some cases, we relied on signals and asynchronous signals handlers to lift calls to
Lecture 25: Slow System Calls, Nonblocking I/O
- Making Slow System Calls Fast
- It's possible to configure a traditional descriptor to be nonblocking. When nonblocking descriptors are passed to
read
orwrite
, it'll return as quickly as possible without waiting.- If one or more bytes of data are available when
read
is called, then some or all of those bytes are written into the supplied character buffer, and the number of bytes placed is returned. - If no data is available but the source of the data hasn't been shut down, then
read
will return -1, provided the descriptor has been configured to be nonblocking. Again, this -1 normally denotes that some error occurred, but in this case, errno is set to EWOULDBLOCK. The -1/EWOULDBLOCK
combination is just saying that the call toread
would have blocked had the descriptor been a blocking one. - Of course, if when
read
is called it's clear there'll never be any more data, read will return 0 as it has all along. - All of this applies to the
write
system call as well, though it's rare forwrite
to return -1 unless a genuine error occurred, even if the supplied descriptor is nonblocking.
- If one or more bytes of data are available when
- It's possible to configure a traditional descriptor to be nonblocking. When nonblocking descriptors are passed to
Lecture 25: Slow System Calls, Nonblocking I/O
- Making Slow System Calls Fast
- It's also possible to configure a server socket to be nonblocking. When nonblocking server sockets are passed to
accept
, it'll always return as quickly as possible.- If a client connection is available when
accept
is called, it'll return immediately with a socket connection to that client. - Provided the server socket is configured to be nonblocking,
accept
will return -1 instead of blocking if there are no pending connection requests. The -1 normally denotes that some error occurred, but iferrno
is set toEWOULDBLOCK
, the -1 isn't truly identifying an error, but instead saying thataccept
would have blocked had the server socket passed to it been a traditional (i.e. blocking) socket descriptor.
- If a client connection is available when
- It's also possible to configure a server socket to be nonblocking. When nonblocking server sockets are passed to
Lecture 25: Slow System Calls, Nonblocking I/O
- First example: consider the following
slow-alphabet-server
implementation:
static const string kAlphabet = "abcdefghijklmnopqrstuvwxyz";
static const useconds_t kDelay = 100000; // 100000 microseconds is 100 ms is 0.1 second
static void handleRequest(int client) {
sockbuf sb(client);
iosockstream ss(&sb);
for (size_t i = 0; i < kAlphabet.size(); i++) {
ss << kAlphabet[i] << flush;
usleep(kDelay); // 100000 microseconds is 100 ms is 0.1 seconds
}
}
static const short kSlowAlphabetServerPort = 41411;
int main(int argc, char *argv[]) {
int server = createServerSocket(kSlowAlphabetServerPort);
ThreadPool pool(128);
while (true) {
int client = accept(server, NULL, NULL);
pool.schedule([client]() { handleRequest(client); });
}
return 0;
}
Lecture 25: Slow System Calls, Nonblocking I/O
-
slow-alphabet-server
operates much liketime-server-concurrent
does. - The protocol:
- wait for an incoming connection
- delegate responsibility to handle that connection to a worker within a
ThreadPool
- have worker very slowly spell out the alphabet over 2.6 seconds
- close connection (which happens because the
sockbuf
is destroyed).
- There's nothing nonblocking about
slow-alphabet-server
, but it's intentionally slow to emulate the time a server might take to synthesize a full response.- Many servers, like the one above, push out partial responses to the client. The accumulation of all piecemeal responses becomes the full payload.
- You should be familiar with the partial response concept if you've ever surfed YouTube or any other video content site like it, as you very well know that a video starts playing before the entire video has downloaded.
Lecture 25: Slow System Calls, Nonblocking I/O
- Presented here is a traditional (i.e. blocking) client of the
slow-alphabet-server
:
int main(int argc, char *argv[]) {
int client = createClientSocket("localhost", kSlowAlphabetServerPort);
size_t numSuccessfulReads = 0;
size_t numBytes = 0;
while (true) {
char ch;
ssize_t count = read(client, &ch, 1);
assert(count != -1);
numBytes++;
if (count == 0) break;
assert(count == 1);
numSuccessfulReads++;
cout << ch << flush;
}
close(client);
cout << endl;
cout << "Alphabet Length: " << numBytes << " bytes." << endl;
cout << "Num reads: " << numSuccessfulReads << endl;
return 0;
}
Lecture 25: Slow System Calls, Nonblocking I/O
- Full implementation is right here:
- Relies on traditional client socket as returned by
createClientSocket
. - Character buffer passed to read is of size 1, thereby constraining the range of legitimate return values to be [-1, 1].
- Provided the slow-alphabet-server is running, a client run on the same machine would reliably behave as follows:
- Relies on traditional client socket as returned by
myth57:$ ./slow-alphabet-server &
[1] 7516
myth57:$ ./blocking-alphabet-client
abcdefghijklmnopqrstuvwxyz
Alphabet Length: 26 bytes.
Num reads: 26
myth57:$ time ./blocking-alphabet-client
abcdefghijklmnopqrstuvwxyz
Alphabet Length: 26 bytes.
Num reads: 26
real 0m2.609s
user 0m0.004s
sys 0m0.000s
myth57:$ kill -KILL 7516
[1] Killed ./slow-alphabet-server
Lecture 25: Slow System Calls, Nonblocking I/O
- Presented here is the nonblocking version:
static const unsigned short kSlowAlphabetServerPort = 41411;
int main(int argc, char *argv[]) {
int client = createClientSocket("localhost", kSlowAlphabetServerPort);
setAsNonBlocking(client);
size_t numReads = 0, numSuccessfulReads = 0, numUnsuccessfulReads = 0, numBytes = 0;
while (true) {
char ch;
ssize_t count = read(client, &ch, 1);
if (count == 0) break;
numReads++;
if (count > 0) {
assert(count == 1);
numSuccessfulReads++;
numBytes++;
cout << ch << flush;
} else {
assert(errno == EWOULDBLOCK || errno == EAGAIN);
numUnsuccessfulReads++;
}
}
close(client);
cout << endl;
cout << "Alphabet Length: " << numBytes << " bytes." << endl;
cout << "Num reads: " << numReads << " ("
<< numSuccessfulReads << " successful, "
<< numUnsuccessfulReads << " unsuccessful)." << endl;
return 0;
}
Lecture 25: Slow System Calls, Nonblocking I/O
- Full implementation is right here:
- Because client socket is configured to be nonblocking,
read
is incapable of blocking.- Data available? Expect
ch
to be updated and for theread
call to return 1. - No data available, ever? Expect
read
return a 0. - No data available right now, but possibly in the future? Expect a return value of -1 and
errno
to be set toEWOULDBLOCK
- Data available? Expect
- Because client socket is configured to be nonblocking,
myth57:$ ./slow-alphabet-server &
[1] 9801
myth57:$ ./non-blocking-alphabet-client
abcdefghijklmnopqrstuvwxyz
Alphabet Length: 26 bytes.
Num reads: 11394589 (26 successful, 11394563 unsuccessful).
myth57:$ time ./non-blocking-alphabet-client
abcdefghijklmnopqrstuvwxyz
Alphabet Length: 26 bytes.
Num reads: 11268990 (26 successful, 11268964 unsuccessful).
real 0m2.607s
user 0m0.264s
sys 0m2.340s
myth57:$ kill -KILL 9801
[1] Killed ./slow-alphabet-server myth57:$
- Inspect the output of our nonblocking client.
- Reasonable questions: Is this better? Why rely on nonblocking I/O other than because we can? We'll answer these questions very soon.
- Interesting statistics, and just look at the number of read calls!
Lecture 25: Slow System Calls, Nonblocking I/O
- The
OutboundFile
class is designed to read a local file and push its contents out over a supplied descriptor (and to do so without ever, ever, ever blocking). - Here's an abbreviated interface file:
- The constructor configures an instance of the
OutboundFile
class. -
initialize
supplies the local file that should be used as a data source and the descriptor where that file's contents should be replicated. -
sendMoreData
pushes as much data as possible to the supplied sink, without blocking. It returnstrue
if it's at all possible there's more payload to be sent, andfalse
if all data has been fully pushed out. The fully documented interface file is right here.
class OutboundFile {
public:
OutboundFile();
void initialize(const std::string& source, int sink);
bool sendMoreData();
private:
// implementation details omitted for the moment
}
Lecture 25: Slow System Calls, Nonblocking I/O
- Here's a simple program we can use to ensure the OutboundFile implementation is working:
/**
* File: outbound-file-test.cc
* --------------------------
* Demonstrates how one should use the OutboundFile class
* and can be used to confirm that it works properly.
*/
#include "outbound-file.h"
#include <unistd.h>
int main(int argc, char *argv[]) {
OutboundFile obf;
obf.initialize("outbound-file-test.cc", STDOUT_FILENO);
while (obf.sendMoreData()) {;}
return 0;
}
- The above program prints its own source to standard output.
- A full copy of the program can be found right here.
Lecture 25: Slow System Calls, Nonblocking I/O
- The outbound-file-test.cc presented earlier can be used to confirm the
OutboundFile
class implementation works as expected.- The nonblocking aspect of the code doesn't really buy us anything.
- Only one copy of the source file is being syndicated, so no harm comes from blocking, since there's nothing else to do.
- To see why nonblocking I/O might be useful, consider the following nonblocking server implementation, presented over several slides.
- Our server is a static file server and responds to every single request—no matter how that request is structured—with the same payload. That payload is drawn from an HTML called "expensive-server.cc.html".
- Presented below is the portion of the server implementation that establishes the executable as a server that listens to port 12345 and sets the server socket to be nonblocking.
// expensive-server.html is expensive because it's always using the CPU, even when there's nothing to do
static const unsigned short kDefaultPort = 12345;
static const string kFileToServe("expensive-server.cc.html");
int main(int argc, char **argv) {
int server = createServerSocket(kDefaultPort);
assert(server != kServerSocketFailure);
setAsNonBlocking(server);
cout << "Static file server listening on port " << kDefaultPort << "." << endl;
// more code follows
Lecture 25: Slow System Calls, Nonblocking I/O
- As with all servers, our static file server loops interminably and aggressively accepts incoming connections as quickly as possible.
- The first part of the while loop calls and immediately returns from
accept
. The return is immediate, because server has been configured to be nonblocking. - The code immediately following the accept call branches in one of two directions.
- If
accept
returns a -1, we verify the -1 isn't something to be concerned about. - If
accept
surfaces a new connection , we create an newOutboundFile
on its behalf and append it to the runningoutboundFiles
list of clients currently being served.
- If
- The first part of the while loop calls and immediately returns from
list<OutboundFile> outboundFiles;
while (true) {
// part 1: below
int client = accept(server, NULL, NULL);
if (client == -1) {
assert(errno == EWOULDBLOCK); // confirm -1 isn't a true failure
} else {
OutboundFile obf;
obf.initialize(kFileToServe, client);
outboundFiles.push_back(obf);
}
// part 2: presented on next slide
Lecture 25: Slow System Calls, Nonblocking I/O
- As with all servers, our static file server loops interminably and aggressively accepts incoming connections as quickly as possible.
- The second part executes whether or not part 1 produced a new client connection and extended the
outboundFiles
list. - It iterates over every single
OutboundFile
in the list and attempts to send some or all available data out to the client.- If sendMoreData returns true, the loop advances on to the next client via
++iter
. - If sendMoreData returns false, the relevant OutboundFile is removed from
outboundFiles
before advancing. (Fortunately,erase
does precisely what we want, and it returns the iterator addressing the nextOutboundFile
in the list.)
- If sendMoreData returns true, the loop advances on to the next client via
- The second part executes whether or not part 1 produced a new client connection and extended the
list<OutboundFile> outboundFiles;
while (true) {
// part 1: presented and discussed on previous slide
// part 2: below
auto iter = outboundFiles.begin();
while (iter != outboundFiles.end()) {
if (iter->sendMoreData()) ++iter;
else iter = outboundFiles.erase(iter);
}
}
}
Lecture 25: Slow System Calls, Nonblocking I/O
- The code for
setAsNonblocking
is fairly low-level.- It relies on a function called
fcntl
to do surgery on the descriptor in the descriptor table. - That surgery does little more than toggle some 0 bit to a 1, as can be inferred from the last line of the three line implementation.
- It relies on a function called
- The code for
setAsNonblocking
and a few peer functions are presented below.
void setAsNonBlocking(int descriptor) {
fcntl(descriptor, F_SETFL, fcntl(descriptor, F_GETFL) | O_NONBLOCK); // preserve other set flags
}
void setAsBlocking(int descriptor) {
fcntl(descriptor, F_SETFL, fcntl(descriptor, F_GETFL) & ~O_NONBLOCK); // suppress blocking bit, preserve others
}
bool isNonBlocking(int descriptor) {
return !isBlocking(descriptor);
}
bool isBlocking(int descriptor) {
return (fcntl(descriptor, F_GETFL) & O_NONBLOCK) == 0;
}
Lecture 25: Slow System Calls, Nonblocking I/O
- We've been using the
OutboundFile
abstraction without understanding how it works behind the scenes.- We really should see the implementation (or at least part of it) so we have some sense how it works and can be implemented using nonblocking techniques.
- The full implementation includes lots of spaghetti code.
- In particular, true file descriptors and socket descriptors need to be treated differently in a few places—in particular, detecting when all data has been flushed out to the sink descriptor (which may be a local file, a console, or a remote client machine) isn't exactly pretty.
- However, my implementation is decomposed well enough that I think many of the methods—the ones that I'll show in lecture, anyway—are easy to follow and provide a clear narrative.
- At the very least, I'll convince you that the
OutboundFile
implementation is accessible to someone just finishing up CS110.
Lecture 25: Slow System Calls, Nonblocking I/O
- Here's is the condensed interface file for the OutboundFile class.
-
source
andsink
are nonblocking descriptors bound to the data source and recipient
-
buffer
is a reasonably sized character array that helps shovel bytes lifted from source viaread
calls over to thesink
viawrite
calls. -
numBytesAvailable
stores the number of meaningful characters inbuffer
. -
numBytesSent
tracks the portion ofbuffer
that's been pushed to the recipient. -
isSending
tracks whether all data has been pulled fromsource
and pushed tosink
.
class OutboundFile {
public:
OutboundFile();
void initialize(const std::string& source, int sink);
bool sendMoreData();
private:
int source, sink;
static const size_t kBufferSize = 128;
char buffer[kBufferSize];
size_t numBytesAvailable, numBytesSent;
bool isSending;
// private helper methods discussed later
};
Lecture 25: Slow System Calls, Nonblocking I/O
-
source
is a nonblocking file descriptor bound to some local file- Note that the source file is opened for reading (
O_RDONLY
), and the descriptor is configured to be nonblocking (O_NONBLOCK
) right from the start. - For reasons we've discussed, it's not super important that source be nonblocking, since it's bound to a local file.
- But in the spirit of a nonblocking example, it's fine to make it nonblocking anyway. We just shouldn't expect very many (if any) -1's to come back from our
read
calls.
- Note that the source file is opened for reading (
-
sink
is explicitly converted to be nonblocking, since it might be blocking, andsink
will very often be a socket descriptor that really should be nonblocking.
- The implementations of the constructor and initialize are straightforward:
OutboundFile::OutboundFile() : isSending(false) {}
void OutboundFile::initialize(const string& source, int sink) {
this->source = open(source.c_str(), O_RDONLY | O_NONBLOCK);
this->sink = sink;
setAsNonBlocking(this->sink);
numBytesAvailable = numBytesSent = 0;
isSending = true;
}
Lecture 25: Slow System Calls, Nonblocking I/O
- The first line decides if all data has been read from
source
and written tosink
, and if so, it returnstrue
unless it further confirms all of data written tosink
has arrived at final destination, in which case it returnsfalse
to state that syndication is complete.
- The first call to
dataReadyToBeSent
checks to see ifbuffer
houses data yet to be pushed out. If not, then it attempts toreadMoreData
. If after reading more data the buffer is still empty—that is, a single call toread
resulted in a-1/EWOULDBLOCK
pair, then we returntrue
as a statement that there's no data to be written, no need to try, but come back later to see if that changes. - The call to
writeMoreData
is an opportunity to push data out to sink.
- The implementation of sendMoreData is less straightforward:
bool OutboundFile::sendMoreData() {
if (!isSending) return !allDataFlushed();
if (!dataReadyToBeSent()) {
readMoreData();
if (!dataReadyToBeSent()) return true;
}
writeMoreData();
return true;
}
Lecture 25: Introduction to Non-blocking I/O
By Jerry Cain
Lecture 25: Introduction to Non-blocking I/O
- 977