Principles of Computer Systems
Spring 2019
Stanford University
Computer Science Department
Lecturer: Chris Gregg
createClientSocket
. For now, view it as a built-in that sets up a bidirectional pipe between a client and a server running on the specified host (e.g. myth64
) and bound to the specified port number (e.g. 12345).int main(int argc, char *argv[]) {
int clientSocket = createClientSocket("myth64.stanford.edu", 12345);
assert(client >= 0);
sockbuf sb(clientSocket);
iosockstream ss(&sb);
string timeline;
getline(ss, timeline);
cout << timeline << endl;
return 0;
}
wget
is a command line utility that, given its URL, downloads a single document (HTML document, image, video, etc.) and saves a copy of it to the current working directory.wget
's most basic functionality.main
and parseUrl
functions.parseUrl
dissects the supplied URL to surface the host and pathname components.static const string kProtocolPrefix = "http://";
static const string kDefaultPath = "/";
static pair<string, string> parseURL(string url) {
if (startsWith(url, kProtocolPrefix))
url = url.substr(kProtocolPrefix.size());
size_t found = url.find('/');
if (found == string::npos)
return make_pair(url, kDefaultPath);
string host = url.substr(0, found);
string path = url.substr(found);
return make_pair(host, path);
}
int main(int argc, char *argv[]) {
pullContent(parseURL(argv[1]));
return 0;
}
wget
(continued)
pullContent
, of course, needs to manage everything, including the networking.static const unsigned short kDefaultHTTPPort = 80;
static void pullContent(const pair<string, string>& components) {
int client = createClientSocket(components.first, kDefaultHTTPPort);
if (client == kClientSocketError) {
cerr << "Could not connect to host named \"" << components.first << "\"." << endl;
return;
}
sockbuf sb(client);
iosockstream ss(&sb);
issueRequest(ss, components.first, components.second);
skipHeader(ss);
savePayload(ss, getFileName(components.second));
}
createClientSocket
function for our time-client
. This time, we're connecting to real but arbitrary web servers that speak HTTP.issueRequest
, skipHeader
, and savePayload
subdivide the client-server conversation into manageable chunks.Emulation of wget
(continued)
issueRequest
, which generates the smallest legitimate HTTP request possible and sends it over to the server.
static void issueRequest(iosockstream& ss, const string& host, const string& path) {
ss << "GET " << path << " HTTP/1.0\r\n";
ss << "Host: " << host << "\r\n";
ss << "\r\n";
ss.flush();
}
'\r'
following by '\n'
.flush
is necessary to ensure all character data is pressed over the wire and consumable at the other end.flush
, the client transitions from supply to ingest mode. Remember, the iosockstream
is read/write, because the socket descriptor backing it is bidirectional.skipHeader
reads through and discards all of the HTTP response header lines until it encounters either a blank line or one that contains nothing other than a '\r'.
The blank line is, indeed, supposed to be "\r\n"
, but some servers—often hand-rolled ones—are sloppy, so we treat the '\r'
as optional. Recall that getline chews up the '\n'
, but it won't chew up the '\r'
.static void skipHeader(iosockstream& ss) {
string line;
do {
getline(ss, line);
} while (!line.empty() && line != "\r");
}
wget
we're imitating—would ingest all of the lines of the response header into a data structure and allow it to influence how it treats payload.static string getFileName(const string& path) {
if (path.empty() || path[path.size() - 1] == '/') return "index.html";
size_t found = path.rfind('/');
return path.substr(found + 1);
}
static void savePayload(iosockstream& ss, const string& filename) {
ofstream output(filename, ios::binary); // don't assume it's text
size_t totalBytes = 0;
while (!ss.fail()) {
char buffer[2014] = {'\0'};
ss.read(buffer, sizeof(buffer));
totalBytes += ss.gcount();
output.write(buffer, ss.gcount());
}
cout << "Total number of bytes fetched: " << totalBytes << endl;
}
EOF
, and we write everything we read.scrabbleword-finder
. The source code for this executable—completely unaware it'll be used in a larger networked application—can be found right here.scrabble-word-finder
is implemented using only CS106B techniques—standard file I/O and procedural recursion with simple pruning.cgregg@myth61:$ ./scrabble-word-finder lexical
ace
// many lines omitted for brevity
lei
lex
lexica
lexical
li
lice
lie
lilac
xi
cgregg@myth61:$
cgregg@myth61:$ ./scrabble-word-finder network
en
// many lines omitted for brevity
wonk
wont
wore
work
worn
wort
wot
wren
wrote
cgregg@myth61:$
scrabble-wordfinder
is capable of.
myth54:13133
, we expect http://myth54:13133/lexical
and http://myth54:13133/network
to generate the following payloads:{
time: 0.223399,
cached: false,
possibilities: [
'ace',
// several words omitted
'lei',
'lex',
'lexica',
'lexical',
'li',
'lice',
'lie',
'lilac',
'xi'
]
}
{
time: 0.223399,
cached: false,
possibilities: [
'en',
// several words omitted
'wonk',
'wont',
'wore',
'work',
'worn',
'wort',
'wot',
'wren',
'wrote'
]
}
scrabble-word-finder.cc
to build the core of scrabble-word-finder-server.cc
.scrabble-word-finder
already outputs the primary content we need for our payload. We're packaging the payload as JSON instead of plain text, but we can still tap scrabble-word-finder
to generate the collection of formable words.subprocess_t
type and subprocess
function from Assignment 3.struct subprocess_t {
pid_t pid;
int supplyfd;
int ingestfd;
};
subprocess_t subprocess(char *argv[],
bool supplyChildInput, bool ingestChildOutput) throw (SubprocessException);
main
function implementing our server:int main(int argc, char *argv[]) {
unsigned short port = extractPort(argv[1]);
int server = createServerSocket(port);
cout << "Server listening on port " << port << "." << endl;
ThreadPool pool(16);
map<string, vector<string>> cache;
mutex cacheLock;
while (true) {
struct sockaddr_in address;
// used to surface IP address of client
socklen_t size = sizeof(address); // also used to surface client IP address
bzero(&address, size);
int client = accept(server, (struct sockaddr *) &address, &size);
char str[INET_ADDRSTRLEN];
cout << "Received a connection request from "
<< inet_ntop(AF_INET, &address.sin_addr, str, INET_ADDRSTRLEN) << "." << endl;
pool.schedule([client, &cache, &cacheLock] {
publishScrabbleWords(client, cache, cacheLock);
});
}
return 0;
}
accept
are used to surface the IP address of the client.address
, size
, and the inet_ntop
function until next week, when we'll talk more about them. Right now, it's a neat-to-see!ThreadPool
of size 16.publishScrabbleWords
will rely on our subprocess
function to marshal plain text output of scrabble-word-finder into JSON and publish that JSON as the payload of the HTTP response.publishScrabbleWords
and some of its helper functions.publishScrabbleWords
:static void publishScrabbleWords(int client, map<string, vector<string>>& cache,
mutex& cacheLock) {
sockbuf sb(client);
iosockstream ss(&sb);
string letters = getLetters(ss);
sort(letters.begin(), letters.end());
skipHeaders(ss);
struct timeval start;
gettimeofday(&start, NULL); // start the clock
cacheLock.lock();
auto found = cache.find(letters);
cacheLock.unlock(); // release lock immediately, iterator won't be invalidated by competing find calls
bool cached = found != cache.end();
vector<string> formableWords;
if (cached) {
formableWords = found->second;
} else {
const char *command[] = {"./scrabble-word-finder", letters.c_str(), NULL};
subprocess_t sp = subprocess(const_cast<char **>(command), false, true);
pullFormableWords(formableWords, sp.ingestfd);
waitpid(sp.pid, NULL, 0);
lock_guard<mutex> lg(cacheLock);
cache[letters] = formableWords;
}
struct timeval end, duration;
gettimeofday(&end, NULL); // stop the clock, server-computation of formableWords is complete
timersub(&end, &start, &duration);
double time = duration.tv_sec + duration.tv_usec/1000000.0;
ostringstream payload;
constructPayload(formableWords, cached, time, payload);
sendResponse(ss, payload.str());
}
pullFormableWords
and sendResponse
helper functions.static void pullFormableWords(vector<string>& formableWords, int ingestfd) {
stdio_filebuf<char> inbuf(ingestfd, ios::in);
istream is(&inbuf);
while (true) {
string word;
getline(is, word);
if (is.fail()) break;
formableWords.push_back(word);
}
}
static void sendResponse(iosockstream& ss, const string& payload) {
ss << "HTTP/1.1 200 OK\r\n";
ss << "Content-Type: text/javascript; charset=UTF-8\r\n";
ss << "Content-Length: " << payload.size() << "\r\n";
ss << "\r\n";
ss << payload << flush;
}
getLetters
and the constructPayload
helper functions. I omit the implementation of skipHeaders
—you saw it with web-get
—and constructJSONArray
, which you're welcome to view right here.static string getLetters(iosockstream& ss) {
string method, path, protocol;
ss >> method >> path >> protocol;
string rest;
getline(ss, rest);
size_t pos = path.rfind("/");
return pos == string::npos ? path : path.substr(pos + 1);
}
static void constructPayload(const vector<string>& formableWords, bool cached,
double time, ostringstream& payload) {
payload << "{" << endl;
payload << " time: " << time << "," << endl;
payload << " cached: " << boolalpha << cached << "," << endl;
payload << " possibilities: " << constructJSONArray(formableWords, 2) << endl;
payload << "}" << endl;
}
scrabble-word-finder-server
provided a single API call that resembles the types of API calls afforded by Google, Twitter, or Facebook to access search, tweet, or friend-graph data.