• The first assignment is meant to get you up to speed on the coding you need to be able to do for the class. It is a mix of CS106B and CS107 ideas. 
• The program you will write will search through the (in some cases) enormous Amazon Review database, which we have packaged into two files: the reviews themselves, and an index to all the words in all the reviews.

• The first assignment is meant to get you up to speed on the coding you need to be able to do for the class. It is a mix of CS106B and CS107 ideas. 
• The program you will write will search through the (in some cases) enormous Amazon Review database, which we have packaged into two files: the reviews themselves, and an index to all the words in all the reviews.

• The two files for each database have been created in a format that allows fast binary searching. The databaseFile is built on a data structure that allows O(1) access to a particular review, by index, and the keywordIndexFile is built on a data structure that allows binary searching for words that are found in the reviews. This is where the CS107 stuff comes in: you need to understand the file formats exactly and you need to use pointer arithmetic to parse them.
• You will also use C++ standard template library (STL) classes to do the binary searching in these files. Specifically, you will use the lower_bound function from the STL. The function is a bit subtle -- you need to take some time to understand how it works. For example, it takes an iterator, which in our case is just a pointer to the data. Also, when searching, it returns an "Iterator pointing to the first element that is not less than value, or last if no such element is found." (see the link above for details).
• Once you have worked out how to search for keywords (this is the CS106B part of the assignment), and once you have compiled a data structure that contains the indexes for all reviews that match the user's search query, you need to sort the reviews by a client-provided search function, so the user can get the reviews in a particular order.

• You might be wondering how 185 students are all going to be reading in gigantic database files into shared Myth machines without running out of memory. It turns out that you don't read the files into memory -- we set up what is called a memory map to the files on the disk. This is a CS 110 idea, actually, so let's spend a minute discussing it.
• When a file is memory mapped, the operating system keeps track of requests for particular parts of the file, and only loads the data that is requested into main memory (note: main memory is RAM. Each myth machine has 32GB of RAM, which seems like a lot, but can become scarce with many users).
• Here is where it gets really cool: if two users on the same machine happen to memory map the same file, and both ask for the same portion of data from that file, the OS recognizes this, and simply keeps a single copy in memory. Each user has a pointer that maps to the region of memory where the data is stored, and when they ask for a particular part of the file (using pointer arithmetic), if that particular part of the file is already loaded into memory, then the OS can translate their pointer to the physical memory location and return the data that is there, without needing to go back to the disk at all. It is both fast and saves memory --              in other words,  a great idea!

• To go back to the lower_bound function for a moment: part of the assignment says, "Remember, you must use the lower_bound function and a C++ lambda function to perform a binary search to find keywords."
• What is this about a "C++ lambda"? This is likely a new concept for you, so let's discuss it.
• A lambda function is a function that is usually placed inline as a parameter to another function, which expects the parameter to itself be a function (I N C E P T I O N)
• Before we talk about lambdas specifically, let's back up a bit and recall what it means to pass around function pointers (CS107 stuff)
  • Function pointers provide flexibility. Recall the qsort function:

• The last parameter is a function pointer that defines the comparison function qsort will use when it sorts an array.
• The caller of the qsort function passes in the function pointer, and qsort itself simply calls it, expecting an int return value. qsort does not care about the details of how the comparison is done, it just relies on it to provide a legitimate result.

• Let's look at an example program: (full program here)

• We've created two functions, add and sub, that get called by modifyVec.
• The function<int(int, int)op parameter is a C++ way of creating a function pointer.
• Note on lines 18 and 19, the add and sub functions do not get called immediately -- they get called when modifyVec gets around to calling them. 

• With a lambda function, we can replace the add and sub functions with an inline function (full program here).

• Lines 16-18 and 19-21 are where the magic happens.
• A lambda function has the following signature:

• We will talk about captures in a moment, but for now, see that the params and the body comprise a similar form to our original functions for add and sub.

• So a lambda function is just an inline function. But, it can be more than that. We may want to allow the function to utilize variables from the scope where the function is being called. Let's say we changed modifyVec from this:

To this:

• In other words, now we want the function that calls modifyVec to also handle the value we are updating by. This would be difficult to accomplish with a regular function pointer.
• But, with a lambda function, it is possible.

• Here is our new version, with a modified lambda function:

• In this version, we have captured the variable val, using the bracket notation. This allows the lambda function, when it is called (remember, it isn't called immediately) to use val.
• There are multiple ways to capture variables -- often, we want to capture them by reference. If we wanted to capture val as a reference, we would call it as follows:

• Some more comments on lambda functions:
  • Lambda functions are critical when we have C++ classes, too -- without lambdas, you can't call class functions from a non-class function (this is a key reason why it is necessary for the lower_bound function for assignment 1!)
  • If you want to capture all class variables, you can use [this] as a capture clause.
  • You can capture multiple variables in a capture clause, e.g., [this, val, &myVec] 
  • Basically, any in-scope variable you want to use in the lambda function must be captured in the capture clause.
  • We will use lambda functions a great deal when we get to threading, so learn it well on this assignment.

• The implementation of copy (designed to mimic the behavior of cp) illustrates how to use open, read, write, and close. It also introduces the notion of a file descriptor.
  • man pages exist for all of these functions (e.g. man 2 open, man 2 read, etc.)
  • Full implementation of our own copy, with exhaustive error checking, is right here.
  • Simplified implementation, sans error checking, is on the next slide.

• We have already discussed two file system API calls: open and umask. We are going to look at other low-level operations that allow programmers to interaction with the file system. We will focus here on the direct system calls, but when writing production code (i.e., for a job), you will often use indirect methods, such as FILE *, ifstreams, and ofstreams.
• Requests to open a file, read from a file, extend the heap, etc., all eventually go through system calls, which are the only functions that can be trusted to interact with the system on your behalf. The operating system kernel actually runs the code for a system call, completely isolating the system-level interaction from your (potentially harmful) program.

• The read system call will block until the requested number of bytes have been read. If the return value is 0, there are no more bytes to read (e.g., the file has reached the end, or been closed).
• If write returns a value less than count, it means that the system couldn't write all the bytes at once. This is why the while loop is necessary, and the reason for keeping track of bytesWritten and bytesRead.
• You should close files when you are done using them, although they will get closed by the OS when your program ends. We will use valgrind to check if your files are being closed.

• The file descriptor abstraction provides direct, low level access to a stream of data without the fuss of data structures or objects. It certainly can't be slower, and depending on what you're doing, it may even be faster.
• FILE pointers and C++ iostreams work well when you know you're interacting with standard output, standard input, and local files.
  • They are less useful when the stream of bytes is associated with a network connection.
  • FILE pointers and C++ iostreams assume they can rewind and move the file pointer back and forth freely, but that's not the case with file descriptors associated with network connections.
• File descriptors, however, work with read and write and little else used in this course.
• C FILE pointers and C++ streams, on the other hand, provide automatic buffering and more elaborate formatting options.

• Overview of tee
  • The tee program that ships with Linux copies everything from standard input to standard output, making zero or more extra copies in the named files supplied as user program arguments. For example, if the file contains 27 bytes—the 26 letters of the English alphabet followed by a newline character—then the following would print the alphabet to standard output and to three files named one.txt, two.txt, and three.txt.

• If the file vowels.txt contains the five vowels and the newline character, and tee is invoked as follows, one.txt would be rewritten to contain only the English vowels.

• Full implementation of our own t executable, with error checking, is right here.
• Implementation replicates much of what copy.cdoes, but it illustrates how you can use low-level I/O to manage many sessions with multiple files. The implementation inlined across the next two slides omit error checking.

• Features:
  • Note that argc incidentally provides a count on the number of descriptors that write to. That's why we declare an integer array (or rather, a file descriptor array) of length argc.
  • STDIN_FILENO is a built-in constant for the number 0, which is the descriptor normally attached to standard input. STDOUT_FILENO is a constant for the number 1, which is the default descriptor bound to standard output.
  • I assume all system calls succeed. I'm not being lazy, I promise. I'm just trying to keep the examples as clear and compact as possible. The official copies of the working programs up on the myth machines include real error checking.