Insight UBC: Deliverable 1

Goals and specs

1. Add and remove a dataset containing information about UBC courses

Goals and specs

1. Add and remove a dataset containing information about UBC courses

Courses.zip

addDataset(...)

Parse it

Keep data structure in a variable

Save it to disk

Caching

No need to parse again the same dataset.

No need to go to disk on every request

Goals and specs

2. Answer queries about UBC courses.

Goals and specs

2. Answer queries about UBC courses.

Examples:

"what departments have courses with averages higher than 90%?"

"What courses in CPSC have a high fail rate?"

Goals and specs

2. Answer queries about UBC courses.

{
   "WHERE":{
      "GT":{
         "courses_avg":97
      }
   },
   "OPTIONS":{
      "COLUMNS":[
         "courses_dept",
         "courses_avg"
      ],
      "ORDER":"courses_avg"
   }
}

Goals and specs

2. Answer queries about UBC courses.

{
   "WHERE":{
      "GT":{
         "courses_avg":97
      }
   },
   "OPTIONS":{
      "COLUMNS":[
         "courses_dept",
         "courses_avg"
      ],
      "ORDER":"courses_avg"
   }
}

{ result:
  [ { courses_dept: 'epse', courses_avg: 97.09 },
    { courses_dept: 'math', courses_avg: 97.09 },
    { courses_dept: 'math', courses_avg: 97.09 },
    { courses_dept: 'epse', courses_avg: 97.09 },
    { courses_dept: 'math', courses_avg: 97.25 },
    { courses_dept: 'math', courses_avg: 97.25 },
    { courses_dept: 'epse', courses_avg: 97.29 },
    { courses_dept: 'epse', courses_avg: 97.29 },
    { courses_dept: 'nurs', courses_avg: 97.33 },
    { courses_dept: 'nurs', courses_avg: 97.33 } ] }

Goals and specs

2. Answer queries about UBC courses. Queries follow the following EBNF:

QUERY ::='{'BODY ', ' OPTIONS '}'

BODY ::= 'WHERE:{'  FILTER '}'
OPTIONS ::= 'OPTIONS:{' COLUMNS ', ' ('ORDER:' key )?'}'

FILTER ::= (LOGICCOMPARISON | MCOMPARISON | SCOMPARISON | NEGATION)

LOGICCOMPARISON ::= LOGIC ':[{' FILTER ('}, {' FILTER )* '}]'  
MCOMPARISON ::= MCOMPARATOR ':{' key ':' number '}'  
SCOMPARISON ::= 'IS:{' key ':' [*]? inputstring [*]? '}'
NEGATION ::= 'NOT :{' FILTER '}'

LOGIC ::= 'AND' | 'OR' 
MCOMPARATOR ::= 'LT' | 'GT' | 'EQ' 

COLUMNS ::= 'COLUMNS:[' (key ',')* key ']' 

key ::= string '_' string
inputstring ::= [^*]+

Activity #1

We previously discussed about software engineering process and methodologies.

We learned about waterfall, spiral, and the many variations of scrum: XP, kanban, and more.

What would be the most adequate methodology to build Insight UBC? How would you approach tests, documentation, design, and development?

Take a few minutes to discuss this with your classmates!

Activity #2

Now that we have a better idea about what methodologies and approaches to use, discuss with your classmates how you would break the deliverable 1 tasks into user stories.

Project structure: API

export interface IInsightFacade {
    
    addDataset(id: string,
               content: string,
               kind: InsightDatasetKind):
                   Promise<InsightResponse>;

    removeDataset(id: string): Promise<InsightResponse>;

    performQuery(query: any): Promise<InsightResponse>;

    listDatasets(): Promise<InsightResponse>;
}

Project structure: API

export interface InsightResponse {
    code: number;
    body: InsightResponseSuccessBody | InsightResponseErrorBody;
}

export interface InsightResponseSuccessBody {
    result: any[] | string;
}

export interface InsightResponseErrorBody {
    error: string;
}

export enum InsightDatasetKind {
    Courses = "courses",
    Rooms = "rooms"
}

export interface InsightDataset {
    id: string;
    kind: InsightDatasetKind;
    numRows: number;
}

Project structure: Testing

it("Test addDataset 204", function () {
        let readStream = fs.readFileSync("$PATH_TO_ZIP_FILE").toString('base64');
        return facade.addDataset("courses", readStream, InsightDatasetKind.Courses)
            .then(function (res: InsightResponse) {
                Log.writeResponse(res);
                expect(res.code).to.equal(204);
        }).catch(function (err: InsightResponse) {
            Log.writeResponse(err);
            Log.test("Should not have reached this point: " + JSON.stringify(err));
            expect.fail();
        });
    });

Project structure: Testing

it("Test addDataset 204", function () {
        let readStream = fs.readFileSync("$PATH_TO_ZIP_FILE").toString('base64');
        return facade.addDataset("courses", readStream, InsightDatasetKind.Courses)
            .then(function (res: InsightResponse) {
                Log.writeResponse(res);
                expect(res.code).to.equal(204);
        }).catch(function (err: InsightResponse) {
            Log.writeResponse(err);
            Log.test("Should not have reached this point: " + JSON.stringify(err));
            expect.fail();
        });
    });

base64 string represents the actual zip file

Project structure: Testing

it("Test addDataset 204", function () {
        let readStream = fs.readFileSync("$PATH_TO_ZIP_FILE").toString('base64');
        return facade.addDataset("courses", readStream, InsightDatasetKind.Courses)
            .then(function (res: InsightResponse) {
                Log.writeResponse(res);
                expect(res.code).to.equal(204);
        }).catch(function (err: InsightResponse) {
            Log.writeResponse(err);
            Log.test("Should not have reached this point: " + JSON.stringify(err));
            expect.fail();
        });
    });

In this unit test we are testing the 204 case, you must test the other cases as well!

Project structure: Testing

it("Test async code", function () {
        return facade.asyncFunction(...).then(function (res: InsightResponse) {
                expect(res.code).to.equal(<CODE_NUMBER>);
        }).catch(function (err: InsightResponse) {
            expect.fail();
        });
    });

Every method from the API returns a promise, it means that every method from the API is an async method. Here's the default template for testing a async function:

Project structure: Testing

it("Test async code", function () {
        return facade.asyncFunction(...).then(function (res: InsightResponse) {
                expect(res.code).to.equal(<CODE_NUMBER>);
        }).catch(function (err: InsightResponse) {
            expect.fail();
        });
    });

More on JS promises and async programming next week

Every method from the API returns a promise, it means that every method from the API is an async method. Here's the default template for testing a async function:

Git and GitHub workflow

Your local repo/code

Your teammate's local repo/code

Your team GitHub repository

Code in version 0.0

Git and GitHub workflow

Your local repo/code

Your teammate's local repo/code

Your team GitHub repository

Code in version 0.1

Code in version 0.0

You've changed the code!

Git and GitHub workflow

Your local repo/code

Your teammate's local repo/code

Your team GitHub repository

Code in version 0.1

Code in version 0.0

git add <PATH_TO_MODIFIED_FILES>

git commit -m "New commit"

Git and GitHub workflow

Your local repo/code

Your teammate's local repo/code

Your team GitHub repository

Code in version 0.1

Code in version 0.0

git push origin master

Git and GitHub workflow

Your local repo/code

Your teammate's local repo/code

Your team GitHub repository

Code in version 0.1

git pull origin master

Git and GitHub workflow: Branches

git checkout -b branch_name

Creating and moving to a new local branch

Pushing the new local branch to GitHub

git add <path_to_modified_files>

git commit -m "Message"

git push origin branch_name

Git and GitHub workflow: Branches

git checkout -b branch_name

Creating and moving to a new local branch

Pushing the new local branch to GitHub

git add <path_to_modified_files>

git commit -m "Message"

git push origin branch_name

Merging to master branch

git checkout master
git merge branch_name

General tips and advice

Start early. Fail early.
Have enough time to fix things.

#1

#2

TDD and testing can help you break down the high level goals -- addDataset(), performQuery() -- into smaller, easily testable tasks: saveDatasetToDisk(), validateQuery().

Write these tests first, make sure they fail, then write the code to make them pass

Insight UBC: Deliverable 1

Goals and specs

Goals and specs

Goals and specs

Goals and specs

Goals and specs

Goals and specs

Goals and specs

Activity #1

Activity #2

Project structure: API

Project structure: API

Project structure: Testing

Project structure: Testing

Project structure: Testing

Project structure: Testing

Project structure: Testing

Git and GitHub workflow

Git and GitHub workflow

Git and GitHub workflow

Git and GitHub workflow

Git and GitHub workflow

Git and GitHub workflow: Branches

Git and GitHub workflow: Branches

Git and GitHub workflow: Branches

General tips and advice

Start early. Fail early. Have enough time to fix things.

#1

#2

Use TDD

Start early, please.

#3

#4 Write tests, mostly integration

Start early. I'm warning you.

#5

Avoid writing too many lines inside promises.then() or promises.catch(). This will save you some debugging time.

#6

#7

You: "I'm passing everything locally but failing when I call the bot"

Your code is wrong. Trust me.

#8

#9

When you're stuck:

- Get away from the computer

- Talk a walk

- Drink some coffee

- Get some sleep

- Go play some League (or whatever the cool kids are playing these days)

- Do not panic

#10

Keep communicating with your teammate, always. Communication is key.

Resources

Start early. Fail early.
Have enough time to fix things.