Understanding Modules

Hour 02

What are NodeJS Modules?

• Brings the concept of a 'Standard Library' to NodeJS
• Allows developers to extend NodeJS through custom modules
• Implements the CommonJS modules standard
• A module is defined in a single file (one to one correspondence with file)
• A module can export public functions which can be used by other modules
• Variables local to the module are private to the module

CommonJS

• Is a standard specification for modualrizing JavaScript
• CommonJS does not provide any libraries of its own
• Specifies require() and an exports object which work in conjunction to provide the module 
• NodeJS standard library and all 3rd Party libraries use CommonJS standard to modularize the code 

Consuming Modules

To consume an existing module use the require function

// Require a code module or a module installed by NPM
var module = require('module-name');


// Require a module relative to current path
var my-module = require('./my-module');

Defining Modules

To define a module, use exports global variable

exports

vs

module.exports

/**
* File: my-math.js
* Style 1 - Export single function
*/

var square = function (n) { return n*n;};

exports.square = square

///////////////////////////////////////////////////////

/**
* File: my-math-lib.js
* Style 2 - Set of functions
*/

var square = function (n) { return n*n;};
var cube = function(n) {return square(n) * n;};

exports.square = square;
exports.cube = cube;

//////////////////////////////////////////////////////

/**
* File: my-math-lib-module.js
* Style 3 - Set of functions in module style
*/

var square = function (n) { return n*n;};
var cube = function(n) {return square(n) * n;};

module.exports = {
    square:square;
    cube: cube;
};

///////////////////////////////////////////////////////

/**
* File employee.js
* Style 4 : Object Constructor
*/

module.exports = function (firstname, lastname, department) {
    this.firstname = firstname;
    this.lastname = lastname;
    this.department = department;

    this.print = function() {
        return firstname + ' ' + lastname + ' - ' + department;
    }
};

/**
* file employee-service.js
*/
var Employee = require ('../models/employee');

var employee1 = new Employee('Jack', 'Peters', 'Mechanical');
console.log(employee1.print());

Resolving Modules

Core Modules
    binary modules - from lib folder
    Preferentially loaded over user defined names

File Modules
    Prefixed with /, ./ or ../
    If exact name is not found appends .js, .node or .json and does the search. 
    Throws exception if module is not found

node_modules folder
    If not available in native modules and no prefix such as /, ./, ../ etc

Folders
    Loads from main entry in package.json
    If package.json is not available in the folder, 
    tries with /index appended to the end of the require string [.js,  .node, .json]

http://fredkschott.com/post/2014/06/require-and-the-module-system/

node_modules folder

• automatically created by npm when a module is installed
• node_modules are not checked into version control system
• They maintain deeply nested folder structure to manage each package dependency 
• Due to deep nesting, it does not work very well Windows

Creating and Publishing above sea level modules

• https://quickleft.com/blog/creating-and-publishing-a-node-js-module/
• http://www.hacksparrow.com/how-to-write-node-js-modules.html

Modules Best Practices

• http://www.mircozeiss.com/node-js-require-s-best-practices/
• https://github.com/mattdesl/module-best-practices

Introduction

Hour 01

What is NodeJS?

• NodeJS is an open-source, cross-platform JavaScript runtime environment based on Chrome's V8 JavaScript Engine
• NodeJS applications are written in JavaScript Language and can run on many different environments supported by the NodeJS runtime (OS X, Windows,  Linux, Free BSD & many more)

Why NodeJS

• Speaks the lingua franca of the web!
• Single language for the server and front-end
• Dynamically Typed
• Speed & non blocking I/O
• Massive Ecosystem (npm modules ~198k packages)
• Vibrant Community
• Cross Platform (FreeBSD, Linux, OS X, Windows)
• Seriously Productive
• Builtin support for realtime and streaming
• Fun, Easy and Enjoyable.

http://www.toptal.com/nodejs/why-the-hell-would-i-use-node-js

Where is node used today?

• Netflix
• eBay
• New York Times
• PayPal
• Medium
• Linkedin
• Uber

NodeJs Use Cases

• Rest API's
• Streaming & Realtime Applications
• Command Line Tools

When to avoid?

• CPU Intensive Applications
• RDBMS Data Stores with lots of de-normalized data
• Web applications with lots of server side rendering

Node Bindings (Socket Library, HTTP etc) 

Node Standard Library

Event Loop

(Libev)

V8

Thread Pool

(Libeio)

DNS

(C-ares)

Crypto

(OpenSSL)

JavaScript

C++

• Single Threaded
• Event Loop
• Non Blocking IO
• Excellent Support for HTTP

Installing NodeJS & NPM - USING NVM

# https://github.com/creationix/nvm

# For Linux & Mac OS
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.29.0/install.sh | bash
or Wget:
wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.29.0/install.sh | bash




# For Windows Machine
nvmw - https://github.com/hakobera/nvmw
nvm-windows - https://github.com/coreybutler/nvm-windows (has a windows installer)

NodeJS Development Environment

• Get a good shell (if windows!)
• Install Git (Git Bash)
• A decent text editor (Sublime Text, Notepad++)
• Install jshint - {asl install -g jshint}
• Master the shell!

All About asl

• NPM stands for Node Package Manager
• Written by Isaac Schlueter
• Manages 3rd party package installation
• From 0.6.0 version of nodejs comes preinstalled
• Used for initializing the nodejs applicaitons
• Keeping track of out dated dependencies

NPM Commands

# npm commands

npm --version                            # displays the version of npm
npm init                                 # inits a new module with a package.json file
npm install <module>                     # installs the module and all its dependency
npm install -g                           # globally installs the module
npm install [-g] <module>  --save        # saves the module to the package.json dependencies section
npm install [-g] <module>  --save-dev    # saves the module to the package.json dev-dependencies section
npm ls -g                                # list global modules
npm ls <module>                          # list by a module bane
npm rm <module> # remove (alias)         # remove an installed module
npm outdated                             # list outdated modules
npm view                                 # view details of an installed module
npm prune                                # removes extraneous (unwanted) packages
npm search                               # search public packages from npm registry
npm update                               # updates all dependent modules based changes to package.json

npm help                                 # help on npm

Package.json

# Package.json, is present in the root directory of the nodejs applicaiton. It defines the property of the application / package.

# An npm init command generates a package.json file by interactively asking a set of questions

# Attributes of package.json

name             - name of the package
version          - version of the package
description      - description of the package
homepage         - homepage of the package
author           - author of the package
contributors     - name of the contributors to the package
dependencies     - list of dependencies. dependencies mentioned here in are installed to node_module folder of the package.
devDependencies  - 
repository       - repository type and url of the package
main             - entry point of the package
keywords         - keywords
scripts          - scripts to be run during various times of the project's lifecycle (example, start, test, pack etc)
config           - set configuration parameters used in package scripts

REFER: http://browsenpm.org/package.json

jshint

# JSHint is a static code analysis tool for JavaScript


# installation

    npm install -g jshint || npm i -g jshint

# or install it as your dev dependency

    npm install jshint --save-dev


# configuration (customize jshint)

    1)  --config <file>
    2)  .jshintrc
    3)  jshintConfig 
 

Example - Basic

mkdir hello-node && cd $_
npm init

# answer the questions asked by npm init
# touch index.js

    "use strict";

    console.log("Hello, world from nodejs");

# run the program
    node index.js

# Add the start up script
"scripts": {

    start: "node index.js"
}

# Run using scripts:start attribute

npm start

Example - Advanced

# touch server.js

"use strict";

var http = require("http");
var port = process.env.PORT || 8888;

http.createServer( function (req, res) {
	res.writeHead(200, {"Content-Type": "text/plain"});
	res.write("Hello, World from nodejs");
	res.end();
}).listen(port);

console.log("HTTP Server listening to port :" + port);

Example - Non Blocking I/O

# touch nonBlockingDemo.js

"use strict";

var fs = require('fs');

fs.readFile('read-nb.txt', 'utf8', function(err, contents) {

	console.log(contents);
});

console.log('Hello there, the file is being read asynchronously');

Extras - Closures

# touch no-closure.js

"use strict";


function sayHello (message) {
    console.log(message);
}


function createMessage(name) {
    return "Hi " 
    + name 
    + " welcome to Node in 10 hours training!";
}

var message = createMessage('Jack');

sayHello(message);

# touch closure.js

"use strict";

// Define the closure
function HelloMaker (name) {

    var message = 'Hi ' 
    + name 
    + ' welcome to Node in 10 hours training!';

    return function sayHello() {
        console.log(message);
    }
}

// Create the closure
var helloJack = HelloMaker('Jack');

// Use the closure
helloJack();

Event Loop & Event Emitter

Hour 03

What is an Event Loop?

• A software pattern
• Facilitates async I/O
• In NodeJS Event Loop is core concept for handling high throughput
• Though event loop helps in asynchronous programming the loop itself is synchronous
• An event loop in NodeJS continues to run until the last callback is completed

Event Loop in NodeJS

Source:

https://nodesource.com/blog/understanding-the-nodejs-event-loop

• Depends on multiple libraries (libev)
• Used OS efficiently to hold connections & resources
•  

Traditional Vs Event Driven I/O

// Traditional Model
posts = db.query("SELECT * from posts WHERE author = '%john%'");

formattedPosts = formatPosts(post);

sendPosts(formattedPosts);

// create a callback
var postHandler = function(err, posts) {
    formatPosts(posts, function (err, formattedPosts) {
        sendPosts(formattedPosts, function(err) {
            //Handle Error
        };
    });
};

// var formatPosts = function (posts, callback) {};
// var sendPosts = function (formattedPosts, callback) {};

// Query the database with a callback
db.query("SELECT * FROM posts WHERE author ='%jack%'", postHandler);

Blocking the Event Loop

// Guess what happens here?

var opened = false;

setTimeout(function() {
    opened = true;
}, 1000);

while (!opened) {
    // wait till it is opened
}

console.log('Opened now!');

Event Emitter

• Many NodeJS objects emit event
• All objects that emit events are instances of events.EventEmitter
• Examples
  • TCP Server emit 'connect' every time a client connects
  • ReadStream can emit 'data' each time data is available for reading

Event Emitter - Methods

There are few more

https://nodejs.org/api/events.html#events_class_events_eventemitter

Example - Add Listener

"use strict";

var rl = require('readline').createInterface({
  input: require('fs').createReadStream('readme.txt')
});

rl.on('line', function (line) {
    console.log('Line from file:', line);
});

rl.on('close', function () {
    console.log('Closed::The end');
});

Example - Once

// Manual Method

var events = require('events'),
                emitter = new events.EventEmitter();
function listener() {
        console.log('one Timer');
        emitter.removeListener('oneTimer', listener);
}
emitter.on('oneTimer', listener);
emitter.emit('oneTimer');
emitter.emit('oneTimer');


// Using Once

var events = require('events'),
                emitter = new events.EventEmitter();
/* Use Once */
emitter.once('onceOnly', function() {
        console.log('one Only');
});
emitter.emit('onceOnly');
emitter.emit('onceOnly');

Example - Create EventEmitter

var EventEmitter = require('events').EventEmitter;
var util = require('util');

var Person = function (firstname, lastname) {
    this.firstname = firstname;
    this.lastname = lastname;
};

util.inherits(Person, EventEmitter);

// Create an method that emits an event
Person.prototype.sayHello = function () {
    this.emit('greet', {firstname: this.firstname, lastname: this.lastname});
};


// Create an instance of person and say hello

var p = new Person('Peter', 'Parker');

p.on('greet', function (data){
    console.log('Got a greet event from ' + data.firstname); 
});

Example - Emit Frequently

"use strict";

var util = require('util'),

EventEmitter = require('events').EventEmitter;

var Ticker = function(limit){ 
    var self = this; 
    
    this.limit = limit || 10;
    this.counter = 0;

    setInterval(function() {
        self.emit('tick');
        self.counter += 1;

        if(self.counter === self.limit) {
            console.log('limit reached -- removing event listener');
            self.emit('limit');
        }

    }, 1000);
};

util.inherits(Ticker,EventEmitter)

var ticker = new Ticker(); 

var tickListener = function(){
  console.log('TICK');
};

ticker.on('tick',tickListener);

ticker.on('limit', function(){
    ticker.removeListener('tick', tickListener);
    ticker.removeAllListeners('limit');
})

// As an exercise, remove the event listener after 100 ticks

Networking

Hour 05

Networking in NodeJS

• TCP
• UDP
• HTTP
• TLS
• WebSockets

TCP Server & Client

// TCP Server 
var net = require('net');

net.createServer(function(socket){

    // new connection4
    socket.on('data', function(data) {
        // got data
    });

    socket.on('end', function(data) {
        // connection closed
    });

    socket.write('Some string');

}).listen(4001);

// TCP Client
var net=require('net');
var port=80;
var host='www.google.com'; 
// host is optional, if ommitted defaults to localhost
var conn = net.createConnection(port,host);

conn.on('data',function(data){ 
    console.log('some data has arrived')
});

conn.on('close',function(){ 
    console.log('connection closed');
});

conn.write("Hello, World!");
conn.close();

UDP Server & Client

// UDP Server 
var dgram = require('dgram');

var server=dgram.createSocket('udp4'); // can be 'udp6' for ipv6

server.on('message',function(message,rinfo){
  console.log('server got message: ' + message + ' from ' + rinfo.address + ':' + rinfo.port);
});

server.on('listening',function(){
    var address = server.address();
    console.log('server listening on ' + address.address + ':' + address.port);
});

server.bind(4000);

// UDP Client

var dgram = require('dgram');
var client = dgram.createSocket('udp4');
var message = new Buffer('A Datagram Message!!!');
client.send(message,0,message.length,4000,'localhost');
client.close();

HTTP Server

// HTTP Server 
var util = require('util');
var http = require('http');
var url = require('url');

http.createServer(function(req,res){ 
    res.writeHead(200, {'Content-Type':'application/json'});
    var response = url.parse(req.url, true);
    res.end(util.inspect(response));
}).listen(4000);


// HTTP Client
var http=require('http');
var options={
    host: 'localhost', port: 4000,
    path: '/'
};

http.get(options,function(res)
{ 
    console.log('got response: ' + res.statusCode);
}).on('error',function(err){ 
    console.log('got error: ' + err.message)
});

TLS / SSL

// Works based on public / private key combination
// based on openssl library

// Generate Private Key
// openssl genrsa -out my_private_key.pem 1024

// Public Key (CSR)
// openssl req -new -key my_private_key.pem -out my_public_key.pem

// Self Signed Certificate with CSR
// openssl x509 -req -in my_public_key.pem -signkey my_private_key.pem -out my_cert.pem

SSL Client

var tls = require('tls'), 
    fs = require('fs'), 
    port = 3000,
    host = 'localhost', 
    options = {
      key : fs.readFileSync('./my_private_key.pem'),
      cert : fs.readFileSync('./my_cert.pem')
    };

// To avoid errors due to self signed certificate
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0"

var client=tls.connect(port,host,options,function(){ 
        console.log('connected');
        console.log('authorized: ' + client.authorized); 
        
        client.on('data', function(data) {
            client.write(data); 
            // just send data back to server });
        });
});

SSL Server

var tls = require('tls'), 
    fs = require('fs'), 
    options = {
      key : fs.readFileSync('./my_private_key.pem'),
      cert : fs.readFileSync('./my_cert.pem')
    };

tls.createServer(options,function(s){ 
    s.pipe(s);
}).listen(4040);

Websocket Server

// npm install websocket
// Please note that there are many other websocket implementations example socket.io

var server = require('websocket').server, 
    http = require('http');

var socket = new server({
    httpServer: http.createServer().listen(1337)
});

socket.on('request', function(request) {
    var connection = request.accept(null, request.origin);

    connection.on('message', function(message) {
        console.log(message.utf8Data);
        connection.sendUTF('Welcome');
        setTimeout(function() {
            connection.sendUTF('Sending message back to the websocket client');
        }, 1000);
    });

    connection.on('close', function(connection) {
        console.log('connection closed');
    });
}); 

Websocket Client

<!DOCTYPE html>

<head>
    <meta charset="utf-8">
    <title>WEBSOCKET CLIENT</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootswatch/3.3.2/paper/bootstrap.min.css">
</head>
<body>
	<div id="content"></div>

	<script type="text/javascript">
    		var content = document.getElementById('content');
    		var socket = new WebSocket('ws://localhost:1337');
    		socket.onopen = function () {
        		socket.send('hello from the client');
   		 };

    		socket.onmessage = function (message) {
        		content.innerHTML += message.data +'<br />';
    		};

    		socket.onerror = function (error) {
        		console.log('WebSocket error: ' + error);
    		};
	</script>
</body>
</html>

Filesystem & Buffers

Hour 04

Buffers

• JavaScript cannot natively handle binary data
• JavaScript is good at handling unicode encoded strings
• Server side javascript needs to handle various binary formats 
• Initially NodeJS used strings, but this lead to problems, hence Buffer was introduced
• Buffer is like an integer array, with its memory allocated outside V8 engine 
• Buffers cannot be resized.
• To convert a buffer to JavaScript object and vice versa, we must provide an encoding
• Supported encodings are 'ascii', 'utf-8', 'utf16le', 'ucs2', 'base64' and 'binary'

Creating Buffers

var Buffer = require('buffer');

// Couple of ways to create buffers

var buffer1 = new Buffer(1024);

// Defaults to utf-8
var buffer2 = new Buffer ('Hello, Friends!');

var buffer3 = new Buffer ([8, 6, 7, 5, 3, 0, 9]);

console.log(buffer1.toString('utf-8'));
console.log('\n');
console.log(buffer2.toString('utf-8'));
console.log('\n');
console.log(buffer3.toString('utf-8'));

Buffer Operations

// Write to a buffer

var buf = new Buffer(100);

buf.write('Hello, World!');
buf.write(' John!', 13);


// Reading from buffer
console.log(buf.toString());

// Reading from buffer like an array
console.log(buf[10].toString());


// Check if the buffer is actually a buffer
console.log("Buffer.isBuffer  " + Buffer.isBuffer(buf));

// Check bytelength required to encode
snowman = "☃";
console.log('Snowman length: ' + snowman.length);
console.log('Buffer Required for snowman: ' + Buffer.byteLength(snowman));

// Check the length of the buffer

console.log('Buffer length is ' + buf.length);

Slice & Copy Buffer

// Slice a buffer
// Slice does not create new memory, it uses the original memory underneath

var buffer = new Buffer ('this is the string in my buffer');
var slice = buffer.slice(10,20);

console.log(slice.toString());


// Copy a buffer
var copy = new Buffer(10);
var targetStart = 0,
    sourceStart = 10,
    sourceEnd = 20;

buffer.copy(copy, targetStart, sourceStart, sourceEnd);

console.log(copy.toString());

// manupulate the slice
slice.write('_-_-_-_-_-');

console.log(buffer.toString());

Buffer Exercise

// Slice a buffer
// Slice does not create new memory, it uses the original memory underneath

var buffer = new Buffer ('this is the string in my buffer');
var slice = buffer.slice(10,20);

console.log(slice.toString());


// Copy a buffer
var copy = new Buffer(10);
var targetStart = 0,
    sourceStart = 10,
    sourceEnd = 20;

buffer.copy(copy, targetStart, sourceStart, sourceEnd);

console.log(copy.toString());

// manupulate the slice
slice.write('_-_-_-_-_-');

console.log(buffer.toString());

File System

• NodeJS has streaming api's for file system access
• NodeJS provides fs module that abstracts the file system access
• Most methods in file system module works asynchronously, but they also have their synchronous counterparts
• fs module provides a lot of methods for file and directory operations
• Ensure all opened files are closed in your node applications.

Synchronous & Asynchronous operations

"use strict";

var fs = require('fs');

// Asynchronous
fs.stat('./', function (err, stats) {

	if(err) {
		console.log(err.message);
		return;
	}
	
	console.log(stats);
});


// Synchronous
try {
	var myStat = fs.statSync('./notAvailable.txt');
	console.log(myStat);
}
catch (e) {
	console.log(e); 
}

File Size

"use strict";

var fs = require('fs');

fs.stat(__dirname+'/readme.txt',function(err,stats){ if (err) { throw err; }
    console.log(stats.size);
});

Read a File - Buffer

var fs = require('fs'); 

fs.open('./readme.txt','r',function(err,fd){
    if (err) throw err;
    var readBuffer = new Buffer(1024),
    bufferOffset = 0,
    bufferLength = readBuffer.length,
    filePosition = 100;

    fs.read(fd, readBuffer, bufferOffset, bufferLength, filePosition, function(err, readBytes) {
        if (err) throw err;
        console.log('just read ' + readBytes + ' bytes'); if (readBytes > 0) {
        console.log(readBuffer.slice(0, readBytes));
      }
    }); 
});

Write to a File

var fs=require('fs');

fs.open('./readme.txt','a',function(err,fd){ 
    var writeBuffer = new Buffer('writing this string'),
    bufferOffset = 0,
    bufferLength = writeBuffer.length, 
    filePosition = null;
    fs.write(fd,writeBuffer, bufferOffset, bufferLength, filePosition, function(err, written) {
        if (err) { throw err; }
          console.log('wrote ' + written + ' bytes');
        }
    ); 
});

Listing Directory Structure

"use strict";

var fs = require('fs');

fs.readdir('.', function (err, files) {

  if (err) throw err;

  files.forEach( function (file) {
      fs.stat('./' + file, function (err, stats) {
      
          if (err) throw err;
      
          if (stats.isFile()) {
            console.log("%s is file", file);
          }
          else if (stats.isDirectory ()) {
              console.log("%s is a directory", file);
          }
          console.log('stats:  %s',JSON.stringify(stats));
        }); 
    });
});

Watching Files

"use strict";

var fs = require('fs');

fs.watchFile('./watch.txt', function (curr, prev) {
  console.log('the current mtime is: ' + curr.mtime);
  console.log('the previous mtime was: ' + prev.mtime);
});

fs.writeFile('./watch.txt', "changed", function (err) {
  if (err) throw err;
  console.log("file changed");
});

STREAMS

Hour 06

Streams

• Abstract interface implemented by various objects in nodejs
• They are readable, writable or both  (Duplex)
• All streams are instances of EventEmitter
• A Duplex stream implements the methods of both readable and writable stream
• We can implement our own stream by implementing appropriate interface method
  • Read only - _read
  • write only - _write, _writev
  • Read & Write - _read, _write, _writev
  • Transform - _transform, _flush [*Duplex]

Reading

"use strict";

var fs = require('fs');

try {
    var rstream = fs.createReadStream('./readme.txt');

    rstream.on('readable', function() {
        console.log('readable ', rstream.read());
    });

    rstream.on('end', function(){
        console.log('end');
    });

    // Uncomment and see what happens

    // rstream.on('data', function(data){
    //    console.log("data ::" + data);
    // })

}
catch (e) {
    console.log('Error occured ' + e);
}

Readstreams

can be paused and resumed

Writing

"use strict";

var fs = require('fs');

var wstream = fs.createWriteStream ('readme.txt');

var count = 1000;

while (count > 0) {
    wstream.write(count + ' \n');
    count -= 1;
}

wstream.end('');

wstream.on('end', function(){
    wstream.close();
})

TESTING

Hour 07

Automated Testing

• Unit Testing
• API Testing
• Acceptance Testing