Serverless Gardens
IoT + Serverless
twitter.com/@johncmckim
johncmckim.me
medium.com/@johncmckim
Software Engineer at A Cloud Guru
John McKim
@johncmckim
Contribute to Serverless Framework
https://acloud.guru
Serverless Framework
https://serverless.com
Agenda
- What is Serverless
- Why I built this project
- Overall Architecture
- Design of each Microservice
- GraphQL + Lambda
- What I learnt
- Questions
What is Serverless?
Serverless
FaaS + The Herd
What is Serverless?
A Serverless Architecture is an event driven system that utilises FaaS and other fully managed services for logic and persistence.
Why choose Serverless?
Benefits
- Easier Operations Mangement
- Reduced Operational Cost
- Reduced Development Time / Cost
- Highly Scalable
- Loosely Coupled systems
Why build this?
For fun and learning
The Problem
Caring for my Garden
Serverless Garden
IoT Service
AWS IoT Service
How It works
Device Gatway
Protocols
- MQTT - devices
- MQTT over Web Sockets - browsers
- HTTP - last resort
Device Gatway
Authentication
- X.509 Certificates - Mutual TLS
- IAM - Signed Requests
- Cognito - tokens
Device
Fake Device
const awsIot = require('aws-iot-device-sdk');
const device = awsIot.device({
'keyPath': './certificates/private.pem.key',
'certPath': './certificates/certificate.pem.crt',
'caPath': './certificates/verisign-ca.pem',
'clientId': 'garden-aid-client-test-js',
'region': 'ap-southeast-2'
});
device
.on('connect', function() {
const topic = 'garden/soil/moisture';
const message = JSON.stringify({
DeviceId: 'test-js-device',
Recorded: (new Date()).toISOString(),
Level: level
});
device.publish(topic, message, {});
});
Demo
Fake Device
Rules Engine
Message Selection & Transformation
SQL Statement
- FROM — MQTT topic
- SELECT — transforms the data
- WHERE (optional)
SELECT DeviceId, Recorded, Level FROM 'garden/soil/moisture'
Rules Engine
Actions
-
Lambda
-
DynamoDB
-
ElasticSearch
-
SNS
-
SQS
-
Kinesis
-
CloudWatch
-
Republish to another MQTT topic.
Rules Engine
IoT Rule in serverless.yml
SensorThingRule:
Type: AWS::IoT::TopicRule
Properties:
TopicRulePayload:
RuleDisabled: false
Sql: "SELECT DeviceId, Recorded, Level FROM '${{opt:stage}}/garden/soil/moisture'"
Actions:
-
DynamoDB:
TableName: { Ref: MoistureData }
HashKeyField: "ClientId"
HashKeyValue: "${clientId()}"
RangeKeyField: "Timestamp"
RangeKeyValue: "${timestamp()}"
PayloadField: "Data"
RoleArn: { Fn::GetAtt: [ IotThingRole, Arn ] }
-
Lambda:
FunctionArn: { Fn::GetAtt: [ checkMoistureLevel, Arn ] }
Notifications Service
- Single purpose functions
- High cohesion
- Loose coupling
Messaging Options
Amazon Simple Queue Service (SQS)
Benefits
- Dead letter queues
- Reliable
Drawbacks
- No integration with Lambda
- Difficult to build scalable processor
- Single processor / queue
Fully Managed message queuing service.
Messaging Options
Amazon Kinesis Streams
Benefits
- Integrates with Lambda
- Batched messages
- Ordered messages
Drawbacks
- Single lambda / shard
- Scale per shard
- Log jams
- Messages expire
Capture and store streaming data.
Messaging Options
Amazon Simple Notification Service (SNS)
Benefits
- Integrates with Lambda
- Fan out multiple Lambdas
Drawbacks
- Small message size
- 3-5 retry's then drop message
Full managed messaging and Pub/Sub service
Notification Service
Check Level
const AWS = require('aws-sdk');
const sns = new AWS.SNS();
const publish = (msg, topicArn, cb) => {
sns.publish({
Message: JSON.stringify({
message: msg
}),
TopicArn: topicArn
}, cb);
};
module.exports.checkLevel = (event, context, cb) => {
if(event.Level < 2.5) {
const msg = 'Moisture level has dropped to ' + event.Level;
const topicArn = process.env.mositureNotifyTopic;
publish(msg, topicArn, cb);
cb(null, { message: msg, event: event });
return;
}
cb(null, { message: 'No message to publish', event: event });
}
Notifications Service
Slack Notifier
const BbPromise = require('bluebird');
const rp = require('request-promise');
const util = require('util');
const notify = (msg) => {
return rp({
method: 'POST',
uri: process.env.slackWebHookUrl,
json: true,
body: {
text: msg,
},
});
}
module.exports.notify = (event, context, cb) => {
console.log(util.inspect(event, false, 5));
const promises = [];
event.Records.forEach(function(record) {
if(record.EventSource !== 'aws:sns') {
console.warn('Recieved non sns event: ', record);
return;
}
const notification = JSON.parse(record.Sns.Message);
promises.push(notify(notification.message));
});
return BbPromise.all(promises)
.then(() => cb(null, { message: 'success' }))
.catch(cb);
};
Demo
Slack Notifications
Web Services
Web Client
- React SPA
- Firebase Hosting
- Auth0 for authentication
Web Backend
- GraphQL API
- API Gateway + Lambda
- Data in DynamoDB
- Custom authoriser
Web Services
API Gateway
What is it?
- HTTP Endpoint as a Service
- Integrates with Lambda
- Convert HTTP Request to Event
- Can delegate Authorization
Web Services
Auth0 Authentication
Web Services
Authentication with GraphQL
const networkInterface = createNetworkInterface(GRAPHQL_URL);
networkInterface.use([{
applyMiddleware(req, next) {
if (!req.options.headers) {
req.options.headers = {}; // Create the header object if needed.
}
// get the authentication token from local storage if it exists
const idToken = localStorage.getItem('idToken') || null;
if (idToken) {
req.options.headers.Authorization = `Bearer ${idToken}`;
}
next();
},
}]);
Web Services
Custom Authorizer
const utils = require('./auth/utils');
const auth0 = require('./auth/auth0');
const AuthenticationClient = require('auth0').AuthenticationClient;
const authClient = new AuthenticationClient({
domain: process.env.AUTH0_DOMAIN,
clientId: process.env.AUTH0_CLIENT_ID,
});
module.exports.handler = (event, context, cb) => {
console.log('Received event', event);
const token = utils.getToken(event.authorizationToken);
if (!token) {
return cb('Missing token from event');
}
const authInfo = utils.getAuthInfo(event.methodArn);
return authClient.tokens.getInfo(token)
.then((userInfo) => {
if (!userInfo || !userInfo.user_id) {
throw new Error('No user_id returned from Auth0');
}
console.log(`Building policy for ${userInfo.user_id} with: `, authInfo);
const policy = new AuthPolicy(userInfo.user_id, authInfo.accountId, authInfo);
policy.allowMethod(AuthPolicy.HttpVerb.POST, '/graphql');
const result = policy.build();
console.log('Returning auth result: ', result, result.policyDocument.Statement);
return result;
})
.catch((err) => {
console.log(err);
return 'Unauthorized';
});
};
Demo
Dashboard
What is GraphQL?
type Project {
name: String
stars: Int
contributors: [User]
}
{
project(name: "GraphQL") {
stars
}
}
{
"project": {
"stars": 4462
}
}
Schema
Results
Query
Why GraphQL?
- One endpoint (per service) to access your data
- The client chooses the response format
- No versioning *
import gql from 'graphql-tag';
import { connect } from 'react-apollo';
import MoistureChart from '../../pres/Moisture/Chart';
export default connect({
mapQueriesToProps({ ownProps, state }) {
return {
moisture: {
query: gql`{
moisture(hours: ${ownProps.hours}, clientId: "${ownProps.clientId}") {
date, moisture
}
}`,
variables: {},
pollInterval: 1000 * 30, // 30 seconds
},
};
},
})(MoistureChart);
GraphQL Query
GraphQL Schema
const graphql = require('graphql');
const tablesFactory = require('./dynamodb/tables');
const MoistureService = require('./services/moisture');
const tables = tablesFactory();
const moistureService = MoistureService({ moistureTable: tables.Moisture });
const MoistureType = new graphql.GraphQLObjectType({
name: 'MoistureType',
fields: {
date: { type: graphql.GraphQLString },
moisture: { type: graphql.GraphQLFloat },
}
});
const schema = new graphql.GraphQLSchema({
query: new graphql.GraphQLObjectType({
name: 'Root',
description: 'Root of the Schema',
fields: {
moisture:
name: 'MoistureQuery',
description: 'Retrieve moisture levels',
type: new graphql.GraphQLList(MoistureType),
args: {
clientId: {
type: graphql.GraphQLString,
},
hours: {
type: graphql.GraphQLInt,
defaultValue: 1
},
},
resolve: (source, args, root, ast) => {
const hours = args.hours > 0 ? args.hours : 1;
return moistureService.getLastHours(args.clientId, hours);
}
}
}
})
});
module.exports = schema;
AWS Lambda
const graphql = require('graphql');
const schema = require('./schema');
module.exports.handler = function(event, context, cb) {
console.log('Received event', event);
const query = event.body.query;
return graphql.query(schema, event.body.query)
.then((response) => {
cb(null, response)
})
.catch((error) => {
cb(error)
});
}
Demo
GraphQL Query
GraphQL on AWS Lambda
Single Lambda Design
GraphQL on AWS Lambda
Lambda Tree Design
Summary
Serverless + IoT
My Experiences
- No server operations
- Cost - $0
- Use *aaS services
- Focus on developing functionality
- Iterate quickly & scale
Alternative Options
IoT Service
Device Shadows
- Stores Device State
- Get current state
- Track state
Alternative Options
Notifications Service
- Monolithic Notification Lambda
- Other notification services
- Facebook Messenger
- Sms - Twillio, Nexmo
Alternative Options
Web Services
- Front-end Framework
- Angular
- Vue
- Elastic Search instead of DynamoDB
- Web Service own Data Store
What did I learn?
- Know your services well
- Know what services exist
- Selecting Boundaries is hard
- Automation is always worth it
- GraphQL is awesome
Many things
Resources
Code + Reading
- github.com/garden-aid
- serverless.zone
Frameworks & Tools
- serverless.com
- AWS
- Firebase
- Auth0
Thanks for Listening!
Questions?
twitter.com/@johncmckim
johncmckim.me
medium.com/@johncmckim
Serverless Gardens
By John McKim
Serverless Gardens
- 137