IoT Cloud Platform

from scratch

IoT Cloud Platform

  • Bootstrap a server
  • Fast iterations
  • Be as much efficient as possible
  • Do not reinvent the wheel
  • Find a market fit before going in full production
  • Be flexible
  • Be ready to change everything
  • Prototyping
  • Fast deployments
  • Focus on the product not the technology

Startup Goals

"Be wrong as fast as you can"

Pixar philosophy

Bootstrap a Server

Python

  • High productivity
  • Flexibility
  • Fast prototyping
  • Huge support
  • Huge ecosystem
  • Good portability
  • Arguably poor performance
  • Arguably messy
  • Arguably problems with async and multi thread (GIL...)

Bootstrap a Server

Flask

  • Microframework
  • Super simple
  • Pythonic
  • Explicit (easy refactoring)
  • Extendible
  • Good support
  • Good community
  • Arguably fast
  • Arguably safe
from flask import Flask
app = Flask(__name__)


@app.route("/")
def hello():
    return "Hello World!"
from flask import Flask, request, jsonify

...

@app.route('/api/resource/<uuid>', methods=['POST'])
def create_resource(uuid):
    content = request.get_json()
    ...
    return jsonify({'uuid': uuid})

Alternatives I like

Pyramid

Database

MongoDB or Postgres

Database

MongoDB or Postgres

  • Document based
  • Distributed
  • Data model flexibility
  • Document atomic
  • Horizontal scaling
  • Big Data with high concurrency, where data integrity is not important
  • Table based
  • Monolithic
  • SQL language
  • Database atomic
  • Vertical scaling
  • Tons of features...

Database

MongoDB or Postgres

Pick one, I don't care too much.

The focus should be on the product now.

+

Alembic

  • ORM
  • Postgres support
  • Migrations
  • Auto detect schema changes
  • Upgrade + Downgrade
from pymongo import MongoClient

client = MongoClient()
db = client.test

result = db.very_important.insert_one({
    'the answer to the meaning of life, the universe, and everything': 42,
})

(and you can still have migrations with external tools)

  1.  Don't reinvent the wheel

  2.  Be efficient

  3.  PAAS

Platform as a Service

  • Easy deploy
  • Infrastructure problems solved
  • Resource allocation
  • Scaling

...

  • Deploy with git
  • Plugin system
  • Auto-scaling
    (pay what you use)
  • Easy to use
  • App pipelines
    (dev -> staging -> production)
  • Teams support
  • HTTP based
  • Free tiers

git -> build -> slug -> dyno

  • it builds a slug (snapshot) from the source
  • it runs a VM from the slug 
  • it runs one or more processes (dyno) in the VM

web: gunicorn hello:app

worker: celery worker --app=tasks.app

Procfile

Messaging: MQTT

Publish-Subscribe Protocol

Broker

C1

C1

Backend

Publish

Publish

Subscribe

Publish

import paho.mqtt.client as mqtt

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
    print("Connected with result code "+str(rc))

    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed.
    client.subscribe("$SYS/#")

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.payload))

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("mqtt.something.com", 1883, 60)


# Blocking
# client.loop_forever()

# Threaded
client.loop_start()

MQTT Broker

  • Safe message delivery
  • Arguably fast (thousands of devices)
  • Resource-constrained devices
  • Less features (ex. no queues)
  • Simpler than AMQP
  • Smaller memory footprint
  • CloudMQTT plugin for Heroku
  • Mosquitto (Open Source)

VM on demand

  • Auto-setup
  • Auto-delete
  • Full control
  • Complete API
  • Static IP
  • Snapshots
  • Create from snapshots

python-digitalocean

import digitalocean

manager = digitalocean.Manager(token="my secret")

my_droplets = manager.get_all_droplets()

for droplet in my_droplets:
    droplet.shutdown()
import digitalocean

droplet = digitalocean.Droplet(token="my secret",
                               name='Example',
                               region='nyc2',
                               image='ubuntu-14-04-x64',
                               size_slug='512mb',
                               backups=True)

droplet.create()

Linux is everywhere

GO

  • Fast
  • Easy to learn
  • Designed for concurrency
  • Deploy everywhere
  • Static independent binaries
  • Wonderful standard library
  • Awesome community
  • Awesome tools
  • Ideal for embedded
    development
import (
        mqtt "github.com/eclipse/paho.mqtt.golang"
)

func main() {

    ...

    opts := mqtt.NewClientOptions().AddBroker("tcp://somewhere:1883")

    client := mqtt.NewClient(opts)
    client.connect()

    client.Subscribe("my/topic", 0, func(client mqtt.Client, msg mqtt.Message) {

        // Do something with payload
        msg.payload()
    })

    ...

    client.Publish("my/topic", 0, false, "mymessage")
    ...
}

paho.mqtt.golang

Json Payloads

package main

import "encoding/json"
import "fmt"

type Payload struct {
    Field1 int      `json:"field1"`
    Field2 []string `json:"field2"`
}

func main() {

    p := &Payload{
	Field1: 10,
	Field2: []string{"one", "two", "three"},
    }

    output, _ := json.MarshalIndent(p, "", "    ")

    fmt.Println(string(output))

    payload := Payload{}
    json.Unmarshal(output, &payload)

    fmt.Println(payload)
}

{
    "field1": 10,
    "field2": [
        "one",
        "two",
        "three"
    ]
}


{10 [one two three]}

Thank you!

IoT Cloud Platform from Scratch

By Gendo Ikari

Made with Slides.com

IoT Cloud Platform from Scratch

  • 61

More from Gendo Ikari