Anatomy of a Full Stack Project

Yuval Adam

  • Full stack developer and consultant
  • Open source, privacy & security
  • Django Girls TLV co-organizer
  • https://yuv.al
  • @yuvadm

Project Overview

Stop Motion @ Jerusalem Film Festival

  • Stop motion stations designed for children
  • Hardware by Gil Adam and Jonathan Ward

Stop Motion

Stop Motion

From web app to real-life app

Block Diagrams

Stop Motion Stations

High Level Overview

Stop Motion Stations

Raspberry Pi Stations

  • Simple image-based UI
    • GTK? Kivy? QT? Finally, Tk
  • Receive GPIO commands
  • Capture camera frames

System

  • Arch Linux ARM
    • Customizable
    • Lightweight (raspbian is bloated)
  • systemd units for running daemons
  • systemd for network connections
  • Auto-login && startx
    • via ~/.bash_profile
    • should have used systemd but was lazy

Image-based UI

API

  • Movie package
    • frames 0000.jpg, 0001.jpg, ....
    • MANIFEST
    • (NO)PUBLISH flag
  • rsync all files to big screen unit for processing
  • done

Scalability

  • How do we preview at 10 frames per second?
    • Low res thumbnails - in memory
    • High res preview - overlay feh slideshow
  • Long running UI process
    • Check carefully for memory leaks!
    • Worst case - process dies and restarts

Big Screen Processing

Processing Unit

  • Receive movie packages
  • Process *.JPG -> MP4
  • Display on central screen

Processing Unit

  • Odroid XU4
  • 2 x quad core ARM
  • 2GB RAM, Mali ARM GPU
  • $80

Processing Flow

  • Python-based daemon
  • Watch for incoming movie packages
  • ffmpeg
    • compile 4 second title frame
    • compile 10fps movie
    • concat title + movie
  • upload final assets to S3
  • mpv
    • control via UNIX socket commands

Scalability

  • compile movies
  • maintain a smooth display on TV
  • ffmpeg is a monster, limit threads

Network

Router

  • TP-Link WDR4300
  • OpenWrt
    • Open source Linux firmware for routers
  • Uplink via 4G netstick
  • WLAN coverage for all devices

Tunneling

  • Remote access for maintenance and debugging
  • Dynamic DNS good enough on landline
    • No good on 4G
  • OpenVPN tunnel
    • Use local server for reduced latency
  • Map chosen ports -> 22 on each device
    • 2200X > 10.0.0.X:22

Security

  • Enable SSH only from VPN/LAN, or:
  • Disable password login (key-based ONLY)
    • or you will get scanned and pwned
  • Leave no port open #IoT

Website

Website

Website

  • Simple Flask application
  • Pull metadata from S3
  • Stash into Redis
  • All movies are served directly from S3
    • in-browser or downloaded
  • Deployed on Heroku

Poor Man's Background Task

@app.after_request
def update_data(response):
    try:
        if not redis.get('fresh'):
            redis.setex('fresh', 'yes', 60)
            run_expensive_update_operation()
    except ConnectionError as e:
        if app.debug:
            run_expensive_update_operation()
    return response

Full stack

  • Prepare to take care of everything
  • 'Secondary' tasks are now high-priority
    • Logging
    • Security
    • Debugging
  • Performance and scalability
    • know your hardware

Questions?

Anatomy of a Full Stack Project

By Yuval Adam

Anatomy of a Full Stack Project

  • 3,120