Visualizing

serverless systems

@lynnlangit

Belgrade 2019

Image

Serverless
Applications

@lynnlangit

  • Web Apps
  • Mobile Apps
  • Edge / IoT
  • Data Pipelines

Time: 2 days
Max Cost/Month: $5

Year 2009

@lynnlangit

Link

Original

Serverless Services

@lynnlangit

Link

@lynnlangit

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: >-
  An Amazon S3 trigger that retrieves metadata for the object that has been
  updated.
Parameters:
  BucketNamePrefix:
    Type: String
    Default: sam-example
Resources:
  s3getobject:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: nodejs8.10
      CodeUri: .
      Description: >-
        An Amazon S3 trigger that retrieves metadata for the object that has
        been updated.
      MemorySize: 128
      Timeout: 3
      Policies:
        - S3CrudPolicy:
            BucketName: !Sub "${BucketNamePrefix}-get-object"
      Events:
        BucketEvent1:
          Type: S3
          Properties:
            Bucket: !Ref Bucket1
            Events:
              - 's3:ObjectCreated:*'
  Bucket1:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${BucketNamePrefix}-get-object"

What does this "code" do?

Build this!

  • Use a public cloud
  • Use microservices design
  • Make 2 services transactional
  • Process file-based data 
  • Expose the result to authenticated web clients

@lynnlangit

"White Board"

@lynnlangit

Link

Reference Architecture

@lynnlangit

Link

Reference Architecture

@lynnlangit

Link

(Lack of) Reference

@lynnlangit

Link

Which System view?

@lynnlangit

Link

BIAS

@lynnlangit

Serverless

only Lambda(s)?

@lynnlangit

Serverless Services

@lynnlangit

Type Name AWS Examples
Compute Functions Lambda
Files Buckets S3
Data NoSQL
Streams		 DynamoDB
Kinesis
Other... Gateway
ETL
Orchestration
Pub/Sub
Machine Learning		 API Gateway
Glue
Step Functions
SNS
Rekognition

a Translation problem:
Pictures ? language

@lynnlangit

@lynnlangit

Link

 <-- SERVERLESS -->

Pattern:

Visualize Services

Serverless
Applications

@lynnlangit

  • Web Apps
  • Mobile Apps
  • Edge / IoT
  • Data Pipelines

WHY

Genomic Sequencing for Improved Human Health

@lynnlangit

By 2050
~ 50% sequenced

@lynnlangit

Link

Speed up Serverless

@lynnlangit

Link

Time: 1 Day

Speed: 80% faster

Pattern:

Visualize Logs

@lynnlangit

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"

What does this "code" do?

Build this!

  • Use a public cloud
  • Use modular design
  • Process file-based data 
  • Expose the result to authenticated web clients
  • Create a reproducible design (publications)

@lynnlangit

"White Board"

@lynnlangit

Link

OSS ML for Bioinformatics

@lynnlangit

Link

500 hours

@lynnlangit

Link

@lynnlangit

complexity

@lynnlangit

Link

  • True Cost of Execution Environment
  • Data quality / features
  • ML Model Hyperparameters
  • Can this be done 'Serverless'?

This is text

 

 

 

 

TEN

Minutes

@lynnlangit

500 Hours

First Architecture -> HOURS

@lynnlangit

Link

Pattern:

Create intermediate step

SECOND Architecture -> Minutes

link

@lynnlangit

Pattern:

Serverless Data

Link

@lynnlangit

pipeline =Notebook, ML++

link

@lynnlangit

What

Went

wrong?

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

Link

@lynnlangit

Visualize

reproducible systems

@lynnlangit

Dialect

a particular form of a language that is peculiar to a specific region or social group

synonyms: regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

 

What is the dialect for SERVERLESS SYSTEMS?

@lynnlangit

Are Systems Furniture?

@lynnlangit

Link

@lynnlangit

effective Dialect?

@lynnlangit

visual understanding

Serverless Systems

@lynnlangit

Link

what Is
Missing?

@lynnlangit

Movement

@lynnlangit

Link

Systems Over Time

@lynnlangit

Link

Beauty

@lynnlangit

Link

Sketching

@lynnlangit

Link

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

@lynnlangit

Link

Link

@lynnlangit

Artists draw because
they want to convey something they can't describe"
- Bret Victor

Link

@lynnlangit

Dynamic Systems - Bret Victor

practically

dynamic system visualizations

@lynnlangit

a Translation problem:
Picturing Living Systems

@lynnlangit

Link

@lynnlangit

Dynamic GCP 'Sliders'

Levels - c4 MODELS

@lynnlangit

Link

Integrated Tools

Genomics Files
Visualized

@lynnlangit

Link

DRIFT - Cloud Formation

link

@lynnlangit

Design->Build -Stackery

link

@lynnlangit

Design/Costs - Cloudcraft

link

@lynnlangit

Link

@lynnlangit

Pipeline design- Orange

Bioinformatics Again--
What is System data?

@lynnlangit

Type Name Notes
User data FASTA, FASTQ
BAM		 Raw sequencer files
Binary user data
Application(s)
Java / Scala		 Custom ML
GATK		 Bioinfo tools / jar files
Config JSON
YAML		 Tool configuration
Vendor cloud services
Log CSV Job run history files

pipeline =Notebook, ML++

link

@lynnlangit

Link

@lynnlangit

Link

@lynnlangit

Scaling Serverless - Terra

Pattern:

Serverless Data

Link

@lynnlangit

Visual Tool - Terra

Pattern:

Visualize Data/Process

Link

@lynnlangit

Deliver Value Faster

Time: 2 days
Cost: $5

Year 2019

Living Serverless Systems

@lynnlangit

Link

//TODO

  • Visualize Entire Systems (code, config & data)
  • Account for Visual Grammar(s) & Bias
  • Show best level(s) of detail
  • Show systems over time 
  • Test system reproducibility

@lynnlangit

Visualizing

serverless systems

@lynnlangit

Belgrade 2019

Image