Visualizing

Serverless Systems

London 2019

Lynn Langit

Why Do System Pictures Matter?

Reference Architecture

Reference Architecture

(Lack of) Reference

a Translation problem:
Pictures ? language

Visual Detail

Services

BIAS

do icons matter?

Serverless

AWS Machine Learning

Serverless

GCP Machine Learning

Serverless

Applications

  • Web Apps
  • Edge / IoT
  • Data Pipelines

By 2050
~ 50% sequenced

WHY

Genomic Sequencing for Improved Human Health

PerspEctive

Tools for Research - GT-Scan2

Time: 1 Day

Speed: 80% faster

GOAL:

Fix Performance

GOAL:

Build Faster

Build on Another Cloud

Keep Building

Next

can we solve a more difficult challenge?

500HRS

Hello Variant Spark

TEN

Minutes

500 Hours

First Architecture -> HOURS

2nd Architecture -> Minutes

3rd Architecture - full pipeline

What

Went

wrong?

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"


What does this "code" do?

BIAS

 

toward adding complexity

Complexity VIA multidimensionalty

 WITH SET [Top10ProductsByProfit] AS
             TopCount( [Product].[Product Categories].[Product].members,
                 10,[measures].[Internet Gross Profit])
         MEMBER [ProductProfitrank] AS
             rank([Product].[Product Categories].Currentmember,
                 [Top10ProductsByProfit])
    
    SELECT {[measures].[Internet Gross Profit],
            [measures].[ProductProfitrank]} on COLUMNS,
           [Top10ProductsByProfit] on ROWS
         FROM [Adventure Works]
         WHERE [Customer].[Country].[Canada]

Reading

"The Dimensionality of Visual Space"

"A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration"

"Understanding Visualization: A Formal Approach using Category Theory and Semiotics"

"Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches"

"Visualizing MNIST: An Exploration of Dimensionality Reduction"

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

what Is
Missing?

Are Systems Furniture?

Are Systems Graphs?

Dialect for Systems

a particular form of a language that is peculiar to a specific region or social group

synonyms: regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

 

 

visual Dialect

effective Dialect?

visual understanding

Picturing living Systems

a Translation problem:
 

Data visualization is all about melding the visual and the conceptual"
- David McCandless

What Makes Good Information Design?

If you can walk you can dance,
If you can talk you can sing.
"


- African Proverb

Triangles

How to Draw
A
Triangle

Is Sketching Best?

How to show Movement?

How to Show System Change Over Time?

How to Show Beauty?

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

How to Show by Not Showing?

BIAS

 

not looking outward

Artists draw because they want to convey something they can't describe"
- Bret Victor

practically

developing visual grammars for systems

VisuAl
Dimensionality

Reduction

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Cloud Genomics - Terra

Pattern:

Serverless Data

Visual Tool - Terra

Pattern:

Visualize Data/Process

Do
more

Imperial College of London

//TODO #NOW

ACTION TARGET INFO
Visualize Entire System Code, Config & Data
Use  Visual Grammars Account for bias
Reduce Dimensionality Show 3-5 aspects max
Show System State Over time
Verify Pictured Objects Reproducibility

//Todo #NEXT
Visualize living

Systems

Artist in Residence - The Broad Institute

Visualizing

Serverless Systems

@lynnlangit

Looking
Forward

Promising Directions

CombinIng Views -
Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Integrated Observability - Thundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

"a new way to see, hear and interpret scientific data"

Products to WAtch

Company Product/Service Notable Lifecycle
AWS Cloud Formation
X-Ray
Drift Detection
Log Visualization
ALL
GCP  GCP console Integration of estimated service costs ALL
Stackery
CloudCraft
Draw Config Generates YAML
Monitors costs
BUILD/DEPLOY
Thundra
Epsagon
Log Aggregation Includes external services
Application / transactions
OPERATE
Datadog ARM for Serverless  Log Visualization OPERATE
Honeycomb Honeycomb Query and visualize event patterns OPERATE

Visualizing Serverless Systems

By Lynn Langit

Visualizing Serverless Systems

Serverless Computing - London 2019

  • 1,600