Visualizing

Serverless Systems

London 2019

Lynn Langit

Why Do System Pictures Matter?

Reference Architecture

(Lack of) Reference

a Translation problem:
Pictures ? language

Visual Detail

Services

BIAS

do icons matter?

Serverless

AWS Machine Learning

Serverless

GCP Machine Learning

Serverless

Applications

Web Apps
Edge / IoT
Data Pipelines

By 2050
~ 50% sequenced

WHY

Genomic Sequencing for Improved Human Health

Link

PerspEctive

Tools for Research - GT-Scan2

Time: 1 Day

Speed: 80% faster

GOAL:

Fix Performance

GOAL:

Build Faster

Build on Another Cloud

Keep Building

500HRS

Hello Variant Spark

TEN

Minutes

500 Hours

First Architecture -> HOURS

2nd Architecture -> Minutes

3rd Architecture - full pipeline

What

Went

wrong?

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"

What does this "code" do?

BIAS

toward adding complexity

Complexity VIA multidimensionalty

 WITH SET [Top10ProductsByProfit] AS
             TopCount( [Product].[Product Categories].[Product].members,
                 10,[measures].[Internet Gross Profit])
         MEMBER [ProductProfitrank] AS
             rank([Product].[Product Categories].Currentmember,
                 [Top10ProductsByProfit])
    
    SELECT {[measures].[Internet Gross Profit],
            [measures].[ProductProfitrank]} on COLUMNS,
           [Top10ProductsByProfit] on ROWS
         FROM [Adventure Works]
         WHERE [Customer].[Country].[Canada]

Reading

"The Dimensionality of Visual Space"

"A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration"

"Understanding Visualization: A Formal Approach using Category Theory and Semiotics"

"Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches"

"Visualizing MNIST: An Exploration of Dimensionality Reduction"

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

what Is
Missing?

Are Systems Furniture?

Are Systems Graphs?

Dialect for Systems

a particular form of a language that is peculiar to a specific region or social group

synonyms:

regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

visual Dialect

effective Dialect?

visual understanding

Picturing living Systems

a Translation problem:

Data visualization is all about melding the visual and the conceptual"
- David McCandless

What Makes Good Information Design?

If you can walk you can dance,
If you can talk you can sing. "

- African Proverb

Triangles

How to Draw
A Triangle

Is Sketching Best?

How to show Movement?

How to Show System Change Over Time?

How to Show Beauty?

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

How to Show by Not Showing?

BIAS

not looking outward

Artists draw because they want to convey something they can't describe"
- Bret Victor

practically

developing visual grammars for systems

VisuAl
Dimensionality

Reduction

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Cloud Genomics - Terra

Pattern:

Serverless Data

Visual Tool - Terra

Pattern:

Visualize Data/Process

Do
more

Imperial College of London

//TODO #NOW

ACTION	TARGET	INFO
Visualize	Entire System	Code, Config & Data
Use	Visual Grammars	Account for bias
Reduce	Dimensionality	Show 3-5 aspects max
Show	System State	Over time
Verify	Pictured Objects	Reproducibility

//Todo #NEXT
Visualize living

Systems

Artist in Residence - The Broad Institute

Visualizing

Serverless Systems

@lynnlangit

Presentation Links

Looking
Forward

Promising Directions

CombinIng Views -
Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Integrated Observability - Thundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

"a new way to see, hear and interpret scientific data"

Products to WAtch

Company	Product/Service	Notable	Lifecycle
AWS	Cloud Formation X-Ray	Drift Detection Log Visualization	ALL
GCP	GCP console	Integration of estimated service costs	ALL
Stackery CloudCraft	Draw Config	Generates YAML Monitors costs	BUILD/DEPLOY
Thundra Epsagon	Log Aggregation	Includes external services Application / transactions	OPERATE
Datadog	ARM for Serverless	Log Visualization	OPERATE
Honeycomb	Honeycomb	Query and visualize event patterns	OPERATE

Visualizing

Why Do System Pictures Matter?

Reference Architecture

Reference Architecture

(Lack of) Reference

a Translation problem: Pictures ? language

Visual Detail

Services

BIAS

AWS Machine Learning

GCP Machine Learning

Serverless

Applications

By 2050 ~ 50% sequenced

WHY

PerspEctive

Tools for Research - GT-Scan2

Build on Another Cloud

Keep Building

Next

500HRS

Hello Variant Spark

TEN

Minutes

500 Hours

First Architecture -> HOURS

2nd Architecture -> Minutes

3rd Architecture - full pipeline

What

Went

wrong?

BIAS

Complexity VIA multidimensionalty

Reading

what Is Missing?

Are Systems Furniture?

Are Systems Graphs?

Dialect for Systems

visual Dialect

effective Dialect?

visual understanding

Picturing living Systems

a Translation problem:

What Makes Good Information Design?

Triangles

How to Draw A Triangle

Is Sketching Best?

How to show Movement?

How to Show System Change Over Time?

How to Show Beauty?

How to Show by Not Showing?

BIAS

practically

VisuAl Dimensionality

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

Cloud Genomics - Terra

Visual Tool - Terra

Do more

Imperial College of London

//TODO #NOW

//Todo #NEXT Visualize living

Systems

Artist in Residence - The Broad Institute

Visualizing

Looking Forward

CombinIng Views - Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Integrated Observability - Thundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

Products to WAtch

a Translation problem:
Pictures ? language

By 2050
~ 50% sequenced

what Is
Missing?

How to Draw
A Triangle

VisuAl
Dimensionality

Do
more

//Todo #NEXT
Visualize living

Looking
Forward

CombinIng Views -
Dynamic GCP 'Sliders'