Visualizing

Cloud Systems

Chicago 2020

Lynn Langit

Why Do System Pictures Matter?

Cloud Infrastructure

TYPE METHOD INFO
WebUI click to create AWS, GCP, Alibaba Cloud
Script  bash-like
Pythonic
awscli, gcloud, aliyun
Deployment Template
YAML/JSON
visual tool
AWS CloudFormation,
GCP Deployment
FUN template
Infrastructure language
visual tool
Terraform
Vertical
bioinformatics
language
 
WDL, CWL
Nextflow
Vertical WebUI visual tool Terra.bio on GCP
Galaxy Project on AWS

BIAS

do icons matter?

By 2050
~ 50% sequenced

Tools for Research - GT-Scan2

Time: 1 Day

Speed: 80% faster

Next

can we solve a more difficult challenge?

500HRS

TEN

Minutes

BIAS

 

solving difficult problems

First Architecture -> HOURS

2nd Architecture -> Minutes

3rd Architecture - full pipeline

What

Went

wrong?

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"


What does this "code" do?

BIAS

 

toward adding complexity

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

What Makes Good Information Design?

Reading

- "The Dimensionality of Visual Space"

- "A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration"

- "Understanding Visualization: A Formal Approach using Category Theory and Semiotics"

- "Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches"

- "Visualizing MNIST: An Exploration of Dimensionality Reduction"

"Artists draw because they want to convey something they can't describe"
- Bret Victor

"If you can walk you can dance,
If you can talk you can sing.
"

- African Proverb

Triangles

How to Draw
A Triangle

visual understanding

BIAS

 

not looking outward,

using only your own context

Visual Grammars

for Cloud Systems

Dialect for Systems

a particular form of a language that is peculiar to a specific region or social group

synonyms: regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

 

 

entry point

Executable Pathways

Dimensionality

Reduction

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Perspective

User | Developer | DevOps

Cloud Genomics - Terra

import "https://raw.githubusercontent.com/broadinstitute/viral-pipelines/master/pipes/WDL/tasks/tasks_taxon_filter.wdl" as taxon_filter
import "https://raw.githubusercontent.com/broadinstitute/viral-pipelines/master/pipes/WDL/tasks/tasks_assembly.wdl" as assembly

workflow assemble_denovo_with_deplete {
  
  call taxon_filter.deplete_taxa

  call taxon_filter.filter_to_taxon {
    input:
      reads_unmapped_bam = deplete_taxa.cleaned_bam
  }

  call assembly.assemble {
    input:
      reads_unmapped_bam = filter_to_taxon.taxfilt_bam
  }

  call assembly.scaffold {
    input:
      contigs_fasta = assemble.contigs_fasta,
      reads_bam = filter_to_taxon.taxfilt_bam
  }

  call assembly.refine_2x_and_plot {
    input:
      assembly_fasta = scaffold.scaffold_fasta,
      reads_unmapped_bam = deplete_taxa.cleaned_bam
  }
}

Text

What does this "code" do?

//TODO #NOW

ACTION TARGET INFO
Visualize Entire System Code, Config & Data
Entry point (Hello...)
Use  Visual Grammars Account for bias, audience
Reduce Dimensionality Show 3-5 aspects max
Show System State Over time
Verify Pictured Objects Reproducibility

Visualizing

Cloud Systems

@lynnlangit

Looking
Forward

Promising Directions

CombinIng Views -
Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Cloud MOnitoring

TYPE METHOD INFO
WebUI log viewer AWS CloudWatch, GCP Operations, Alibaba Logs
Alerts  bash-like
Pythonic
awscli, gcloud, aliyun
Aggregations script AWS Log Metrics,
GCP Operations Metrics
WebUI language
visual tool
Honeycomb
Thundra
Vertical WebUI language
visual tool
Terra.bio JobRunner on GCP
Nextflow Tower
Galaxy Project Job Executions on AWS

Integrated Observability - Thundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

"a new way to see, hear and interpret scientific data"

Products to WAtch

Company Product/Service Notable Lifecycle
AWS Cloud Formation
X-Ray
Drift Detection
Log Visualization
ALL
GCP  GCP console Integration of estimated service costs ALL
Stackery
CloudCraft
Draw Config Generates YAML
Monitors costs
BUILD/DEPLOY
Thundra
Epsagon
Log Aggregation Includes external services
Application / transactions
OPERATE
Datadog ARM for Serverless  Log Visualization OPERATE
Honeycomb Honeycomb Query and visualize event patterns OPERATE

//Todo #NEXT
Visualize living

Systems

Artist in Residence - The Broad Institute

Visualizing Serverless Systems

By Lynn Langit

Visualizing Serverless Systems

GOTO Chicago - 2020

  • 1,135