Visualizing

cloud systems

@lynnlangit

Berlin 2019

Why Do System Pictures Matter?

Reference Architecture

Reference Architecture

(Lack of) Reference

a Translation problem:
Pictures ? language

Visual Detail

Services

BIAS

do icons matter?

Cloud

Applications

  • Web Apps
  • Edge / IoT
  • Data Pipelines

PerspEctive

By 2050
~ 50% sequenced

WHY

Genomic Sequencing for Improved Human Health

Tools for Research - GT-Scan2

Time: 1 Day

Speed: 80% faster

GOAL:

Fix Performance

GOAL:

Build Faster

Build on Another Cloud

Next Challenge

  • Use a public cloud to minimize cost
  • Create a reproducible design (publications)
  • Use modular design
  • Process file-based data
  • Expose the result to authenticated web clients

500HRS

Hello Variant Spark

TEN

Minutes

500 Hours

First Architecture -> HOURS

2nd Architecture -> Minutes

3rd Architecture - full pipeline

What

Went

wrong?

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"


What does this "code" do?

BIAS

 

toward adding complexity

Complexity VIA multidimensionalty

 WITH SET [Top10ProductsByProfit] AS
             TopCount( [Product].[Product Categories].[Product].members,
                 10,[measures].[Internet Gross Profit])
         MEMBER [ProductProfitrank] AS
             rank([Product].[Product Categories].Currentmember,
                 [Top10ProductsByProfit])
    
    SELECT {[measures].[Internet Gross Profit],
            [measures].[ProductProfitrank]} on COLUMNS,
           [Top10ProductsByProfit] on ROWS
         FROM [Adventure Works]
         WHERE [Customer].[Country].[Canada]

what Is
Missing?

Are Systems Furniture?

Are Systems Graphs?

Dialect for Systems

a particular form of a language that is peculiar to a specific region or social group

synonyms: regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

 

 

visual Dialect

Reading

"The Dimensionality of Visual Space" - link
"A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration" - link
"Visual Interaction with Dimensionality Reduction: A Structured Literature Analysis" - link
"Understanding Visualization: A Formal Approach using Category Theory and Semiotics" - link
"Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches" - link
"Visualizing MNIST: An Exploration of Dimensionality Reduction" - link

effective Dialect?

visual understanding

living Systems

a Translation problem:
Picturing Living Systems

Data visualization is all about melding the visual and the conceptual"
- David McCandless

What Makes Good Information Design?

If you can walk you can dance,
If you can talk you can sing.
"


- African Proverb

Triangles

How to Draw
A
Triangle

Is Sketching Best?

How to show Movement?

How to Show System Change Over Time?

How to Show Beauty?

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

How to Showby Not Showing?

BIAS

 

not looking outward

Artists draw because they want to convey something they can't describe"
- Bret Victor

Dynamic Systems - Bret Victor

practically

developing visual grammars for systems

VisuAl
Dimensionality

Reduction

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Cloud Genomics - Terra

Pattern:

Serverless Data

Visual Tool - Terra

Pattern:

Visualize Data/Process

Deliver Value Faster

Dev Time: 2 days
Run Cost: $5

//TODO #NOW

ACTION TARGET INFO
Visualize Entire System Code, Config & Data
Use  Visual Grammars Account for bias
Reduce Dimensionality Show 3-5 aspects max
Show System State Over time
Verify Pictured Objects Reproducibility

//Todo #NEXT
Visualize living

Systems

Visualizing

cloud systems

@lynnlangit

Berlin 2019

Looking
Forward

Promising Directions

CombinIng Views -
Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Integrated Observability - THundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

"a new way to see, hear and interpret scientific data"

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

GOTO: Visualizing Cloud Systems

By Lynn Langit

GOTO: Visualizing Cloud Systems

Berlin 2019 Talks - for GOTO

  • 1,660