Visualizing

cloud systems

@lynnlangit

Berlin 2019

Image

Why Do System Pictures Matter?

Verbal
nonVerbal
Visual

CLOud

Applications

Web Apps
Edge / IoT
Data Pipelines

Service Views

Compute
Data
Other

Build this!

Minimize cost
Use a public cloud
Design using microservices patterns
Process file-based data
Expose the result to authenticated web clients

"White Board"

Link

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: >-
  An Amazon S3 trigger that retrieves metadata for the object that has been
  updated.
Parameters:
  BucketNamePrefix:
    Type: String
    Default: sam-example
Resources:
  s3getobject:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: nodejs8.10
      CodeUri: .
      Description: >-
        An Amazon S3 trigger that retrieves metadata for the object that has
        been updated.
      MemorySize: 128
      Timeout: 3
      Policies:
        - S3CrudPolicy:
            BucketName: !Sub "${BucketNamePrefix}-get-object"
      Events:
        BucketEvent1:
          Type: S3
          Properties:
            Bucket: !Ref Bucket1
            Events:
              - 's3:ObjectCreated:*'
  Bucket1:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${BucketNamePrefix}-get-object"

What does this "code" do?

Reference Architecture

Link

(Lack of) Reference

Link

Year 2009

Link

Goal:
Cost/Month: $5

PAIN:
Development Time = Weeks

a Translation problem:
Pictures ? language

Visual Detail

Link

Services

The most important one

BIAS

for build vs buy

System Views: Is "Control Room" the Goal?

Cloud

Applications

Web Apps
Edge / IoT
Data Pipelines

System Views

Design
Development
Deployment
Operations

Link

Challenge: Visualize ALL System Views

Which System view?

Link

Goal:
Deploy Daily

PAIN:
Development Time = months

Which System view?

Link

BIAS

toward single role perspective

Link

PerspEctive

Cloud

Applications

Web Apps
Edge / IoT
Data Pipelines

By 2050
~ 50% sequenced

Link

WHY

Genomic Sequencing for Improved Human Health

Link

Time: 1 Day

Speed: 80% faster

GOAL:

Fix Performance

GOAL:

Build Faster

Build this!

Use a public cloud to minimize cost
Create a reproducible design (publications)
Use modular design
Process file-based data
Expose the result to authenticated web clients

"White Board"

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"

What does this "code" do?

Link

OSS ML for Bioinformatics

Link

500HRS

TEN

Minutes

500 Hours

First Architecture -> HOURS

Link

GOAL:

Create intermediate step

2nd Architecture -> Minutes

link

GOAL:

Reduce Cost w/Serverless Data

Link

3rd Architecture - full pipeline

What

Went

wrong?

BIAS

toward adding complexity

Complexity VIA multidimensionalty

Link

 WITH SET [Top10ProductsByProfit] AS
             TopCount( [Product].[Product Categories].[Product].members,
                 10,[measures].[Internet Gross Profit])
         MEMBER [ProductProfitrank] AS
             rank([Product].[Product Categories].Currentmember,
                 [Top10ProductsByProfit])
    
    SELECT {[measures].[Internet Gross Profit],
            [measures].[ProductProfitrank]} on COLUMNS,
           [Top10ProductsByProfit] on ROWS
         FROM [Adventure Works]
         WHERE [Customer].[Country].[Canada]

Link

what Is
Missing?

Are Systems Furniture?

Link

Are Systems Graphs?

Link

Reading

"The Dimensionality of Visual Space" - link
"A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration" - link
"Understanding Visualization: A Formal Approach using Category Theory and Semiotics" - link
"Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches" - link
"Visualizing MNIST: An Exploration of Dimensionality Reduction" - link
"Visual Interaction with Dimensionality Reduction: A Structured Literature Analysis" - link

Visual Dialect for Systems?

a particular form of a language that is peculiar to a specific region or social group

synonyms:

regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

effective Dialect?

visual understanding

living Systems

Link

a Translation problem:
Picturing Living Systems

How to show Movement?

Link

How to Show System Change Over Time?

Link

How to Show Beauty?

Link

Is Sketching Best?

Link

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

Link

Showing by Not Showing

BIAS

not looking outward

Link

Artists draw because they want to convey something they can't describe"
- Bret Victor

Link

Dynamic Systems - Bret Victor

practically

dynamic system visualizations

VisuAl
Dimensionality

Reduction

'3-to-5 Aspect' Architectures

link

SECOND Architecture -> Minutes

link

3rd Arch -> Broad Reach

link

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Link

Cloud Genomics - Terra

Pattern:

Serverless Data

Link

Visual Tool - Terra

Pattern:

Visualize Data/Process

Link

Deliver Value Faster

Dev Time: 2 days
Run Cost: $5

Year 2019

Link

//TODO #NOW

ACTION	TARGET	INFO
Visualize	Entire System	Code, Config & Data
Use	Visual Grammar	Account for bias
Reduce	Dimensionality	Show 3-5 aspects max
Observe	System State	Over time
Verify	Objects	Reproducibility