Visualizing

cloud systems

@lynnlangit

Berlin 2019

Why Do System Pictures Matter?

Verbal
nonVerbal
Visual

CLOud

Applications

  • Web Apps
  • Edge / IoT
  • Data Pipelines

 

Service Views

  • Compute
  • Data
  • Other

Build this!

  • Minimize cost
  • Use a public cloud
  • Design using microservices patterns
  • Process file-based data
  • Expose the result to authenticated web clients

"White Board"

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: >-
  An Amazon S3 trigger that retrieves metadata for the object that has been
  updated.
Parameters:
  BucketNamePrefix:
    Type: String
    Default: sam-example
Resources:
  s3getobject:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: index.handler
      Runtime: nodejs8.10
      CodeUri: .
      Description: >-
        An Amazon S3 trigger that retrieves metadata for the object that has
        been updated.
      MemorySize: 128
      Timeout: 3
      Policies:
        - S3CrudPolicy:
            BucketName: !Sub "${BucketNamePrefix}-get-object"
      Events:
        BucketEvent1:
          Type: S3
          Properties:
            Bucket: !Ref Bucket1
            Events:
              - 's3:ObjectCreated:*'
  Bucket1:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName: !Sub "${BucketNamePrefix}-get-object"

What does this "code" do?

Reference Architecture

Reference Architecture

(Lack of) Reference

Year 2009

Goal:
Cost/Month: $5

PAIN:
Development Time =  Weeks

a Translation problem:
Pictures ? language

Visual Detail

Services

The most important one

BIAS

for build vs buy

System Views: Is "Control Room" the Goal?

Cloud

Applications

  • Web Apps
  • Edge / IoT
  • Data Pipelines

 

System Views

  • Design
  • Development
  • Deployment
  • Operations

Challenge: Visualize ALL System Views

 

Which System view?

Goal:
Deploy Daily

PAIN:
Development Time =  months

Which System view?

BIAS

toward single role perspective

PerspEctive

Cloud

Applications

  • Web Apps
  • Edge / IoT
  • Data Pipelines

By 2050
~ 50% sequenced

WHY

Genomic Sequencing for Improved Human Health

Time: 1 Day

Speed: 80% faster

GOAL:

Fix Performance

GOAL:

Build Faster

Build this!

  • Use a public cloud to minimize cost
  • Create a reproducible design (publications)
  • Use modular design
  • Process file-based data
  • Expose the result to authenticated web clients

"White Board"

terraform {
  backend "s3" {
    bucket  = "variant-spark-july"
    key     = "variantspark-k/tfstate"
    region  = "us-west-2"
    profile = "default"
  }
}

provider "aws" {
  profile = "${var.profile}"
  region  = "${var.default_region}"
}

provider "aws" {
  alias   = "use1"
  profile = "${var.profile}"
  region  = "us-east-1"
}

module "eks-vpc" {
  source         = "modules/eks-vpc"
  "cluster-name" = "${var.cluster-name}"
}

module "eks-master-role" {
  source = "modules/eks-master-role"
}

module "eks-master-security-group" {
  source = "modules/eks-master-security-group"
  vpc_id = "${module.eks-vpc.vpc_id}"
}

module "eks-worker-role" {
  source = "modules/eks-worker-role"
}

module "eks-worker-security-group" {
  source = "modules/eks-worker-security-group"

  vpc_id                   = "${module.eks-vpc.vpc_id}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  "cluster-name"           = "${var.cluster-name}"
}

module "eks" {
  source                   = "modules/eks"
  cluster_name             = "${var.cluster-name}"
  vpc_id                   = "${module.eks-vpc.vpc_id}"
  public_subnets           = "${module.eks-vpc.public_subnets}"
  master_role_arn          = "${module.eks-master-role.master_role_arn}"
  master_security_group_id = "${module.eks-master-security-group.security_group_id}"
  worker_role_arn          = "${module.eks-worker-role.worker_role_arn}"
  worker_profile_name      = "${module.eks-worker-role.worker_profile_name}"
  worker_security_group_id = "${module.eks-worker-security-group.security_group_id}"
  caller_profile           = "${var.profile}"
  eks_key_name             = "${module.kops-ssh.key_name}"
  spark_user_arn           = "${module.kops-user.arn}"
}

module "kops-user" {
  source = "modules/kops-user"
}

module "state-storage" {
  source = "modules/state-storage"
}

module "input-bucket" {
  source = "modules/input-bucket"
}

module "kops-ssh" {
  source = "modules/ssh-key"
}

data "aws_availability_zones" "available" {}

data "template_file" "env" {
  template = "${file("templates/env.tpl")}"


What does this "code" do?

OSS ML for Bioinformatics

500HRS

TEN

Minutes

500 Hours

First Architecture -> HOURS

GOAL:

Create intermediate step

2nd Architecture -> Minutes

GOAL:

Reduce Cost w/Serverless Data

3rd Architecture - full pipeline

What

Went

wrong?

BIAS

 

toward adding complexity

Complexity VIA multidimensionalty

 WITH SET [Top10ProductsByProfit] AS
             TopCount( [Product].[Product Categories].[Product].members,
                 10,[measures].[Internet Gross Profit])
         MEMBER [ProductProfitrank] AS
             rank([Product].[Product Categories].Currentmember,
                 [Top10ProductsByProfit])
    
    SELECT {[measures].[Internet Gross Profit],
            [measures].[ProductProfitrank]} on COLUMNS,
           [Top10ProductsByProfit] on ROWS
         FROM [Adventure Works]
         WHERE [Customer].[Country].[Canada]

what Is
Missing?

Are Systems Furniture?

Are Systems Graphs?

Reading

  • "The Dimensionality of Visual Space" - link

  • "A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration" - link
  • "Understanding Visualization: A Formal Approach using Category Theory and Semiotics" - link
  • "Introduction: The Geometry of the Visual Field—Early Modern and Contemporary Approaches" - link
  • "Visualizing MNIST: An Exploration of Dimensionality Reduction" - link

  • "Visual Interaction with Dimensionality Reduction: A Structured Literature Analysis" - link

Visual Dialect for Systems?

a particular form of a language that is peculiar to a specific region or social group

synonyms: regional language, local language, local speech, vernacular, idiom;

regionalisms, localisms;

informal lingo

 

 

effective Dialect?

visual understanding

living Systems

a Translation problem:
Picturing Living Systems

How to show Movement?

How to Show System Change Over Time?

How to Show Beauty?

Is Sketching Best?

"<The> guarantees form a negative architecture - a set of things that you know can’t happen in various pieces of your system" - Michael Feathers

Showing by Not Showing

BIAS

 

not looking outward

Artists draw because they want to convey something they can't describe"
- Bret Victor

Dynamic Systems - Bret Victor

practically

dynamic system visualizations

VisuAl
Dimensionality

Reduction

'3-to-5 Aspect' Architectures

SECOND Architecture -> Minutes

3rd Arch -> Broad Reach

20 Samples w/

80 MILLION Features

1hour / ~ $ 3 USD

Cloud Genomics - Terra

Pattern:

Serverless Data

Visual Tool - Terra

Pattern:

Visualize Data/Process

Deliver Value Faster

Dev Time: 2 days
Run Cost: $5

Year 2019

//TODO #NOW

ACTION TARGET INFO
Visualize Entire System Code, Config & Data
Use  Visual Grammar Account for bias
Reduce Dimensionality Show 3-5 aspects max
Observe System State Over time
Verify Objects Reproducibility

//Todo #NEXT
Visualize living

Systems

Visualizing

cloud systems

@lynnlangit

Berlin 2019

Looking
Forward

Promising Directions

CombinIng Views -
Dynamic GCP 'Sliders'

Add Levels - c4 MODELS

Extendable IDE - VSCODE

Visualize DAta Files (and code)

Design to Build - Stackery

Generalizations- Cell-Based Architecture

Detect DRIFT - Cloud Formation

Integrated Observability - THundra

Executable Pipeline- Orange

Executable Diagrams?

Design/Costs - Cloudcraft

Data Immersion - Allosphere

"a new way to see, hear and interpret scientific data"

If we spoke a different language, we would perceive a somewhat different world." - Wittgenstein

Visualizing Cloud Systems

By Lynn Langit

Visualizing Cloud Systems

Berlin 2019 Talks

  • 523
Loading comments...

More from Lynn Langit