Better engineering practices

Robus Gauli

@robusgauli

About {me}

Name

Work

More about me @ https://robus.dev

Architecture, Infrastructure, Operations a.k.a Drawing boxes and arrows

Robus Gauli

Tech

Go, Python, Rust, C/C++, K8s, Open X ....

TOPIC {1}

Dependency Management

Onboarding process of dev by dev

Onboarding (1 month)

1. Git clone the repo

2. Go through the README

3. terraform apply / yarn start

Onboarding (6 months)

1. Git clone the repo

2. Go through the README

3. `terraform start`

4. Oh you need terraform v0.12.24

5. Need yarn version v1.32 as well

6. Oh, you need Rabbitmq

7. And postgres, airflow

8. Wait postgres version need to be above 10 for issues we can't figure out yet

9. Please install kubectl and awscli because we use s3

......

Meanwhile your CI/CD pipeline

Timeout

Out of sync versions

Universe installation

What did we miss?

We dockerized our app but didn't manage to isolate Operational dependencies.

Note: Application dependencies and Operational dependencies are not same thing

Application Dependencies

Operational Dependencies

Required by Application runtime

Required during development to deployment to facilitate build, test and release.

For example: Node JS is application dependency for your express app.

For example: AWSCLI is your operational dependency for your app in order to authenticate against docker registry in your cloud. You don't package AWSCLI in your app. You use it to facilitate deployment.

Can we fix it somehow?

Yup, let's look at one approach

1. Create Dockerfile with all of the Operational Dependencies baked into it

3. Execute the command

2. Create build/test scripts that wraps your Operational Docker Image

Note: This is operational dockerfile. It is not the same as application dockerfile.

1. Create Dockerfile with all of the Operational Dependencies baked into it

FROM alpine:3.12

ENV TERRAFORM_VERSION=0.12.24
ENV KUSTOMIZE_VERSION=v3.4.0
ENV HELM_VERSION=v3.3.0

RUN apk update && \
  apk add --no-cache \
  bash \
  git \
  make \
  jq \
  python3 \
  curl \
  wget \
  openssl \
  gettext \
  gomplate \
  ncurses \
  unzip \
  py-pip && \
  pip install -U awscli && \
  pip3 install -U kubernetes && \
  wget -q https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz -O - | tar -xzO linux-amd64/helm > /usr/local/bin/helm && \
  chmod +x /usr/local/bin/helm && \
  curl -L https://dl.k8s.io/v1.21.2/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl && \
  chmod +x /usr/local/bin/kubectl && \
  curl -L https://github.com/vmware-tanzu/velero/releases/download/v1.6.1/velero-v1.6.1-linux-amd64.tar.gz --output - | tar -xzO velero-v1.6.1-linux-amd64/velero  > /usr/local/bin/velero && \
  chmod +x /usr/local/bin/velero && \
  curl -L \
  https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \
  -o /tmp/terraform.zip && \
  unzip /tmp/terraform.zip -d /usr/local/bin/ && \
  curl -L \
  https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz \
  -o /tmp/kustomize.tar.gz && \
  tar -xzf /tmp/kustomize.tar.gz -C /tmp && \
  mv /tmp/kustomize /usr/local/bin/kustomize && \
  rm -rf /tmp/* && \
  rm -rf /var/cache/apk/* && \
  rm -rf /var/tmp/*

## Create a command: terraform init (via operational docker image proxy)
docker/terraform-init:
  docker run \
    --rm \
    --entrypoint=terraform \
    -v $(APP_ROOT):/app \
    -w /app \
    --env AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
    --env AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
    --env TF_CLI_CONFIG_FILE=$(TF_CLI_CONFIG_FILE) \
     \
     awsregistry/operational-deps-image init

2. Create build/test scripts that wraps your Operational Docker Image

Makefile

## Create a command
docker/build:
  docker run \
    --rm \
    --entrypoint=yarn \
    -v $(APP_ROOT):/app \
    -w /app \
     awsregistry/operational-deps-image build

Another example of build react app

Makefile

3. Execute the command

~/home
> make docker/terraform-init

From your terminal

docker/terraform-init:
  docker run \
    --rm \
    --entrypoint=terraform \
    -v $(APP_ROOT):/app \
    -w /app \
    --env AWS_ACCESS_KEY_ID=$(AWS_ACCESS_KEY_ID) \
    --env AWS_SECRET_ACCESS_KEY=$(AWS_SECRET_ACCESS_KEY) \
    --env TF_CLI_CONFIG_FILE=$(TF_CLI_CONFIG_FILE) \
     \
     awsregistry/operational-deps-image init

Let's approach visually

~/home
> make docker/terraform-init

Command to execute

Docker image with operational deps

upload

test

build

apply

Execution inside container

mounted source code inside container

Difference

~/home
> yarn build

~/home
> make docker/build

Before

Now

What did it buy you?

Yup that's it. No joke

Yup that's it. No joke

No dependencies to Build, Test, Lint, Upload, Deploy or anything that is required in your development lifecycle

What about CI?

Local

CI/CD

make docker/build

make docker/test

make docker/deploy

To build

To test

To deploy

make docker/dry-run

make docker/lint

make docker/format

To dry run

To lint

To format

master:
  - step:
      name: Run Lint and Plan
      deployment: prod-plan
      script:
        - make docker/terraform-lint
        - make docker/terraform-plan-out
        
  - step:
      name: Run Test
      deployment: prod-plan
      script:
        - make docker/test
        - make docker/integration-test
        
  - step:
      name: Apply terraform plan
      deployment: prod-apply
      trigger: manual
      script:
        - make docker/terraform-apply

Sample Bitbucket CI scripts from project

{Conclusion}

Don't expect developer to manage dependencies.

Isolating Operational Complexity is equally, if not more important than isolating Application Complexity

Your CI/CD, Staging environments & Local should share common operational and application dependencies.

TOPIC {2}

Programming language, features & patterns

{Features & Patterns}

Language features

* Map/Reduce/Filter

* Classes & Objects

* First class functions, Immutability, closure, polymorphism, inheritance, ABC, etc

* Generators, async/await, coroutines, pattern matching, etc

* Modern error handling. I call it "throw everything and handle nothing"

Programming patterns*

* Singleton pattern

* Factory pattern

* Try/Except control flow

* Multi hierarchy pattern

Step back

Let's evaluate programming from the First principle/Axioms

What does it mean to program?

INPUT

f(x)

OUTPUT

sqr(x)

"robus"

len(x)

Query(user)

find(x)

{name: "john"}

robus.dev

http(x)

<html >

Side effects

None

Requirements for computation

Read from memory

int x = &y

Write to memory

x = 56

Conditional branching

if (x > 45) {...}

Control flow construct

for (..) {break}

Yup that's all. Verified by Turing and Alonzo Church

Example Case study of Golang

No try/catch/exception

No classes & objects

No generators & coroutines

And yet, it empowers 100% of cloud native infrastructure (docker, kubernetes, fargate, etcd, falco, lambdas, etc)

No inheritance

No map, reduce, filters, while, do while

No abstract base class and super/sub class

No macros and templates

What did they do right?

No try/catch/exception

No classes & objects

No generators & coroutines

No inheritance

No map, reduce, filters, while, do while

No abstract base class and super/sub class

No macros and templates

Errors are just values.

Struct

for(...)

Interface

Example Error pattern

user := User.get(id)
mentors.add(user)
parents.add(user)

No error handling

try {
  user := User.get(id)
} catch (USERMISSINGEXC) {
  user := User.create({name: "mac"})
  notifiy(USER_CREATED)
} finally {
  mentors.add(user)
  parents.add(user)
}

Using try/catch

err, user := User.get(id)

if (err == USER.MISSING) {
  user = User.create({name: "mac"})
  notify(USER_CREATED)
}

mentors.add(user)
parents.add(user)

Error as a value

{Conclusion}

One way of doing things

You might not require Singleton pattern if you don't have global state in your language

Rule of thumb: Less features => Good feature

Make sure errors are first class values. Ideally, no compilation without explicit error handling

Use language that has native construct for side effects management. For example: Interface to swap out I/O dependencies in Go/Java/Typescript, Monad to wrap side effects in Haskell and JS, etc

{Programming Language}

Compilation/Transpilation free

Dynamic Types

* Less code => Good code

* I call it "any type programming pattern"

Time to 200 OK

Things that goes wrong

No compiler in your tooling means you are the compiler

Runtime crashes like these "234" + 34

Monkey Patching, Duck Typing

* ~~Readability & Correctness~~

Dynamic types

* ~~Performance~~

* ~~Refactor~~

* ~~Contracts via Types~~

{Conclusion}

Rely on strongly typed language as a guide during refactoring/changes.

Use language & tools that ensures program correctness using Static Types

* Value correctness [NEXT SLIDE]

* Memory and thread safety

* mypy, go race, rust analyzer, flow, typescript, elm, etc

Avoid "Monotype" language, or make it compilation target

* For example, JS is more of a compilation target than a language. [NEXT SLIDE]

* Dropbox had to introduce type annotation in a language to make sense of their 4 million lines of code.

Coding = 10% new code & 90% changes in average. Look at VIM's default mode

Compiler errors

Compilation target

// Compile to JS from C
emcc -o main.js main.c

// Run via Node
node main.js

#include <stdio.h>

int sum(int* numbers, int size) {
  int result = 0;
  for (int i = 0; i < size; i++) {
    int number = numbers[i];
    result += number;
  }
  return result;
}

int main() {
  int numbers[5] = {1, 2, 3, 4, 5};

  int result = sum(numbers, 5);
  printf("The result is: %d\n", result);
}

C to JS/WASM using emscripten compiler tool chain

TOPIC {3}

Cloud & Microservices

Unicorn

Services with bounded concern

Scale

Distributed

Observability

Messaging

Gateways

Tools

AWS/Azure/GCP Services

Container orchestrations

Storage(s3/File Store, EBS/ABS)

Databases

SNS, SQS, Events Hub, Azure Service Bus

Durable & Steps

? Local development

Development Perspective

? Debugability

? Metrics, Logs & Traces

10x = 1/10x

How can we potentially fix this?

Microservice trait

? Compute

? Storage

? Functions

? Databases

? Observability

Note: Not a microservice architecture

? Cost

{CONCLUSION}

Tools & Solutions with open standards & protocols

Tools & solutions with ease of use in local settings

Tools & Solutions with managed support from cloud

Tools & Solutions with cloud agnostic property

Initial Architecture with Microservice trait ???

Thank {you}

Find me at https://robus.dev

Github handle https://github.com/robusgauli

Better engineering practices (l1)

By robus

Better engineering practices (l1)

Better engineering practices

About {me}

TOPIC {1}

Dependency Management

Onboarding process of dev by dev

Onboarding (1 month)

Onboarding (6 months)

Meanwhile your CI/CD pipeline

What did we miss?

Application Dependencies

Operational Dependencies

Can we fix it somehow?

Let's approach visually

Difference

What did it buy you?

What about CI?

Local

CI/CD

{Conclusion}

TOPIC {2}

Programming language, features & patterns

{Features & Patterns}

Step back

Requirements for computation

Example Case study of Golang

What did they do right?

Example Error pattern

{Conclusion}

{Programming Language}

Things that goes wrong

{Conclusion}

Compiler errors

Compilation target

TOPIC {3}

Cloud & Microservices

Unicorn

Tools

Development Perspective

How can we potentially fix this?

Microservice trait

{CONCLUSION}

Thank {you}

Find me at https://robus.dev

Github handle https://github.com/robusgauli

Better engineering practices (l1)

More from robus