Buzzwords
!=
happines

Introduction

Piotr Stapp

  • Unique name - just sing a song: "Don't Stapp me know" ;)
  • CEO @ Dotnetomaniak
  • Author @ PoznajKubernetes & PoznajDocker
  • ex - Senior software/DevOps/Architect @mBank/@FinAi/@Allegro
  • SAFe System Principal Architect @ Demant Technology Center
  • M.Sc (distction) Oxford Brooks University in Web Tech
  • M.Sc. Warsaw University of Technology in Computer Science
  • And ......

https://www.linkedin.com/in/piotrstapp/

Agile -> Scrum -> SAFe

The truth

How to grow?

By day

By night

How to grow? My case

  • Work (normal one)
  • "Other" work
    • commercial
    • or not
  • Reading
  • Conferences (Rzemiosło Craft-IT 💚)
  • Trainings

Case 1 - Testing!

About the project

  • 10+ years on the market
  • Niche one
  • Old tech on the board including:
    • logic in the DB
    • logic in BL, View, DTO, ... -> everywhere
  • No tests + no test cases
  • Multiple versions

 

But it makes $$$

Input

  • Test tutorials - ✅
  • Books about testing - ✅
  • Meetings "we should write test" - ✅
  • Conferences & trainings - ✅
  • Buzzwords: TDD, test coverage, gherkin, (x)Unit, selenium

 

Result: zero tests in solution - ✅

Code only


public HugeDTOWithSomeLogic UpdateSomething(HugeDTOWithSomeLogic dto)
{
    var result = db.RunStoredProcedure(dto.GetImportantPartDetails());
    if(result.Id==-1)
    {
    	db.CreateNewImportantPart(result);
    }
    else
    {
    	db.UpdateImportantPart(result);
    }
    if(dto.Total-dto.Checkout<0)
    {
    	service.SendNotificationAboutMissingItems();
        dto.Total = dto.Checkout;
    }
    ///A lot of lines that do some calculations
    
    return dto;
}

Propositions?

Case 2 - Startup "P"

About the project

  • Greenfield but not exactly
  • Old system in:
    • CoffeeScript
    • Google Maps + wkhtmltopdf
    • Wordpress
  • New system:
    • Cheapest offer wins
    • CTO as a Service 🙉🙊🙈
  • Skeleton in the cupboard
    • Azure hosting without Azure knowledge
    • Azure  managed by third party

Technologies in Azure

  • Kubernetes - ✅
  • WordPress on Kubernetes - ✅
  • PVC for WordPress - ✅
  • MySQL on VM  - ✅
  • Postgres on VM - ✅
  • Manual backup for databases - ✅
  • Azure sponsorship - ✅

"Typical" problems

  • Google Maps versioning VS. wkhtmlpdf:
    • QtWeb 3.8.x. from 2013
    • Internet Explorer 11 has retired on June 15, 2022 -> Google Maps stopped support in April
  • Azure costs after subscription ends ~60 euro per day
    • Infinite backup
    • Too big machines
    •  Application gateway
    • ....
  • New "free" subscription - migration

New project

  • Should replace old one, but ...
  • Cheapest offer wins, so ...
  • Still AKS with DB on VM - shared infra == lower $$$
  • New technologies:
    • Python
    • VUE.js

Better?

Case 3 - Startup "A"

Before we start

About the project

  • Greenfield without anything
  • MVP built by software house
  • New team arrived partially
  • Around 8 developers

Technologies

  • Microservices - ~5 microservices - ✅
  • Kubernetes - ✅
  • Queue - RabbitMQ/Azure Service bus - ✅
  • Web + mobile app - ✅
  • Terraform - ✅
  • Unit tests - ✅
  • Integration tests - ✅
  • Tilt - ✅
  • No simple "build & run" in IDE - ✅
  • Jenkins - ✅
  • VPN to external services - ✅
  • SQL in docker in local development - ✅
  • Warehouse - ✅

#1 - How to deploy?

  • A bunch od microservices
  • Mobile app (with nice contract generator 💚)
  • Problem with contracts
  • Multiple tasks doing in parallel

#2 - Local env as prod

  • TDD + K8s + tilt
  • Projects based on "CQRS" framework
  • SQL Server in docker + integration tests
  • Tilt as magic:
    • Discover changes
    • Build incrementally with live update
    • Run all tests (including integration on SQL)
  • Everything in Terraform on local env too

Tilt

#3 - DevEx

  • TDD + K8s + tilt == "easy" setup
  • "CQRS" framework == even simple change is @$#$@
  • SQL Server in docker == MacOS "loves" it
  • Tilt as magic:
    • Discover changes - "fast" builds
    • Build incrementally with live update - HDD is empty
    • Run all tests - what they check?
  • Everything in Terraform on local env too - each dev know TF well, agree?

#3 - CI/CD

  • Jenkins -> GitHub Actions
  • Build per:
    • commit
    • PR
    • request
  • `kubectl apply` as fire & forget - what could go wrong?
  • $$$ ??

#3 - CI/CD - deployment

#4 - VPN to "big corp"

  • Infrastructure is rather complex
  • Basic connection:
    • setup VM and test from it
    • VM in subnet
    • public addresses through VPN 😅
  • TF versus manual config
  • IP blocking

Typical problems

  • How to debug?
  • What about contracts and versions
  • TF succeed but the application didn't - now what?
  • Delay in deployment
  • Manual change in infra vs TF state
  • VPN config

Case 4 - Project "X"

About the project

  • Almost greenfield
  • Dotnet for backend, python for ML, Angualr in front
  • Microservices/modular monolith, but deployed as 2 apps in Docker:
    • all dotnet services + front
    • all python in one
  • Separate DevOps team
  • No cloud

Technologies

  • Docker - ✅
    • with ssh docker run
    • no restart policies
  • Jenkis - ✅
  • Large models (~500MB - 1500MB) - ✅

DevOps versus Team

  • Jenkis has problem with disk space:
    • checkout large models
    • big repositories
    • build per branch
    • poetry with global cache
  • Deployment as "someone" wants
  • No log access

Case 4 - Project "reuse"

About the project

  • Typical datalake solution in Spark
  • Complicated parsing logic already written in dotnet
  • Spark supports easily
    • JVM stuff
    • Python stuff

DRY

  • Rewriting == two 
  • How to?
    • Two separated version
    • Rewrite to common "platform" like C/C++
    • Auto convert with cs2j or CSharpToPython
    • Use custom container in Spark with preinstalled dotnet tool in shell
    • Use API call from Spark

Case 5 - Project "compliance"

About the project

  • 350 Messages per second
  • Cloud infra
  • Mix of anonymous and personal data
  • GDRP and others

Technologies

  • Azure "stuff" with full DevOps - ✅
  • Serverless - ✅
  • Event hub - ✅
  • API management - ✅
  • Test piramid - ✅
  • Datalake with Data Bricks - ✅

Problem #1

  • GDPR and others for personal data:
    • removing
    • migrating
  • Data location - multiple regions

Problem #2

  • Latency in different regions:
    • EMEA
    • AMER
    • APAC
    • Australia
    • China?
  • High availability
  • Money vs speed vs HA

Problem #3 - compliance

  • Personal vs anonymous
  • Can we migrate anonymous?
  • Can we correlate anonymous with personal?
  • How to store data in files? One or multiple?

Case 6 - The most important one

About the project

  • Small crew - 4+
  • 16 years+ (started 2007-06-02)
  • Challenging

Familly

The end

Questions?

https://craft-it.pl/feedback/

Buzzwordy szczęścia nie dają

By Piotr Stapp

Made with Slides.com

Buzzwordy szczęścia nie dają

  • 67

More from Piotr Stapp