Buzzwords
!=
happines
Introduction
Piotr Stapp
- Unique name - just sing a song: "Don't Stapp me know" ;)
- CEO @ Dotnetomaniak
- Author @ PoznajKubernetes & PoznajDocker
- ex - Senior software/DevOps/Architect @mBank/@FinAi/@Allegro
- SAFe System Principal Architect @ Demant Technology Center
- M.Sc (distction) Oxford Brooks University in Web Tech
- M.Sc. Warsaw University of Technology in Computer Science
- And ......
Agile -> Scrum -> SAFe
The truth
How to grow?
By day
By night
How to grow? My case
- Work (normal one)
- "Other" work
- commercial
- or not
- Reading
- Conferences (
RzemiosłoCraft-IT 💚) - Trainings
Case 1 - Testing!
About the project
- 10+ years on the market
- Niche one
- Old tech on the board including:
- logic in the DB
- logic in BL, View, DTO, ... -> everywhere
- No tests + no test cases
- Multiple versions
But it makes $$$
Input
- Test tutorials - ✅
- Books about testing - ✅
- Meetings "we should write test" - ✅
- Conferences & trainings - ✅
- Buzzwords: TDD, test coverage, gherkin, (x)Unit, selenium
Result: zero tests in solution - ✅
Code only
public HugeDTOWithSomeLogic UpdateSomething(HugeDTOWithSomeLogic dto)
{
var result = db.RunStoredProcedure(dto.GetImportantPartDetails());
if(result.Id==-1)
{
db.CreateNewImportantPart(result);
}
else
{
db.UpdateImportantPart(result);
}
if(dto.Total-dto.Checkout<0)
{
service.SendNotificationAboutMissingItems();
dto.Total = dto.Checkout;
}
///A lot of lines that do some calculations
return dto;
}
Propositions?
Case 2 - Startup "P"
About the project
- Greenfield but not exactly
- Old system in:
- CoffeeScript
- Google Maps + wkhtmltopdf
- Wordpress
- New system:
- Cheapest offer wins
- CTO as a Service 🙉🙊🙈
- Skeleton in the cupboard
- Azure hosting without Azure knowledge
- Azure managed by third party
Technologies in Azure
- Kubernetes - ✅
- WordPress on Kubernetes - ✅
- PVC for WordPress - ✅
- MySQL on VM - ✅
- Postgres on VM - ✅
- Manual backup for databases - ✅
- Azure sponsorship - ✅
"Typical" problems
- Google Maps versioning VS. wkhtmlpdf:
- QtWeb 3.8.x. from 2013
- Internet Explorer 11 has retired on June 15, 2022 -> Google Maps stopped support in April
- Azure costs after subscription ends ~60 euro per day
- Infinite backup
- Too big machines
- Application gateway
- ....
- New "free" subscription - migration
New project
- Should replace old one, but ...
- Cheapest offer wins, so ...
- Still AKS with DB on VM - shared infra == lower $$$
- New technologies:
- Python
- VUE.js
Better?
Case 3 - Startup "A"
Before we start
About the project
- Greenfield without anything
- MVP built by software house
- New team arrived partially
- Around 8 developers
Technologies
- Microservices - ~5 microservices - ✅
- Kubernetes - ✅
- Queue - RabbitMQ/Azure Service bus - ✅
- Web + mobile app - ✅
- Terraform - ✅
- Unit tests - ✅
- Integration tests - ✅
- Tilt - ✅
- No simple "build & run" in IDE - ✅
- Jenkins - ✅
- VPN to external services - ✅
- SQL in docker in local development - ✅
- Warehouse - ✅
#1 - How to deploy?
- A bunch od microservices
- Mobile app (with nice contract generator 💚)
- Problem with contracts
- Multiple tasks doing in parallel
#2 - Local env as prod
- TDD + K8s + tilt
- Projects based on "CQRS" framework
- SQL Server in docker + integration tests
- Tilt as magic:
- Discover changes
- Build incrementally with live update
- Run all tests (including integration on SQL)
- Everything in Terraform on local env too
Tilt
#3 - DevEx
- TDD + K8s + tilt == "easy" setup
- "CQRS" framework == even simple change is @$#$@
- SQL Server in docker == MacOS "loves" it
- Tilt as magic:
- Discover changes - "fast" builds
- Build incrementally with live update - HDD is empty
- Run all tests - what they check?
- Everything in Terraform on local env too - each dev know TF well, agree?
#3 - CI/CD
- Jenkins -> GitHub Actions
- Build per:
- commit
- PR
- request
- `kubectl apply` as fire & forget - what could go wrong?
- $$$ ??
#3 - CI/CD - deployment
#4 - VPN to "big corp"
- Infrastructure is rather complex
- Basic connection:
- setup VM and test from it
- VM in subnet
- public addresses through VPN 😅
- TF versus manual config
- IP blocking
Typical problems
- How to debug?
- What about contracts and versions
- TF succeed but the application didn't - now what?
- Delay in deployment
- Manual change in infra vs TF state
- VPN config
Case 4 - Project "X"
About the project
- Almost greenfield
- Dotnet for backend, python for ML, Angualr in front
- Microservices/modular monolith, but deployed as 2 apps in Docker:
- all dotnet services + front
- all python in one
- Separate DevOps team
- No cloud
Technologies
- Docker - ✅
- with ssh docker run
- no restart policies
- Jenkis - ✅
- Large models (~500MB - 1500MB) - ✅
DevOps versus Team
- Jenkis has problem with disk space:
- checkout large models
- big repositories
- build per branch
- poetry with global cache
- Deployment as "someone" wants
- No log access
Case 4 - Project "reuse"
About the project
- Typical datalake solution in Spark
- Complicated parsing logic already written in dotnet
- Spark supports easily
- JVM stuff
- Python stuff
DRY
- Rewriting == two
- How to?
- Two separated version
- Rewrite to common "platform" like C/C++
- Auto convert with cs2j or CSharpToPython
- Use custom container in Spark with preinstalled dotnet tool in shell
- Use API call from Spark
Case 5 - Project "compliance"
About the project
- 350 Messages per second
- Cloud infra
- Mix of anonymous and personal data
- GDRP and others
Technologies
- Azure "stuff" with full DevOps - ✅
- Serverless - ✅
- Event hub - ✅
- API management - ✅
- Test piramid - ✅
- Datalake with Data Bricks - ✅
Problem #1
- GDPR and others for personal data:
- removing
- migrating
- Data location - multiple regions
Problem #2
- Latency in different regions:
- EMEA
- AMER
- APAC
- Australia
- China?
- High availability
- Money vs speed vs HA
Problem #3 - compliance
- Personal vs anonymous
- Can we migrate anonymous?
- Can we correlate anonymous with personal?
- How to store data in files? One or multiple?
Case 6 - The most important one
About the project
- Small crew - 4+
- 16 years+ (started 2007-06-02)
- Challenging
Familly
The end
Questions?
Buzzwordy szczęścia nie dają
By Piotr Stapp
Buzzwordy szczęścia nie dają
- 50