Advertisement agency meets a cloud

Migrating 260 domains from shared hosting to GKE

root@localhost:~# whoami

maciek@amplification.pl

https://www.linkedin.com/in/macieksawicki/

Disclaimer

Drama Characters

The Company

End client. Most prominent over-the-counter drugs manufacture in Poland. They have about 30 brands. Each brand has its own logo, own brand manager, and website.

Foomedica

The creative agency specialized in pharmaceutical products marketing.

They were responsible for the development and hosting of The-Company's websites

They used dedicated servers.

oror

A creative agency that was going to take responsibility for The Company digital portfolio

They love Google Cloud and they come up with the idea to migrate from dedicated servers to GCP

What? Why? How?

What?

~40 websites

~260 domains

80% PHP
Ruby, Nodejs

MySQL

DIY 'CMS'

DIY oAuth server

The Company controlled (almost) all domains

NS were delegated to Foomedica

(digression) OODA loop

The OODA loop is the cycle observe–orient–decide–act, developed by military strategist and United States Air Force Colonel John Boyd. Boyd applied the concept to the combat operations process, often at the operational level during military campaigns.

Reconciliation loop

Desired state

Move everything to (Google) cloud
Reasonable infrastructure costs
Operated by two ops after hours
Easy deployments

Simple plan

Plan v0.0.1

Inventorization
Obtaining all assets, access to all systems
Detailed analysis of obtained assets
Reading documentation
Prepare base infrastructure on GCP
Scheduling maintenance window, communication with all stakeholders
Deployment on staging
Actual migration

a few things didn't go as planned

a few problems...

no documentation at all
no access to old infrastructure
limited contact with Foomedica
no git - just tar.gz dumps
some dumps were creating by archiving data on servers, but there were broken symlinks
some dumps were actually from git but versions mismatched with the content of the live websites
many bugs (broken links, etc.) on live websites that we found after the migration
forget about deployment scripts or dependencies documentation

Biggest mistakes during planing phase:

no proper audit of old infrastructure (we weren't able to prove that some things were already broken before migration)
made friends with Foomedica admin too late

What now?

One more thing

chapter #0 lean project management infrastructure

project management tools

Slack
Trello
G Suite

chapter #1 GCP setup

GCP check list

Enable billing,
Enable billing alerts,
Set up accounts and permissions,
Use two-factor authentication,
Decide which region do you plan to use,
If you use VMs plan your network topology,
Buy support plan before going live,
Optional - enable billing export to BigQuery

standing on the shoulders of giants

https://github.com/terraform-google-modules/terraform-google-project-factory
https://github.com/GoogleCloudPlatform/professional-services

chapter #2 - deploy GitLab

chapter #3 - DNS zones

DNS import

gcloud dns record-sets import -z=examplezonename --zone-file-format path-to-example-zone-file

We didn't get zone files for all domains :(

DNS verification

Simple python script with dnspython lib to compare DNS entries on live name servers with GCP name servers

chapter #4 - simple monitoring

simple monitoring

We decided to use Status Cake
Simple python script to import all domains using API
Slack notifications
Saved our back many times
We have a love-hate relationship with this tool

chapter #5 containerization

containerization

I have discovered a truly marvelous solution to this, which this slide is too small to contain.
put all stuff into GitLab
use GitLab Ci for building the images

chapter #6 k8s time

deployment in GKE

deploy GKE cluster (using gcloud command)
deploy Cloud SQL
deploy Cloud SQL proxy, Nginx ingress, Kube lego
deploy all websites
adjust probes and resources
NFS VM

chapter #7 SEO/brand protection domains

domains?

each website had one main domain + 5-6 additional domains (ex: .eu, .com, .com.pl)
ingress per brand
we wrote simple sidecar in go for managing redirections (to control everything via env vars in deployments definitions)

chapter #8 mission accomplished?

00:00

update to point to Kubernetes ingress instead of Foomedica's servers
Profit?

Bonus level - what happened after the migration?

and they lived happily ever after?

Move everything to (Google) cloud - ✔
Reasonable infrastructure costs - ✔
Operate by two ops after hours - ✔
Easy deployments - ✔