2018 - DevOps Progress
DevOps Numbers before shutdown
| % availability | Minutes Uptime | Minutes Downtime |
|---|---|---|
| 99.99% | 609061 | 59 |
| Average sessions (month) | Average request (month) | Users on database (total) |
|---|---|---|
| 2,755,763 | ~66,960,000 | 1,254,959 |
7d - DevOps Numbers
| Total Songs processed | Songs to be processed | Songs processed (by min) |
|---|---|---|
| 9002845 | 37209958 | 25 |
| Total Songs on database | Average to a song being processed |
|---|---|
| 46212803 | 28.33s |
Time Frame 2018
May
Play shutdown
Time Frame 2018
May
June
- First research about lambda dev environment
- A lot of improvements to pro to handling the traffic for desktop
- Finished to sync all aws assets with terraform
- Migrate all jobs to Circle2.0
Play shutdown
Time Frame 2018
May
Play shutdown
June
July
- Reduced aws billing in 50%
- Lambda research to use CI/CD and development pipeline
- Poc using pgpool instead of pgbouncer
- Poc using k8s
- Poc Spark cluster
Time Frame 2018
May
Play shutdown
June
July
August
- Released backend services in lambda in production
- Released pgpoll in production
- Deleted python backend and ec2 instances for playweb
- Created NPM repository
Time Frame 2018
May
Play shutdown
June
July
August
September
- Finished k8s poc and planned first migration
- Created vault plugin to integrate with lambda environment
- Shutdown all ec2 instances
- Automatic feed consumption for artists and releases
Time Frame 2018
May
Play shutdown
June
July
August
September
- Finished k8s environment
- Migrated Jenkins to k8s
- Migrated Kibana to k8s
- Migrated wiki to k8s
- Migrated rundeck to k8s
October
Time Frame 2018
May
Play shutdown
June
July
August
September
October
November
- Pip private repository research
- Lambda for repopulate the queue for 7d (work in progress)
-
ReInvent2018
Time Frame 2018
May
Play shutdown
June
July
August
September
October
November
December
Nailed It!
2019 - DevOps Challenges
Next challenges
- Migrate all services to docker and serverless
- Improve services to 3rd managed party (rds, dynamoDB etc...)
- Improve Serverless environment
- Move microservices to serverless
- Improve Flask-philo framework to modules to be serverless friendly
Why Serverless?
- Easy life to devOps team - No patches, no server, no security OS level
- Easy life to Dev team - Small piece of codes, focus on the code, no vagrant all docker and code managed
- Easy integration with all services at AWS
- Easy and quick reaction to problems
- High availability and scaling automatically
- No tuning needed