Team DIA's recent
work
a.k.a.
"why so much DI, and so little A"?
Agenda
- DIA Team Scope
- Immutable Infrastructure - intro
- Immutable Infrastructure - demo
- Immutable Infrastructure and Team DIA's work
Team scope
Build and maintain an infrastructure...
...that can support services, which:
- collect,
- transform,
- analyse,
- deliver
data
Team scope
Build and maintain an infrastructure...
...that can support services, which:
- collect,
- transform,
- analyse,
- deliver
data
Team scope
Build and maintain an infrastructure...
...that can support services, which:
- collect,
- transform,
- analyse,
- deliver
data
Infrastructure building - Guiding principles
- Immutable infrastructure
- "Immutable infrastructure is comprised of immutable components that are replaced for every deployment, rather than being updated in-place." - Link
- Infrastructure as Code
- Immutable infrastructures allow you define your components (the software they run, their bootstrapping logic etc.) in code that you can check into and track in version control.
- Added transparency, self-documenting projects, easier maintenance, less risky software updates, reproducible environments
A practical example - The past
- Company uses MySQL version 5.5 to store its transactional data
- Developer would like to use new feature that was introduced in MySQL 5.7
- Developer opens ticket with SysOps
- Developer waits
- Developer waits
- SysOps picks up ticket and evaluates the risks of the update, devises complex recovery strategies should things blow up
- Developer waits
- SysOps logs onto server that runs MySQL 5.5 and tries to manually update to the new version
A practical example - The past
- Problems:
- Speed (the lack thereof)
- Touching running systems is high risk - could result in data loss, extensive down times
- Obscure, potentially undocumented changes
- All the changes are platform-dependent - the code that the developer was thinking about using after the update might not work with the specific MySQL installation
A practical example - Immutable infrastructure
- In the Immutable Infrastructure vocabulary, MySQL is a component.
- Actually, MySQL 5.5 and MySQL 5.7 are both totally separate, independent components.
- The concept of Immutable Infrastructure was popularised by Docker, which allows you to package your components as platform-independent executable binaries.
- Demo time!
A practical example - Immutable infrastructure
- Advantages:
- Fast
- Safe - easy rollback
- Self-documenting component definitions
- Reproducible - all you need is docker
Our work in the Immutable Infrastructure domain
- Defined tons of components
- Each component can be run
either as a Docker container or an
AWS EC2 instance - i.e. most of
these projects contain two set of
scripts
Our work in the Immutable Infrastructure domain (cont.)
- Image building vs. image provisioning using the docker example:
- docker build: Create component/image with all necessary software preinstalled on it
- docker run: Run a specific version of the component/image with some configuration specified at runtime (e.g. MYSQL_ROOT_PASSWORD)
- In the demo, we run the containers by hand
- But in reality we also want to codify this process for the same reasons we want to codify our components' contents
Our work in the Immutable Infrastructure domain - Where are we know?
- 80% of components and 90% of core components are done
- 80% of the provisioning set-up is done
- What will we have at 100%?
- A robust real-time data pipeline that reads all data change events in the MySQL servers and pushes them to Kafka
- A fully functional system of distributed services.
- Terraform demo!