A Journey into DevOps and Drupal
Ashok Modi - DrupalCamp LA 2017
Engineer - CARD.com
Drupal - 11 years
Systems Programmer - 12 years
Enjoy thinking about
team workflows in devops
Likes shorter talks
Lot of slides in this one
Largely deals with customized stack
If you use Pantheon/Acquia/Platform/etc
Great! You're already ahead
of the game.
Hopefully you find a useful
nugget of information here.
Devops-fu at CARD.com not perfect
Lots of room for improvement.
also happy with how far we've come :)
Offer reloadable debit cards
Great brands - Star Trek, Garfield, etc
20 employees - 7 engineers
Largely built on Open Sourced Tools
Ionic / AngularJS
Go (Image processing)
Many requirements for our app
Card numbers (at least last 4)
PIN passes through systems
Interact mainly with 3rd party APIs
Zendesk (support tickets)
FB (ads and marketing)
Over 1000 tests
Use Behat (primary)
And we've simplified a lot of things
But there were things to learn (while at CARD.com and before CARD.com)
History (some personal)
Started with FTP
Wow, it sucks
Slow as ****
Usually only option
And we run updates manually!
Lesson: Use version control
And we got version control (svn/git)
Wait, no we're not
What about database updates?
Clear the cache?
Its more complicated nowadays
composer update (all ur phps)
yarn (all ur frontends)
drush updb -y
drush updb -y --entity-updates (D8)
drush cc/cr all (D7/D8)
And you forget a step
****, time to rollback and try this again...
Lesson: Create a script
Created a script
Consistently same steps
Could handle multiple servers
****, we're a team
Need other dependencies installed
Script is growing
Team is growing, need some
What about dev/stage/test
Lesson: Code is not configuration. Use a configuration tool.
Or a combination of the bunch
Lots of recipes can be found
specific to Drupal or whatever
your stack might need
Currently Ansible + Docker
Vault to store sensitive credentials securely
Groups for specific servers
Tags to run/exclude specific tasks
Ansible for our servers
Docker for some R stuff
Docker for local, which is created by Ansible
So I have everything now!
What about testing?
What about accountability?
What about the database
and local setup?
Lesson: Sharing assets is helpful
Sharing is caring
If you know how to provision servers
Provision local as well!
Create database dumps
Save them to dropbox
Team gets db dumps via dropbox
What about that sensitive data
Replicate data to separate database first
Run sanitize scripts
There is a sql sanitize module in there
Create dump of that and save to dropbox
40+GB file comes down to 200MB
Make the dropbox directory a one-way
street for other accounts to read but not write.
Easy to script locally to bring over DB
Team has committed project for local setup via docker + shell scripts
What about files?
Uses files on production instead of locally :)
Lesson 4: Generalized build server
Server has scripts and dependencies
to do your business
Can be triggered on git push/pull/etc
Ours is manual (will explain why)
Create db snapshots and save to
We could even add testing?
What is that about testing?
Jenkins now has capability for creating and managing different branches of your code to validate testing.
UI is funky as hell (blueocean is nicer)
PCI requirements mean dev environment cannot talk to live.
Its debatable but you may need a separate build environment altogether?
Infrastructure for this can be a PITA
Lesson: Find a good approach to testing
Hooks for adding services
Pull request based workflows
All solid options for testing
Wait, those are ways to host code, not test!
Use a testing service that integrates into those
Gitlab has built-in testing
Use Jenkins and its hooks for tests
Use a service
TravisCI (github only)
ProboCI (we use managed
version at CARD.com)
Upload db dump to proboci using their tools
Along with any other assets
Each PR = new test suite runs
Green = manual review by peer on team
Completely removed our need for a separate
dev/stage/UA site (for now anyways)
What if I don't need testing?
Lesson: You do
You just don't know it yet
Use a test framework
Behat/Codeception for Functional/BDD
PHPUnit for unit testing
Integrated browser testing
We test with headless Chrome
Tests are relatively easy to grok
Easy to implement
For Behat and Drupal to natively work
This has gone on for a while now...
Where are you going with this?
Install Docker and git
Install dev-docker repo
Follow instructions in README
Ready to go!
Local - New Feature request
Create new branch
cmd to refresh db in README
Repeat loop as necessary
Local Workflow cont'd
Review on github
Automated tests on Probo run w/
Merge into dev branch.
Create new branch to merge to master
Once merged, use jenkins to deploy
to production using one click
Monitor for errors on newrelic
Easily have capacity for many deploys to production
If only we knocked out code that quickly!
What happens behind the scenes?
Jenkins runs an ansible playbook called Ansistrano
Makes it easy to ensure we don't get code conflicts
Site still remains running if failed dependencies
Continues serving old code
Easier to manage going back a version of code
Updates settings vars, compiles frontend, etc
Can use any tool
You could deploy new docker containers if that is your setup
You can use chef-deploy
Just do a git pull if that's all your code needs
Jenkins frontend abstracts away the actual deployment behind the scenes
Lessons so far
Use version control
That alone should open your eyes to a lot of stuff
Make it easy to set up servers
Make it easy to set local dev environment
Share needed assets (database, files)
Make it easy to refresh locally with new data
Lots of tools to help with that
Abstract workflows with tools
Make it easy to deploy code
Pain points for us
Many different dashboards for our toolsets
Jenkins for some code/infrastructure
R dashboards for marketing
Newrelic/graylog for code
Nagios for uptime monitoring
Two different provisioning tools
Ansible versus Docker
Figure out our next needs
Consolidate our dashboards
Move all to graylog
More research needed
Move all to docker
Chatops to deploy?
The Phoenix Project
The Devops Handbook