A Journey into DevOps and Drupal

Ashok Modi - DrupalCamp LA 2017

About Presenter

  • BTMash
  • Engineer - CARD.com
  • Drupal - 11 years
  • Systems Programmer - 12 years
  • Enjoy thinking about
    team workflows in devops
  • Likes shorter talks
    • Lot of slides in this one
      though

About presentation

  • Largely deals with customized stack
  • If you use Pantheon/Acquia/Platform/etc
    • Great! You're already ahead
      of the game.
    • Hopefully you find a useful
      nugget of information here.
  • Devops-fu at CARD.com not perfect
    • Lots of room for improvement.
    • But also happy with how far we've come :)

About CARD.com

  • Offer reloadable debit cards
  • Great brands - Star Trek, Garfield, etc
  • 20 employees - 7 engineers
  • Largely built on Open Sourced Tools
    • Drupal 7
    • Ionic / AngularJS
    • R
    • Go (Image processing)

Many requirements for our app

PCI Requirements

  • Card numbers (at least last 4)
  • PIN passes through systems
  • SSN
  • Full Name
  • Birthday

Interact mainly with 3rd party APIs

  • Blueshift (email)
  • Bank Processing
  • Zendesk (support tickets)
  • FB (ads and marketing)

Functional Requirements

  • Over 1000 tests
    • Use Behat (primary)
    • PHPUnit (secondary)

And we've simplified a lot of things

But there were things to learn (while at CARD.com and before CARD.com)

History (some personal)

  • Started with FTP

Wow, it sucks

  • Insecure
  • Permissions hell
  • Slow as ****
  • Usually only option

And we run updates manually!

Lesson: Use version control

And we got version control (svn/git)

  • ssh server
  • git pull

DONE!!!

Wait, no we're not

  • What about database updates?
  • Compiling code?
  • Clear the cache?
  • Backups?

Its more complicated nowadays

  • ssh server
  • cd directory
  • git pull
  • composer update (all ur phps)
  • yarn (all ur frontends)
  • drush updb -y
  • drush updb -y --entity-updates (D8)
  • drush cc/cr all (D7/D8)

And you forget a step

****, time to rollback and try this again...

Lesson: Create a script

Created a script

  • Consistently same steps
  • Could handle multiple servers

****, we're a team

  • Need other dependencies installed
  • Script is growing
  • Team is growing, need some
    accountability
  • What about dev/stage/test
    sites???

2 lessons

Lesson: Code is not configuration. Use a configuration tool.

Use Ansible/Chef/Puppet/Salt/Docker

  • Or a combination of the bunch
  • Lots of recipes can be found
    specific to Drupal or whatever
    your stack might need

Currently Ansible + Docker

  • Vault to store sensitive credentials securely
  • Groups for specific servers
  • Tags to run/exclude specific tasks
  • Ansible for our servers
  • Docker for some R stuff
    • Docker for local, which is created by Ansible

So I have everything now!

  • Nope
  • What about testing?
  • What about accountability?
  • What about the database
    and local setup?

Lesson: Sharing assets is helpful

Sharing is caring

  • If you know how to provision servers
    • Provision local as well!
      • Virtualbox
      • Docker
      • DevDesktop/Kalabox
  • Create database dumps
  • Save them to dropbox
  • Team gets db dumps via dropbox

What about that sensitive data

  • Replicate data to separate database first
  • Run sanitize scripts
    • https://www.drupal.org/project/paranoia
    • There is a sql sanitize module in there
  • Create dump of that and save to dropbox
    • 40+GB file comes down to 200MB
  • Make the dropbox directory a one-way
    street for other accounts to read but not write.

Easy to script locally to bring over DB

Team has committed project for local setup via docker + shell scripts

What about files?

  • https://www.drupal.org/project/stage_file_proxy
    • Uses files on production instead of locally :)

Lesson 4: Generalized build server

Install Jenkins/GoCD/Bamboo

  • User logins
  • Server has scripts and dependencies
    to do your business
  • Shows pass/fail
  • Can be triggered on git push/pull/etc
    • Ours is manual (will explain why)
  • Create db snapshots and save to
    dropbox automatically
  • We could even add testing?

What is that about testing?

  • Jenkins now has capability for creating and managing different branches of your code to validate testing.
    • UI is funky as hell (blueocean is nicer)
    • PCI requirements mean dev environment cannot talk to live.
      • Its debatable but you may need a separate build environment altogether?
      • Infrastructure for this can be a PITA

Lesson: Find a good approach to testing

Gitlab/Github/Bitbucket

  • Hooks for adding services
  • Pull request based workflows
  • All solid options for testing

Wait, those are ways to host code, not test!

You're right

Use a testing service that integrates into those

  • Gitlab has built-in testing
  • Use Jenkins and its hooks for tests
  • Use a service
    • TravisCI (github only)
    • CircleCI
    • CodeShip
    • ProboCI (we use managed
      version at CARD.com)

ProboCI

  • Upload db dump to proboci using their tools
    • Along with any other assets
  • Each PR = new test suite runs
  • Green = manual review by peer on team
  • Completely removed our need for a separate
    dev/stage/UA site (for now anyways)

What if I don't need testing?

Lesson: You do

You just don't know it yet

Use a test framework

  • Behat/Codeception for Functional/BDD
  • PHPUnit for unit testing

Behat/Codeception

  • Integrated browser testing
    • We test with headless Chrome
  • Tests are relatively easy to grok
  • Easy to implement
  • https://github.com/jhedstrom/drupalextension
    • For Behat and Drupal to natively work
      during testing.

This has gone on for a while now...

Where are you going with this?

Our workflow

Local Install

  • Install Docker and git
  • Install dev-docker repo
  • Follow instructions in README
  • Ready to go!

Local - New Feature request

  • Create new branch
  • Refresh db
    • cmd to refresh db in README
  • Loop
    • Do work
    • Commit
    • Push
  • Open PR
    • Repeat loop as necessary

Local Workflow cont'd

  • All gitflow
  • Review on github
    • Automated tests on Probo run w/
      pass/fail
    • Merge into dev branch.
  • Create new branch to merge to master
    • Once merged, use jenkins to deploy
      to production using one click
  • Monitor for errors on newrelic

Easily have capacity for many deploys to production

If only we knocked out code that quickly!

What happens behind the scenes?

  • Jenkins runs an ansible playbook called Ansistrano
    • http://ansistrano.com/
      • Makes it easy to ensure we don't get code conflicts
      • Site still remains running if failed dependencies
      • Continues serving old code
    • Easier to manage going back a version of code
  • Updates settings vars, compiles frontend, etc

Can use any tool

  • You could deploy new docker containers if that is your setup
  • You can use chef-deploy
  • Just do a git pull if that's all your code needs
  • Jenkins frontend abstracts away the actual deployment behind the scenes

Lessons so far

  • Use version control
    • That alone should open your eyes to a lot of stuff
  • Make it easy to set up servers
    • Make it easy to set local dev environment
  • Share needed assets (database, files)
    • Make it easy to refresh locally with new data
  • Encourage testing
    • Lots of tools to help with that
  • Abstract workflows with tools
    • Make it easy to deploy code

Pain points for us

  • Many different dashboards for our toolsets
    • Jenkins for some code/infrastructure
    • R dashboards for marketing
    • Newrelic/graylog for code
    • Nagios for uptime monitoring
  • Two different provisioning tools
    • Ansible versus Docker

Next steps

  • Figure out our next needs
  • Consolidate our dashboards
    • Datadog
    • Move all to graylog
    • More research needed
  • Move all to docker
    • Maybe
  • Chatops to deploy?
    • YOLO

References

  • The Goal
    • https://en.wikipedia.org/wiki/The_Goal_(novel)
  • The Phoenix Project
    • https://itrevolution.com/book/the-phoenix-project/
  • The Devops Handbook
    • http://itrevolution.com/devops-handbook

Thank you

Questions?