About CARD.com

• Offer reloadable debit cards
• Great brands - Star Trek, Garfield, etc
• 20 employees - 7 engineers
• Largely built on Open Sourced Tools
  • Drupal 7
  • Ionic / AngularJS
  • R
  • Go (Image processing)

Many requirements for our app

PCI Requirements

• Card numbers (at least last 4)
• PIN passes through systems
• SSN
• Full Name
• Birthday

Interact mainly with 3rd party APIs

• Blueshift (email)
• Bank Processing
• Zendesk (support tickets)
• FB (ads and marketing)

Functional Requirements

• Over 1000 tests
  • Use Behat (primary)
  • PHPUnit (secondary)

And we've simplified a lot of things

But there were things to learn (while at CARD.com and before CARD.com)

History (some personal)

• Started with FTP

Wow, it sucks

• Insecure
• Permissions hell
• Slow as ****
• Usually only option

And we run updates manually!

And we got version control (svn/git)

• ssh server
• git pull

DONE!!!

Wait, no we're not

• What about database updates?
• Compiling code?
• Clear the cache?
• Backups?

Its more complicated nowadays

• ssh server
• cd directory
• git pull
• composer update (all ur phps)
• yarn (all ur frontends)
• drush updb -y
• drush updb -y --entity-updates (D8)
• drush cc/cr all (D7/D8)

And you forget a step

****, time to rollback and try this again...

Created a script

• Consistently same steps
• Could handle multiple servers

****, we're a team

• Need other dependencies installed
• Script is growing
• Team is growing, need some
  accountability
• What about dev/stage/test
  sites???

2 lessons

Use Ansible/Chef/Puppet/Salt/Docker

• Or a combination of the bunch
• Lots of recipes can be found
  specific to Drupal or whatever
  your stack might need

Currently Ansible + Docker

• Vault to store sensitive credentials securely
• Groups for specific servers
• Tags to run/exclude specific tasks
• Ansible for our servers
• Docker for some R stuff
  • Docker for local, which is created by Ansible

So I have everything now!

• Nope
• What about testing?
• What about accountability?
• What about the database
  and local setup?

Sharing is caring

• If you know how to provision servers
  • Provision local as well!
    • Virtualbox
    • Docker
    • DevDesktop/Kalabox
• Create database dumps
• Save them to dropbox
• Team gets db dumps via dropbox

What about that sensitive data

• Replicate data to separate database first
• Run sanitize scripts
  • https://www.drupal.org/project/paranoia
  • There is a sql sanitize module in there
• Create dump of that and save to dropbox
  • 40+GB file comes down to 200MB
• Make the dropbox directory a one-way
  street for other accounts to read but not write.

Easy to script locally to bring over DB

Team has committed project for local setup via docker + shell scripts

What about files?

• https://www.drupal.org/project/stage_file_proxy
  • Uses files on production instead of locally :)

Install Jenkins/GoCD/Bamboo

• User logins
• Server has scripts and dependencies
  to do your business
• Shows pass/fail
• Can be triggered on git push/pull/etc
  • Ours is manual (will explain why)
• Create db snapshots and save to
  dropbox automatically
• We could even add testing?

What is that about testing?

• Jenkins now has capability for creating and managing different branches of your code to validate testing.
  • UI is funky as hell (blueocean is nicer)
  • PCI requirements mean dev environment cannot talk to live.
    • Its debatable but you may need a separate build environment altogether?
    • Infrastructure for this can be a PITA

Gitlab/Github/Bitbucket

• Hooks for adding services
• Pull request based workflows
• All solid options for testing

Wait, those are ways to host code, not test!

You're right

Use a testing service that integrates into those

• Gitlab has built-in testing
• Use Jenkins and its hooks for tests
• Use a service
  • TravisCI (github only)
  • CircleCI
  • CodeShip
  • ProboCI (we use managed
    version at CARD.com)

ProboCI

• Upload db dump to proboci using their tools
  • Along with any other assets
• Each PR = new test suite runs
• Green = manual review by peer on team
• Completely removed our need for a separate
  dev/stage/UA site (for now anyways)

What if I don't need testing?

Lesson: You do

You just don't know it yet

Use a test framework

• Behat/Codeception for Functional/BDD
• PHPUnit for unit testing

Behat/Codeception

• Integrated browser testing
  • We test with headless Chrome
• Tests are relatively easy to grok
• Easy to implement
• https://github.com/jhedstrom/drupalextension
  • For Behat and Drupal to natively work
    during testing.

This has gone on for a while now...

Where are you going with this?

Our workflow

Local Install

• Install Docker and git
• Install dev-docker repo
• Follow instructions in README
• Ready to go!

Local - New Feature request

• Create new branch
• Refresh db
  • cmd to refresh db in README
• Loop
  • Do work
  • Commit
  • Push
• Open PR
  • Repeat loop as necessary

Local Workflow cont'd

• All gitflow
• Review on github
  • Automated tests on Probo run w/
    pass/fail
  • Merge into dev branch.
• Create new branch to merge to master
  • Once merged, use jenkins to deploy
    to production using one click
• Monitor for errors on newrelic

Easily have capacity for many deploys to production

If only we knocked out code that quickly!

What happens behind the scenes?

• Jenkins runs an ansible playbook called Ansistrano
  • http://ansistrano.com/
    • Makes it easy to ensure we don't get code conflicts
    • Site still remains running if failed dependencies
    • Continues serving old code
  • Easier to manage going back a version of code
• Updates settings vars, compiles frontend, etc

Can use any tool

• You could deploy new docker containers if that is your setup
• You can use chef-deploy
• Just do a git pull if that's all your code needs
• Jenkins frontend abstracts away the actual deployment behind the scenes

Pain points for us

• Many different dashboards for our toolsets
  • Jenkins for some code/infrastructure
  • R dashboards for marketing
  • Newrelic/graylog for code
  • Nagios for uptime monitoring
• Two different provisioning tools
  • Ansible versus Docker

Next steps

• Figure out our next needs
• Consolidate our dashboards
  • Datadog
  • Move all to graylog
  • More research needed
• Move all to docker
  • Maybe
• Chatops to deploy?
  • YOLO