Application Deployments on our Terremark Cloud

(A Play in 3 Acts)

Act I

In which the deployment is planned

Scene 1: Where is the deployment going to be?

  • Give Cloud Engineering the capacity needs - CPU, VM, Storage
  • Get back details about which environment and subnet

Typical hiccups:

- Not enough available capacity and have to wait until there is

- Deployment team doesn't know enough about Verizon cloud and our setup (ssh, keys, vpn etc.)

Scene 2: Application team preparation

  • Are my release artifacts available?
  • Are my Chef cookbooks ready?
  • Are there any Devops or CE functions to be used?

Typical hiccups:

- Cookbooks not sync'ed automatically to Verizon. Should we?

- RPMs are not in the "release" repo or some other reposync issues

- Insufficient information about Devops or CE functions

- Information about Devops / CE functions arriving late in the deployment cycle

Act II

In which the deployment is performed

Scene 1: Composition of deployment artifacts

  • Chef Environment
  • Vagrantfile - VM templates (CPU, Memory, Storage) & Chef Runlists
  • DNS names for VMs decided

Typical hiccups:

- Terremark Vagrant provider does not deploy correctly on Windows and MacOS

- Chef secrets are open to deployer

Scene 2: VM deployment

  • Smoke test VM creation
  • VMs created one at a time
  • DB backups restored or (for in-place upgrades, schema is upgraded)

Typical hiccups:

- VM creation fails because of environment issues - Verizon problem or no IP because of DHCP issues

- Unknown issues in Devops / CE functions during deployment

- DB upgrade / install failures are quite common

- Subnet not configured according to standard template with typical ACLs to Services environment

Scene 3: Stitching it all together

  • DNS A and PTR records for VMs
  • DNS CNAMEs for services
  • haproxy configuration
  • Firewall ACLs for services
  • AWS Route 53 / Incapsula conf
  • Vormetric conf for DBs

Typical hiccups:

- None of this is automated

- Only a few (1 or 2) know how to do this correctly and have the right privileges. Bits and pieces are known to some others

Act III

In which the players troubleshoot and test

Scene 1: "It still doesn't work" - Debugging

  • Connectivity issues including firewall configuration problems
  • IAM integration issues
  • haproxy not working as expected

Typical hiccups:

- Insufficient troubleshooting skills of deployment team

Summarizing the issues

  • Devops cookbooks need more QA and documentation as well as support team for the application teams
  • More automation and standardization of DB platforms and service deployment so that apps don't have to re-invent it each time [but this is a lot of work...]
  • Automate the parts that are not automated - haproxy, DNS, Chef Environment generation, Vormetric/Encryption, Firewall, Public IP

Some things haven't propagated to the App teams

  • PAM using FreeIPA - available but not implemented
  • PAM for databases - not implemented yet
  • Vault for secret management - available but all cookbooks need to be modified to adopt this
  • Percona instead of MySQL master-slave
  • IAM Shibboleth install process - metadata exchange
  • Dockerized IAM available for development but not used at all
Made with Slides.com