Automating
Infrastructure

in the Cloud

Why automating?

• Less error prone
• Time saving
• Increased agility
• Security, security, security

Remember:

Rome wasn't built in a day

Small steps approach

• Automate provisioning of components
• Automate provisioning of whole servers
• Automate security hardening
• Automate provisioning of whole environments

Provisioning components

• Puppet/Chief/shell script
• Complete end-to-end provisioning (install & setup)
• Create reusable modules

Shell scripts

• Already exist
• Some times easier to write
• Underestimated

exec { 'web autodeploy setup':
  environment => ["SMART_TOOL_HOME=$smart_tool_home"],
  cwd         => '/smart/config/prod/services/',
  command     => '/bin/bash setup-webapp-autodeploy-user-and-service.sh',
  unless      => '/usr/bin/test -f /etc/init.d/smart-tool-webapp-autodeploy',
  timeout     => 1800,
}

Provisioning whole servers

• Built from reusable modules

node default, 'web' {
  user { "smart":
    ensure     => "present",
    managehome => true,
  } ->

  class { 'smart-sbt':
    user => 'smart',
    home => '/home/smart'
  }
  include 'smart-nginx'
  include 'smart-env'
  include 'smart-proserv-autodeploy'
  include 'smart-logstashforwarder'
  include 'smart-java8'
}

Configuration Management

• Static at provisioning time
• Read from hiera
• At runtime auto deploy scripts fetch env variables from Riak
• Stored in Riak as plain/text so ordinary curl works fine

Vagrant

• Build whole servers
• Sanity test servers
• Provisioning takes place on local machine
• There is a plugin for building machines in the cloud

Vagrant.configure("2") do |config|
  config.vm.box = "cloud-precise64"
  config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/precise/current/
precise-server-cloudimg-amd64-vagrant-disk1.box"
  config.vm.network :forwarded_port, guest: 443, host: 4567

  ["db", "web", "worker", "nat", "proserv", "restapi", "teamcity", "buildagent"]
  .each do |machinename|
	config.vm.define "#{machinename}" do |machine|
		machine.vm.host_name = "#{machinename}"

		machine.vm.provision :shell, :path => "main.sh"
		machine.vm.provision :puppet do |puppet|
			puppet.manifests_path = "puppet/manifests"
			puppet.options = "--modulepath 
/vagrant/puppet/modules/:/home/vagrant/.puppet/modules:/usr/share/puppet/modules"
			puppet.manifest_file = "#{machinename}-server.pp"
			puppet.hiera_config_path = "puppet/config/hiera.yaml"
			puppet.working_directory = "/vagrant/puppet/config/"
		end

		machine.vm.provider "virtualbox" do |v|
			v.memory = 2048
			v.cpus = 2
		end
	end
  end

Packer

• Provision whole servers in the cloud
• Turn those servers into cloud images

Security hardening

• Completely automated
• Automatically applied to all machines

{
	"type": "shell",
	"script": "security-hardening.sh",
	"execute_command": "{{ .Vars }} sudo -E sh '{{ .Path }}' 
{{ user `root_password` }} /tmp/{{ user `key_file_name` }}"
}

Last step in packer definition

Provisioning whole environments

• Amazon CloudFormation
• Describe whole environment in JSON
  • machines
  • security groups
  • load balancers
  • autoscaling groups
  • databases
• Using UserData we can inject runtime properties
• Parameterise the template (reusable template)

Maintaining automation

• Run it on a build server (scheduled)
• Vagrant can't be used on already virtualised machine
• Unless you use vagrant lxc provider :)

Start automating now...

It pays off
a thousand times!