Chef Metal


Manage Your Clusters





Like You Manage Your Machines







John Keiser
Chef Lead - Team Ronin
Resident Mad Scientist
@jkeiser2

Ye Olde Ops Playbook


Chef!

package 'apache2' do
version '2.0'end
template '/etc/httpd.conf'
service 'apache2'

Chef!

  • Repeatable
  • Self-Documenting
  • Flexible
  • Self-Healing
  • Versionable
  • Readable
  • Testable

YE NEWE OPS PLAYBOOK


Clusters IN OUR DREAMS



real Clusters


the Cluster Litany


  • Version?
  • Test?
  • Re-create?
  • Manage Machines?
  • Add Capacity?


  • Info out of date :(
  • My memory sucks
  • Nothing is checked in anywhere
  • knife ec2 create --args=10,000,000
  • ssh? winrm? IP?
  • Which cloud? Or container? Or VM?
  • I JUST WANT TO CONVERGE


    My First Cluster


    machine 'db' do recipe 'mysql' end
    machine 'web1' do recipe 'apache' end

    My First Cluster

      > CHEF_DRIVER=vagrant:~/vms  > chef-client -z cluster.rbRecipe: talk1::cluster
    * machine[db] action create - run vagrant up db - write file /etc/chef/client.rb on db
    Starting Chef Client, version 11.8.2 Recipe: mysql::default * file[/etc/mysql.txt] action create   - create new file /etc/mysql.txt  Chef Client finished, 1 resources updated - run 'chef-client' on db
    * machine[web1] action create - run vagrant up web1 - write file /etc/chef/client.rb on web1 - run 'chef-client' on web1 Chef Client finished, 2 resources updated


    What.


    • machine resource
    • Drivers

      WHAT.


      • machine resource
      • Drivers
        • Cloud
        • VMs
        • Containers
        • Metal (Razor, etc.)

      It's Chef



      chef-client

      It's Recipes

        private_key 'crypt'
      chef_data_bag_item 'x/y' do secret_path 'crypt' end
      machine 'mydb' do
      file '/etc/crypt', 'crypt' end

      It's Unified


        metal login [node] metal execute [node] metal delete [node] metal converge [node]

        Adding a New Machine


        machine 'db' do recipe 'mysql' end
        machine 'web' do recipe 'apache' end
        machine 'web2' do recipe 'apache' end

        Idempotence!

          > chef-client -z -o cluster.rb Recipe: talk1::cluster
        * machine[db] action create (up to date) * machine[web1] action create (up to date) * machine[web2] action create - run vagrant up web2 - write file /etc/chef/client.rb on web2
        Starting Chef Client, version 11.8.2 Recipe: apache::default * file[/etc/apache.txt] action create   - create new file /etc/apache.txt  Chef Client finished, 1 resources updated - run 'chef-client' on web2
        Chef Client finished, 1 resource updated


        Recycle, Reuse, Renew



        Let's Do EC2



        CHEF_DRIVER=fog:AWS:default

        Yay

          > chef-client -z cluster.rb


        Continuous Integration



        Continuous Integration

        • Check in Metal recipe
        • chef-client on Jenkins node
        • ...


        Provisioning nodes

        • Upload cluster cookbook
        • Wait 5 minutes
        • ...



        Growing the cluster


        machine 'db' do
        recipe 'mysql' end
        machine 'web' do recipe 'apache' end
        machine 'web2' do recipe 'apache' end


        Upscaling


        Upscaling

        • Upscaling is just adding new nodes
        • Chef recipes are code
        • Use a variable for your for loop
        • Drive it from your own monitor data

        1.upto(num_webservers) do machine "webserver#{i}" do recipe 'apache' tag 'webserver' endend

        Downscaling

        Delete what you don't need.

           search('tags:webserver') do |node|    node.name =~ /^webserver(\d+)$/    if $1.to_i > num_webservers      machine node.name do        action :delete      end    end  end

        Parallelism


        
          with_machine_batch 'my_batch'
        
          1.upto(50) do |i|
            machine "webserver#{i}" do
              recipe 'apache'
            end
          end
        
        

        Provisioners

        There are lots already
        chef-metal-fog
        chef-metal-vagrant
        chef-metal-ssh
        chef-metal-lxc
        chef-metal-docker
        chef-metal-vsphere
        Between them, we support
        EC2 - DigitalOcean - Openstack - SoftLayer - VSphere
        VirtualBox - VMWare Fusion
        Bare Metal (SSH, no PXE yet)
        LXC - Docker

        Making a Driver

        • Extend the Driver interface
        • Implement 4 methods
        • ...


        driver.allocate_machine(...)



        Reserves / purchases a machine
        Optionally starting spinup
        Short execution time


        driver.ready_machine(...)


        Starts up machine
        Waits for machine to be ready
        Returns Machine object


        The machine object


        • Execute commands
        • Manipulate files
        • Converge machine

        It's already there


        UnixMachine
        WindowsMachine

        Transports

        • Used by Unix/WindowsMachine
        • Handle execution and file manipulation

        Already there:
        SSHTransport
        WinRMTransport
        LXCTransport
        DockerTransport

        Convergence Strategies


        Handle installation of chef and converge
        "idempotent bootstrap"

        Already there:
        InstallCached
        InstallSh
        InstallMsi
        NoConverge


        class MyProvisioner < Provisioner  def allocate_machine(ah, machine, options)    id = machine.location['id']    if id
        server = lookup_server(id) else
        server = create_server(options) machine.location = { 'id' => server.id } end
        endend

        Parallelism


        allocate_machines

        ready_machines

        Other methods


        connect_to_machine

        delete_machine

        stop_machine

        MACHINE Options


        Shape of the machine

        Freeform Hash

        Driver decides what goes in it

        Not stored anywhere


        Example Vagrant options:
        
          machine_options :vagrant_options => { 'vm.box' => 'precise64' },
            :vagrant_config => 'config.vm.synced_folder "~/code", "/mnt/code"'

        Example EC2 options:
        
          machine_options :bootstrap_options => {
            :image_id => 'ami-2f726546',    :flavor_id => 'c1.xlarge'
        }

        Different Machines Are Different


        machine 'webserver' do
        driver 'fog:GoogleCloud:MyAccount' recipe 'webserver'
        end with_driver 'fog:AWS:default', :region => 'us-west-1' do machine 'monitor' do
        recipe 'monitor'
        end
        machine 'another_webserver' do
        machine_options :region => 'us-east-1' recipe 'webserver'
        end end



        1.0 Soon.


        After 1.0:


        • More Drivers
        • Lightweight Provisioning
        • More Cluster Resources
        • Key Management

        How you can help


        • Write drivers
        • Use it
        • Send feedback
        • File and fix bugs











         https://github.com/opscode/chef-metal 

        Chef Metal

        By John Keiser

        Chef Metal

        • 7,536
        Loading comments...

        More from John Keiser