Chef Metal


Manage Your Clusters





Like You Manage Your Machines







John Keiser
Chef Lead - Team Ronin
Resident Mad Scientist
@jkeiser2

Ye Olde Ops Playbook


Chef!

package 'apache2' do
  version '2.0'end
template '/etc/httpd.conf'
service 'apache2'

Chef!

  • Repeatable
  • Self-Documenting
  • Flexible
  • Self-Healing
  • Versionable
  • Readable
  • Testable

YE NEWE OPS PLAYBOOK


Clusters IN OUR DREAMS



real Clusters


the Cluster Litany


  • Version?
  • Test?
  • Re-create?
  • Manage Machines?
  • Add Capacity?


  • Info out of date :(
  • My memory sucks
  • Nothing is checked in anywhere
  • knife ec2 create --args=10,000,000
  • ssh? winrm? IP?
  • Which cloud? Or container? Or VM?

    • I JUST WANT TO CONVERGE


    My First Cluster


      machine 'db' do    recipe 'mysql'  end
      machine 'web1' do    recipe 'apache'  end

    My First Cluster

      > CHEF_DRIVER=vagrant:~/vms  > chef-client -z cluster.rbRecipe: talk1::cluster
      * machine[db] action create    - run vagrant up db    - write file /etc/chef/client.rb on db
          Starting Chef Client, version 11.8.2
      Recipe: mysql::default        * file[/etc/mysql.txt] action create
          - create new file /etc/mysql.txt      Chef Client finished, 1 resources updated    - run 'chef-client' on db
      * machine[web1] action create    - run vagrant up web1    - write file /etc/chef/client.rb on web1    - run 'chef-client' on web1 Chef Client finished, 2 resources updated


    What.


    • machine resource
    • Drivers

      WHAT.


      • machine resource
      • Drivers
        • Cloud
        • VMs
        • Containers
        • Metal (Razor, etc.)

      It's Chef



      chef-client

      It's Recipes

        private_key 'crypt'
        chef_data_bag_item 'x/y' do    secret_path 'crypt'  end
        machine 'mydb' do
          file '/etc/crypt', 'crypt'  end

      It's Unified


        metal login [node]
metal execute [node]
metal delete [node]
metal converge [node]

        Adding a New Machine


          machine 'db' do    recipe 'mysql'  end
          machine 'web' do    recipe 'apache'  end
         machine 'web2' do   recipe 'apache' end

        Idempotence!

          > chef-client -z -o cluster.rb Recipe: talk1::cluster
          * machine[db] action create (up to date)  * machine[web1] action create (up to date)  * machine[web2] action create    - run vagrant up web2    - write file /etc/chef/client.rb on web2
              Starting Chef Client, version 11.8.2
      Recipe: apache::default        * file[/etc/apache.txt] action create
          - create new file /etc/apache.txt      Chef Client finished, 1 resources updated    - run 'chef-client' on web2
         Chef Client finished, 1 resource updated


        Recycle, Reuse, Renew



        Let's Do EC2



        CHEF_DRIVER=fog:AWS:default

        Yay

          > chef-client -z cluster.rb


        Continuous Integration



        Continuous Integration

        • Check in Metal recipe
        • chef-client on Jenkins node
        • ...


        Provisioning nodes

        • Upload cluster cookbook
        • Wait 5 minutes
        • ...



        Growing the cluster


          machine 'db' do            recipe 'mysql'  end
          machine 'web' do    recipe 'apache'  end
         machine 'web2' do   recipe 'apache' end


        Upscaling


        Upscaling

        • Upscaling is just adding new nodes
        • Chef recipes are code
        • Use a variable for your for loop
        • Drive it from your own monitor data 

        1.upto(num_webservers) do  machine "webserver#{i}" do
    recipe 'apache'    tag 'webserver'  endend

        Downscaling

        Delete what you don't need.

           search('tags:webserver') do |node|    node.name =~ /^webserver(\d+)$/    if $1.to_i > num_webservers      machine node.name do        action :delete      end    end  end

        Parallelism


        
  with_machine_batch 'my_batch'

  1.upto(50) do |i|
    machine "webserver#{i}" do
      recipe 'apache'
    end
  end

        Provisioners

        There are lots already
        chef-metal-fog
        chef-metal-vagrant
        chef-metal-ssh
        chef-metal-lxc
        chef-metal-docker
        chef-metal-vsphere
        Between them, we support
        EC2 - DigitalOcean - Openstack - SoftLayer - VSphere
        VirtualBox - VMWare Fusion
        Bare Metal (SSH, no PXE yet)
        LXC - Docker

        Making a Driver

        • Extend the Driver interface
        • Implement 4 methods
        • ...


        driver.allocate_machine(...)



        Reserves / purchases a machine
        Optionally starting spinup
        Short execution time


        driver.ready_machine(...)


        Starts up machine
        Waits for machine to be ready
        Returns Machine object


        The machine object


        • Execute commands
        • Manipulate files
        • Converge machine

        It's already there


        UnixMachine
        WindowsMachine

        Transports

        • Used by Unix/WindowsMachine
        • Handle execution and file manipulation

        Already there:
        SSHTransport
        WinRMTransport
        LXCTransport
        DockerTransport

        Convergence Strategies


        Handle installation of chef and converge
        "idempotent bootstrap"

        Already there:
        InstallCached
        InstallSh
        InstallMsi
        NoConverge


        class MyProvisioner < Provisioner  def allocate_machine(ah, machine, options)    id = machine.location['id']    if id
              server = lookup_server(id)    else
              server = create_server(options)
      machine.location = {        'id' => server.id      }    end
          endend

        Parallelism


        allocate_machines

        ready_machines

        Other methods


        connect_to_machine

        delete_machine

        stop_machine

        MACHINE Options


        Shape of the machine

        Freeform Hash

        Driver decides what goes in it

        Not stored anywhere


        Example Vagrant options:
        
  machine_options :vagrant_options => { 'vm.box' => 'precise64' },
    :vagrant_config => 'config.vm.synced_folder "~/code", "/mnt/code"'

        Example EC2 options:
        
  machine_options :bootstrap_options => {
    :image_id => 'ami-2f726546',    :flavor_id => 'c1.xlarge'
          }

        Different Machines Are Different


         machine 'webserver' do
           driver 'fog:GoogleCloud:MyAccount'   recipe 'webserver'
         end
 with_driver 'fog:AWS:default', :region => 'us-west-1' do
   machine 'monitor' do
             recipe 'monitor'
           end
           machine 'another_webserver' do
             machine_options :region => 'us-east-1'     recipe 'webserver'
           end end



        1.0 Soon.


        After 1.0:


        • More Drivers
        • Lightweight Provisioning
        • More Cluster Resources
        • Key Management

        How you can help


        • Write drivers
        • Use it
        • Send feedback
        • File and fix bugs











         https://github.com/opscode/chef-metal 

        Chef Metal

        By John Keiser

        Made with Slides.com

        Chef Metal

        • 9,667

        More from John Keiser