How I wrote a Chef Server

while waiting in line at Best Buy



John Keiser

Development Lead at Opscode

Things that annoy me


Takes too long to get started with Chef

chef-client and knife tests are hard

chef-solo is ... unique

I like rewriting things
Common solution to these problems: 

Write a tiny Chef server.

How hard could it be?


A Chef Server ...


Just stores things

One JSON for each thing

Nodes, roles, environments, cookbooks ...

Processing is done on the client (mostly)

chef-client and knife grab stuff with REST

I can do this.



Recipe for a chef server

  • Laptop
  • Folding chair
  • 12 pack of soda
  • Desire for cheap TV

  1. Get in line
  2. Unfold laptop and chair
  3. Mix in soda slowly
  4. ???
  5. Chef server

What I made


In-memory storage
Fast startup
No dependencies
Chef 11 compatible
REST server

chef-zero

I didn't do ...

Security
Persistence
Scalability

These are left as an exercise for the reader.

A beginning: /roles


CRUD for role JSON: 

{
  "name": "base",
  "description": "That Which Belongs To Us",
  "override_attributes": {
    "build_essential": {
      "compiletime": true
    }
  },
  "run_list": [
    "recipe[build_essential]",
    "role[core-packages]"
  ]
}

Endpoints


  • GET /roles
  • POST /roles
  • GET /roles/web
  • PUT /roles/web
  • DELETE /roles/web

The Code: Read


require 'sinatra'
require 'json'
before { content_type 'application/json' }
roles = {}

get('/roles/*')    { |role| roles[role] }

get('/roles') do
  result = {}
  roles.keys.each { |role| result[role] = url("/roles/#{role}") }
  JSON.generate(result)
end

The Code: Write


post('/roles') do
  role_body = request.body.read
  role = JSON.parse(role_body)['name']
  roles[role] = role_body
end

put('/roles/*') { |role| roles[role] = request.body.read }

delete('/roles/*') { |role| roles.delete(role) }

It works!

$ knife role create blah -s http://127.0.0.1:4567
Created role[blah]
$ knife role list -s http://127.0.0.1:4567
blah
$ knife role show blah -s http://127.0.0.1:4567
chef_type:           role
default_attributes:
description:
env_run_lists:
json_class:          Chef::Role
name:                blah
override_attributes:
run_list:

Repeat for


  • Clients
  • Cookbooks
  • Data Bags
  • Environments
  • Nodes
  • Roles
  • Users

Wait, cookbooks too?


Cookbook = JSON with version and list of files.

Liar.

OK, cookbooks aren't QUITE that simple.
  • Two-stage upload (Sandboxes)
  • Depsolver

Two-stage Upload



  1. Ask the server where it wants the files
  2. Upload the files
  3. Upload the cookbook JSON pointing at the files

Depsolver


  1. Finds cookbook dependencies
  2. Finds latest possible cookbook version
  3. Tells nodes what cookbooks to get

Search


Chef Server:
Indexed by solr
Query is solr syntax
Results not always fresh

Chef Zero:
No index
Slow, accurate and fresh

The code is too large to fit in this margin.

Pedant is fucking awesome

https://github.com/opscode/chef-pedant
2712 tests and counting
 

What can it do?


  • Quick way to learn Chef
  • knife and chef-client tests
  • Testing cookbooks (test-kitchen)
  • And more ...

Using chef-zero in tests

require 'chef_zero/rspec'
describe 'knife list' do
  include ChefZero::RSpec
  when_the_chef_server "has plenty of stuff in it" do
    cookbook 'cookbook1', '1.0.0', { 'metadata.rb' => '' }
    data_bag 'bag1', { 'item1' => {}, 'item2' => {} }
    environment 'environment1', {}
    role 'role1', {}
    
    it "knife cookbook list lists cookbooks" do
      knife('cookbook list').should_succeed "cookbook1\n"
    end
  end
end

A Modest Proposal


chef-client --local-server

  • No need for configuration or knife.rb
  • Load your repo into a chef-zero
  • Supports all recipes
  • Use data bags, search, roles, environments
  • Save node after chef-client run (convergence!)

Testing in "Production"

Ignorance Is Bliss
  1. knife download / from production
  2. chef-zero
  3. knife upload -s http://127.0.0.1:8889
  4. vagrant up
  5. knife bootstrap -s http://127.0.0.1:8889

Chef on a Stick


  • Stick chef-zero and your repo on a stick ...
  • /mnt/UsbStick/chef-client --local-server
  • Carry it to the next server
  • Repeat

The cloud is now a stick.

Getting and Using It


gem install chef-zero
chef-zero

knife.rb:
chef_server_url "http://127.0.0.1:8889"

Repository: https://github.com/jkeiser/chef-zero
gem install chef-zero
chef-zero

Questions?


@jkeiser2
jkeiser@opscode.com
Made with Slides.com