DevOps cult

Applied           Style DevOps Kung Fu

Originaly by  Adam Jacob / CTO of Chef / @adamhjk

 

Chef Style DevOps Kung Fu

  • On Github: http://github.com/chef/devops-kungfu
  • Join by PR
  • We work together to develop our principles, forms, and applications
  • Start your own school by forking, removing the names, and changing the content
Creative Commons License

Roadmap

  1. Materialize the vision ​​
  2. Align early  feedback
  3. Balance innovation with real needs
  4. Design  themes, with an outcome for each
  5. Distill those into features, and validate

Vision:

Deploy and operate all infrastructures by code, as applications are code too.

As traditional IT becomes "truck" and cloud native is a "car". Practice DevOps and apply cloud architectures to acquire expertise, tools and operational skills to drive.

Purpose

  • every team needs an back-end systems
  • every solution has an infrastructure
  • every project require repeatable deployment
  • recognized approach cross teams

Believes

  • brings velocity to drive robust solutions     
  • self document infrastructure as a code
  • none laptop has enough memory
  • dirty hands touching ssh is pain in the ass
  • fix, batch, deploy, re-use, automate

Choosed languages and tools that fit the job

This means that Chef can:

  • Build  Deterministic Servers 
  • Deploy and Configure Applications
  • Deploy Infrastructures
  • Update Applications
  • Run audits and notifications

Chef is a configuration management

and automation platform

Overview

How it works

chef-provisioning

recipes

knife-softlayer

knife-ec2

knife-google

Ecosystem

Chef Development Kit (ChefDK)

$ chef generate app cookbook_name
$ chef generate attribute
$ chef generate recipe
$ chef generate lwrp

$ chef-apply name_of_recipe.rb
  • chef 12.0.3
  • chefspec 4.2.0
  • chef-provisioning 0.18
  • chef-provisioning-aws 0.2.1
  • chef-provisioning-fog 0.12
  • chef-vault 2.4.0
  • knife-spork 1.5.0
  • rubocop 0.28.0

Ecosystem

$ knife search node "platform:centos AND languages_ruby_version:1.8*"

$ knife search node "chef_environment:production AND platform:ubuntu"

$ knife ssh 'name:* AND NOT name:chef*' 'sudo postmap -p /etc/postfix/transport'

$ knife bootstrap --environment dev --node-name catalog.lab.dev 10.10.50.15 \
    --ssh-password passw0rd --run-list "role[base]" --server-url "https://10.10.50.2" \
    --json-attributes '{ "hostname": "catalog","name": "catalog.lab.dev", \
    "system":{ "short_hostname":"catalog", "domain_name":"lab.dev" } }' \
    --secret-file .chef/encrypted_data_bag_secret


$ knife node run_list add ora.kb.dev "role[git], recipe[backup::gitlab]" 

$ knife role from file roles/*.rb  
                     
$ knife environment from file staging.rb                     

knife

** KNIFE COMMANDS **

bootstrap, client, configure, cookbook, cookbook site, data bag, delete, deps,
diff, download, edit, environment, exec, list, node, raw, recipe_list, role,
search, serve, show, ssh, ssl check, ssl fetch, status, tag, upload, user

** NODE COMMANDS **
knife node bulk delete REGEX (options)
knife node create NODE (options)
knife node delete NODE (options)
knife node edit NODE (options)
knife node environment set NODE ENVIRONMENT
knife node from file FILE (options)
knife node list (options)
knife node run_list add [NODE] [ENTRY[,ENTRY]] (options)
knife node run_list remove [NODE] [ENTRY[,ENTRY]] (options)
knife node run_list set NODE ENTRIES (options)
knife node show NODE (options)

Ecosystem

** SOFTLAYER COMMANDS **

knife softlayer datacenter list (options)
knife softlayer datacenter show DATACENTER
knife softlayer flavor list (options)
knife softlayer global ip list (options)
knife softlayer image list
knife softlayer key pair create
knife softlayer key pair list
knife softlayer server create (options)
knife softlayer server destroy (options)
knife softlayer server list (options)
knife softlayer server relaunch <NODE NAME> [<NODE NAME>]
knife softlayer vlan create
knife softlayer vlan list (options)
knife softlayer vlan show ID (options)
  • knife-softlayer
  • knife-ec2
  • knife-azure
  • knife-windows

knife plugins

Ecosystem

chef supermarket

Ecosystem

include_recipe 'apache2'                                                                                   

user 'userdoc' do
  supports manage_home: true
  comment 'userdoc (bind user)'
  gid 'users'
  home '/home/userdoc'
  password '$1$9p26dAeZ$qZ7P4XW.ASDG4DSD3234vK0'
end

directory '/var/www/userdoc' do
  mode 00775
  owner 'apache2'
  group 'users'
  recursive true
end

web_app 'userdoc' do
  server_name "udoc.#{node['domain']}"
  server_aliases ["userdoc.#{node['system']['domain_name']}"
  docroot '/var/www/userdoc'
  cookbook 'apache2'
  directory_options 'Indexes FollowSymLinks'
end

cookbooks, recipes

cookbook
|
├── .kitchen.yml
├── Berksfile
├── metadata.rb
├── attributes
│   └── default.rb         
├── recipes
│   └── default.rb
└── templates
    └── default
        └── index.html.erb
# metadata.md

name 'gitzone'
maintainer 'Petr Michalec'
maintainer_email 'epcim@apealive.net'
license 'Apache 2.0'
description 'Installs/Configures gitzone managed zone files for Bind'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.0.15'

recipe 'gitzone::default',   'Install and fully configure gitzone using the install and configure recipes'
recipe 'gitzone::install',   'Install gitzone scripts'
recipe 'gitzone::configure', 'Configure gitzone (including BIND and default zones)'
recipe 'gitzone::configure_zonefile', 'Deploy zone files'

supports 'ubuntu'
supports 'centos'

depends 'bind'
depends 'git'
depends 'sudo'
depends 'system'
depends 'build-essential'
depends 'simple_iptables'

Ecosystem

require 'chef/provisioning'
with_driver 'fog:OpenStack'

...

# PRE ALLOCATE
machine_batch do
  %w(chef web repo ci).each do |m|
    machine "#{m}.lab.ci" do
      action :allocate
      add_machine_options bootstrap_options: PER_MACHINE_BOOTSTRAP_OPTIONS[m] || {}
      retries 1
    end
  end
end

# BOOTSTRAP/CONVERGE NODE
machine 'chef.lab.ci' do
  ohai_hints 'openstack' => '{}'
  attribute %w(apt compile_time_update), true
  attribute %w(resolver nameservers), search(:node, 'role:dns').map \
    { |x| x.automatic.ipaddress } unless search(:node, 'role:dns').empty?
  recipe 'lab_base::default'
  role 'chef'
  add_machine_options bootstrap_options: PER_MACHINE_BOOTSTRAP_OPTIONS['chef'] || {}
  files('/etc/chef/encrypted_data_bag_secret' => "#{dtbgsec}")
  action [:ready, :converge]
  converge true
end

​Chef-Zero

Simple, easy-run, fast-start in-memory Chef server for testing and solo purposes

chef-provisioning, chef-zero

​Chef-Provisioning

A collection of resources that enable the creation of machines and machine infrastructures using the chef-client.

Ecosystem

Analytics & audits

# in audit recipe
control_group 'ssh' do
  control 'password authentication' do
    it 'is disabled' do
      expect(file('/etc/ssh/sshd_config')).to_not \
        match(/^PasswordAuthentication\s+yes/i)
    end
  end
end

# in analytics rule
rules 'failed-audit'
  rule on run_control_group
  when
    status != 'success'
  then
    notify('slack', "{
      'username': 'Audit Alarm', 'text': "{{message.name}} (cookbook {{message.cookbook_name}})\n
      had '{{message.number_failed}}' failed audit test(s) on node '{{message.run.node_name}}'"
    }")
  end
end

Chef WebUI

Hands On

Chef workflow

"simple" web app

PartA, develpop cookbooks/repo/recipes

https://asciinema.org/a/31362

 

PartB - execute serverspec/inspec audit

https://asciinema.org/a/31360

lab_ cookbooks

  • lab_gitlab
  • lab_audit
  • lab_backup
  • lab_base
  • lab_openvpn
  • lab_chef-server
  • lab_bind
  • lab_drone
  • lab_nagios
  • lab_firewall
  • lab_trac
  • lab_redmine
  • lab_openldap
  • lab_artifactory
  • lab_maven
  • lab_docker
  • lab_subversion
  • lab_postfix
  • lab_jenkins
  • lab_sensu
  • lab_oracle
  • lab_websphere
  • lab_owncloud
  • ...

wrappers

  • percona ( MySQL XtraDB Percona cluster )
  • sensu ( Sensu monitoring + Graphite, Grafana )
  • gitzone ( git managed BIND zones )

applications

chef-repo-#{ENV}

  • Chef cookbook as many others
  • Contain definition for roles, environments, data bags
  • Contain chef provisioning recipes
    • for applications, environments, etc ...
  • Contain chef audit recipes
  • ChefSpec, ServerSpec/Specinfra V2 tests  
  • KitchenCI integration tests
  • Rakefile, scripts, core configuration

chef-repo

workflow

rake -T

rake berks                     # Berks install/update/vendor
rake bootstrap:chefonly        # Boostrap CHEF ONLY with SSH
rake bootstrap:ci              # Boostrap CI against OpenStack env
rake bootstrap:gtshub          # Boostrap gtshub using SSH
rake bootstrap:setup           # Bootstrap local cfg for chef-provisioning setup
rake bootstrap:test            # Boostrap TEST against OpenStack env
rake bootstrap:vagrant         # Bootstrap Vagrant env
rake chef:hosts                # Create a hosts.yml file based on knife information
rake chef:hosts_local          # Create a hosts.yml file based on local ChefZero information
rake chef:nodes_local          # Print ip + fqdn from nodes records
rake convert                   # Convert ruby classes to json
rake convert:env_to_json       # Convert ruby environments from ruby to json, creating/overwriting json files
rake convert:metadata_to_json  # Convert all metadata from ruby to json
rake convert:role_to_json      # Convert ruby roles from ruby to json, creating/overwriting json files
rake destroy:ci                # Destroy CI environment
rake dev:bootstrap             # Run task bootstrap:ci in local DEV environment
rake dev:cls                   # Clean up local chef-zero instances
rake dev:setup                 # Setup local development ENV variables
rake foodcritic                # Foodcritic
rake knife:bootstrap           # Bootstrap new node using knife
rake lint                      # Lint & syntax checks
rake provision:chefonly        # Provision CHEF server with SSH
rake provision:ci              # Provision CI environment against Openstack
rake provision:gtshub          # Provision gtshub environment with SSH
rake provision:test            # Provision TEST environment with Openstack
rake rubocop                   # Run RuboCop style and lint checks
rake rubocop:auto_correct      # Auto-correct RuboCop offenses
rake spec                      # Run serverspec to all hosts
rake vagrant:create            # Create the Vagrant machine
rake vagrant:destroy           # Stop and delete the Vagrant machine
rake vagrant:login             # Log into the Vagrant machine via SSH
rake vagrant:provision         # Provision the Vagrant machine with Chef

knife-zero-repo

  • For one-time / simple deployments
  • Runs Chef-Zero in memory
  • Filesystem as persistent storage
  • Commands
    • bootstrap
    • converge
    • chef_client
    • diagnose 

Chef cookbook testing

Continuous Integration test Continuous Deployment cycle

  • ​Cookbooks
    • Unit test (generated by ChefDK using ChefSpec)
    • Integration tests (KitchenCI, ServerSpec)
    • Functional tests (ServerSpec, InSpec)
  • Chef-Repo
    • Functional tests (Rake, ServerSpec, InfraSpec, InSpec)
    • Smoke tests (Integration tests, Audit recipes)

CI Overview

.kitchen.yml

---
# https://docs.chef.io/config_yml_kitchen.html

driver:
    name: vagrant
    use_sudo: false

provisioner:
  name: chef_zero                                                           
  cookbook_path: ['.','cookbooks', 'test/cookbooks']

platforms:
    - name: centos-7.1
      run_list:
        - recipe[selinux::disabled]
    - name: ubuntu-14.04

suites:
  - name: default
    run_list:
        - recipe[lab_base]
        - recipe[lab_gitlab::db]
        - recipe[lab_gitlab::default]
    attributes:
        apt:
            compile_time_update: true
        mysql:
          server_root_password: test
        gitlab:
          database:
            host: 127.0.0.1
            password: test

# vim: sw=2 ts=2 sts=2

Drone.io

.drone.yml

image: epcim/drone-chef-ci:dind

env:
    - CI=drone
    - LABREPO=https://gitlab.lab.local/chef-cookbooks

script:
    - source /root/.bash_profile
    - eval "$(chef shell-init bash)"
    - export LANG="en_US.UTF-8"
    - export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem

    - chef exec berks install
    - chef gem install chef-rewind

    - wrapdocker

    - chef exec rspec spec
    - chef exec kitchen test

    - chef exec foodcritic "-f any -f ~FC014 -f ~FC015 -f ~FC024 -f ~FC043" .
    - chef exec rubocop -a || chef exec rubocop .

notify:
  slack:
    webhook_url: 'https:#hooks.slack.com/services/T0SDDASD43/BSDFD84QY/buISDF436dsfjsdlEoo57qO'
    username: 'drone.io'
    channel: '#lab-cookboks'
    on_started: false
    on_success: true
    on_failure: false

Drone.io addons

Monitoring & analytics

  • Chef Analytics rules
  • Chef Audit recipes
  • Nagios / Sensu
  • ELK stack, Grafana, ...

Only alert on what is actionable

  • Get the attention of the right humans
  • As few alerts as possible
  • Routed to the people who can take action
  • Start with the is it up alert

Chef Audit


# it's just a "serverspec"...

control_group 'SSH Service compliance' do

  control 'ssh service' do
    it 'should be listening on port 22' do
      expect(port(22)).to be_listening
    end
  end

  control 'ssh configuration' do
    it 'ssh disables root logins' do
      expect(file('/etc/ssh/sshd_config')).to contain('PermitRootLogin no')
    end

    it 'is disabled' do
      expect(file('/etc/ssh/sshd_config')).to_not \
        match(/^PasswordAuthentication\s+yes/i)
    end
  end
end


Chef InSpec

# use basic tests
describe package('nginx') do
  it { should be_installed }
end

# extend tests with metadata
control '01' do
  impact 0.7
  title 'Verify nginx service'
  desc 'Ensures nginx service is up and running'
  describe service('nginx') do
    it { should be_enabled }
    it { should be_installed }
    it { should be_running }
  end
end

# implement os dependent tests
web_user = 'www-data'
web_user = 'nginx' if os[:family] == 'centos'

describe user(web_user) do
  it { should exist }

Sensu

Grafana dashboards

Let's rock

Demo


Questions


Feedback

Demo


Questions


Feedback

Demo


Questions


Feedback

Backup slides

Dockerfile

FROM jpetazzo/dind
MAINTAINER Petr Michalec <epcim@apealive.net>

RUN apt-get update
RUN apt-get install -qqy    curl \
                            sudo \
                            git \
                            mercurial \
                            subversion \
                            ca-certificates \
                            locales \
                            jq

RUN echo 'en_US.UTF-8 UTF-8'>>/etc/locale.gen
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV DEBIAN_FRONTEND noninteractive


## CHEF DK ###########################
RUN curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -P chefdk
ENV PATH /opt/chefdk/bin:/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Make Chef DK the primary Ruby/Chef development environment.
RUN echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
RUN eval "$(chef shell-init bash)"


RUN chef gem install kitchen-docker
RUN chef gem install kitchen-openstack
RUN chef gem install chef-sugar
RUN chef gem install chef-rewind
RUN chef gem install serverspec
RUN chef gem install infratester


# berks pre-fetch some common soup of cookbooks
RUN mkdir /tmp/fake_cookbook; cd $_
RUN echo "name 'fake_cookbook'\nmaintainer 'fake_cookbook'\nlicense 'fake_cookbook'\ndescription 'fake_cookbook'\nversion '0.0.1'" > metadata.rb
RUN echo "source 'https://supermarket.chef.io'\nmetadata\n\n" > Berksfile
RUN echo "cookbook '7-zip'\ncookbook 'apache2'\ncookbook 'apt'\ncookbook 'ark'\ncookbook 'bluepill'\ncookbook 'build-essential'\ncookbook 'certificate'\ncookbook 'chef-client'\ncookbook 'chef_handler'\ncookbook 'chef_ruby'\ncookbook 'chef-sugar'\ncookbook 'chef-vault'\ncookbook 'cron'\ncookbook 'database'\ncookbook 'device-mapper'\ncookbook 'git'\ncookbook 'minitest-handler'\ncookbook 'modules'\ncookbook 'ncurses'\ncookbook 'nginx'\ncookbook 'ntp'\ncookbook 'ohai'\ncookbook 'openssh'\ncookbook 'openssl'\ncookbook 'packagecloud'\ncookbook 'pacman'\ncookbook 'perl'\ncookbook 'rbenv'\ncookbook 'readline'\ncookbook 'resolver'\ncookbook 'resource-control'\ncookbook 'rsyslog'\ncookbook 'ruby'\ncookbook 'ruby_build'\ncookbook 'runit'\ncookbook 'subversion'\ncookbook 'sudo'\ncookbook 'sysctl'\ncookbook 'system'\ncookbook 'ulimit'\ncookbook 'users'\ncookbook 'windows'\ncookbook 'xml'\ncookbook 'yum'\ncookbook 'yum-epel'\ncookbook 'zlib'\n" >> Berksfile
RUN chef exec berks install
RUN cd -


## FIX UP'S ##########################
RUN chmod -R 0440 /etc/sudoers
RUN chmod -R 0440 /etc/sudoers.d
# workaround (drone.io has no way yet to modify this image before git clone happens)
RUN git config --global http.sslverify false




VOLUME /var/lib/docker
CMD ["wrapdocker"]

Paleo man DevOps cult

By Petr Michalec

Paleo man DevOps cult

@epcim's applied practices of Chef Style DevOps Kung-Fu, while building lab, dev, PoC infrastructures by code.

  • 2,812