DevOps workshop

w/Chef

About me

Developer. IBMer. Vi(m) lover. DevOps kid. Performing cloud infrastructure and application architecture with passion for the edge thing.

 

I am not obsessed with emails and spreadsheet thing as an ordinary IBMers. You wanna reach me on Twitter.

 

Something about you

Background | Practice | Expectations

Workshop agenda

Morning Intro
Definitions
Kung-fu
Docker
Prerequisites / Lab 0. Docker Lab I.
Midday Chef introduction
Toolbox
Chef workflow
Chef Delivery, Analytics
Chef Lab I. Chef Lab III.
Afternoon Cookbook development
Advanced practices
{FIXME}, Hacketon
ELK, GRAFANA, UCD, Drone.io
Chef Lab II.
TTD, Compliance
CI/CD
T Day 1 Day 2

Day 1

Soft talks, Chef labs

Buzzword

By Definition

IBM

DevOps is an enterprise capability for continuous software delivery that enables clients to seize market opportunities and reduce time to customer feedback

Pick #1

Chef.io

A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners. 

Pick #2

Three ways

The principles that all of the DevOps patterns can be derived from

Pick #3

Forget DEV to OPS !

Practices

  • Automation
  • Continuous Delivery
  • Collaboration across teams
  • Extended visibility
  • Amplified feedback loops, rapid feedback

Feedback loop

  • Do something
  • Test, Measure
  • Analyze
  • Correct
  • Repeat

Business value

  • Enhanced customer experience
  • Increased capacity to innovate    
  • Faster time to value

Design Thinking

What did the Waterfall, Agile, Scrum done wrong?

Buzzwords

Design Thinking

  • Learning about the audience, customer view.
  • Aptly vs. real needs
  • Ideal vs. realistic

DevOps

  • Feedback loops, cooperation, integration     
  • Continuous Delivery

Agile

  • #12 principles, but "frequent delivery" shines
  • ...

IBM DevOps RA

  • Reference Architecture
  • DevOps Capabilities
  • Adoption Patterns
  • Product assigment
  • Solutions for "inovative edge" architerctures

Links and presenatation in workshop materials.

What are hot trends?

What are hot trends?

  • From traditional IT to microservices

  • API loosely coupled architectures and micro services

  • Technology shift on frontends

  • Design for speed and simplicity

  • Cross platform requirements (Go, mobile platforms)

  • Measure for feedback, data analytics, visualization

  • Stream data

  • Dynamic configuration

What do we need then?

Deploy and operate all infrastructures by code, as applications are code too.

As traditional IT becomes "truck" and cloud native is a "car". Practice DevOps and apply cloud architectures to acquire expertise, tools and operational skills to drive.

My Vision

Personal picks

  • DevOps approach
  • Life-cycle in mind
  • Meet deadlines
  • Independent dependability
  • One artifact for all env.
  • Same deployment process
  • Four eyes rule; visibility
  • Team is not isolated unit
  • Share

Applied           Style DevOps Kung Fu

Originaly by  Adam Jacob / CTO of Chef / @adamhjk

 

Chef Style DevOps Kung Fu

Creative Commons License

Break

(~ 5 min)

Prerequisites check

Lab 0. 

Conf. mgmt. tools

Conf. mgmt. tools

Orchestration

 

Infrastructure as code

 

Application configuration

Actually angle of view matters

  • Traditional IT apps & infrastrures
  • Local or distributed (hybrid) ?
  • Platform specific
    • PaaS as CloudFoundry / Bluemix
    • Containers
  • Dynamic configuration and service discovery
  • . . .

Recognized classic tools

  • Chef
  • Puppet
  • Ansible
  • SaltStack

 

w/Agent

Platforms supproted

Language

DSL

Encryption

Config store

ACL

Frequency

Conf. mgmt. tools

Comparison

Chef | Puppet | Ansible | SaltStack | uDeploy| GRTE | CloudFactory | Lotus Automator | . . .

 

 To be considered

  • Whether it's agent less  (Ansible) or need an agent at endpoint (Chef, puppet)

  • The level of abstraction Fabric vs. Chef|Salt|Puppet

  • Whether the aim is to install or install & maintain

  • What level of customized / automated configuration you need to apply vs. how advanced and complex are available community resources (cookbooks etc)

  • Integration to in house tech. (CI,...)

  • Platforms and environments involved

  • Configure vs. deploy

Backup slides

Platforms

Traditional IT

IaaS - OpenStack

IBM Cloud Orchestrator, RedHat CloudForms, HP Helion, ...

PaaS - Cloud Foundry

Heroku, IBM Bluemix

Patterns

Heat patterns, IBM PureApps

Orchestration

  • containers provide the unit of execution for applications in PaaS
  • Container orchestration (Kubernetes, Swarm) vs. Distributed configuration (etcd, consul, ...)

Backup slide

Containers

  • Focused on infrastructure -OpenStack
  • Focused on application - UrbanCode, Chef Delivery, ...

Deployment tools

Chef intro

This means that Chef can:

  • Build  Deterministic Servers 
  • Deploy and Configure Applications
  • Deploy Infrastructures
  • Update Applications
  • Run audits and notifications

Chef is a configuration management

and automation platform

Overview

How it works

chef-provisioning

recipes

knife-softlayer

knife-ec2

knife-google

Ecosystem

Chef Development Kit (ChefDK)

$ chef generate app cookbook_name
$ chef generate attribute
$ chef generate recipe
$ chef generate lwrp

$ chef-apply name_of_recipe.rb
  • chef
  • chefspec
  • chef-provisioning
  • chef-provisioning-aws
  • chef-provisioning-fog
  • Inspec 
  • chef-vault
  • knife-spork
  • rubocop
  • foodcritic

Ecosystem

$ knife search node "platform:centos AND languages_ruby_version:1.8*"

$ knife search node "chef_environment:production AND platform:ubuntu"

$ knife ssh 'name:* AND NOT name:chef*' 'sudo postmap -p /etc/postfix/transport'

$ knife bootstrap --environment dev --node-name catalog.lab.dev 10.10.50.15 \
    --ssh-password passw0rd --run-list "role[base]" --server-url "https://10.10.50.2" \
    --json-attributes '{ "hostname": "catalog","name": "catalog.lab.dev", \
    "system":{ "short_hostname":"catalog", "domain_name":"lab.dev" } }' \
    --secret-file .chef/encrypted_data_bag_secret


$ knife node run_list add ora.kb.dev "role[git], recipe[backup::gitlab]"

$ knife role from file roles/*.rb

$ knife environment from file staging.rb

knife

** KNIFE COMMANDS **

bootstrap, client, configure, cookbook, cookbook site, data bag, delete, deps,
diff, download, edit, environment, exec, list, node, raw, recipe_list, role,
search, serve, show, ssh, ssl check, ssl fetch, status, tag, upload, user

** NODE COMMANDS **
knife node bulk delete REGEX (options)
knife node create NODE (options)
knife node delete NODE (options)
knife node edit NODE (options)
knife node environment set NODE ENVIRONMENT
knife node from file FILE (options)
knife node list (options)
knife node run_list add [NODE] [ENTRY[,ENTRY]] (options)
knife node run_list remove [NODE] [ENTRY[,ENTRY]] (options)
knife node run_list set NODE ENTRIES (options)
knife node show NODE (options)

Ecosystem

** SOFTLAYER COMMANDS **

knife softlayer datacenter list (options)
knife softlayer datacenter show DATACENTER
knife softlayer flavor list (options)
knife softlayer global ip list (options)
knife softlayer image list
knife softlayer key pair create
knife softlayer key pair list
knife softlayer server create (options)
knife softlayer server destroy (options)
knife softlayer server list (options)
knife softlayer server relaunch <NODE NAME> [<NODE NAME>]
knife softlayer vlan create
knife softlayer vlan list (options)
knife softlayer vlan show ID (options)
  • knife-softlayer
  • knife-ec2
  • knife-azure
  • knife-windows

knife plugins

Ecosystem

chef supermarket

Ecosystem

include_recipe 'apache2'

user 'userdoc' do
  supports manage_home: true
  comment 'userdoc (bind user)'
  gid 'users'
  home '/home/userdoc'
  password '$1$9p26dAeZ$qZ7P4XW.ASDG4DSD3234vK0'
end

directory '/var/www/userdoc' do
  mode 00775
  owner 'apache2'
  group 'users'
  recursive true
end

web_app 'userdoc' do
  server_name "udoc.#{node['domain']}"
  server_aliases ["userdoc.#{node['system']['domain_name']}"
  docroot '/var/www/userdoc'
  cookbook 'apache2'
  directory_options 'Indexes FollowSymLinks'
end

cookbooks, recipes

cookbook
|
├── .kitchen.yml
├── Berksfile
├── metadata.rb
├── attributes
│   └── default.rb
├── recipes
│   └── default.rb
└── templates
    └── default
        └── index.html.erb
# metadata.md

name 'gitzone'
maintainer 'Petr Michalec'
maintainer_email 'epcim@apealive.net'
license 'Apache 2.0'
description 'Installs/Configures gitzone managed zone files for Bind'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '1.0.15'

recipe 'gitzone::default',   'Install and fully configure gitzone using the install and configure recipes'
recipe 'gitzone::install',   'Install gitzone scripts'
recipe 'gitzone::configure', 'Configure gitzone (including BIND and default zones)'
recipe 'gitzone::configure_zonefile', 'Deploy zone files'

supports 'ubuntu'
supports 'centos'

depends 'bind'
depends 'git'
depends 'sudo'
depends 'system'
depends 'build-essential'
depends 'simple_iptables'

Ecosystem

require 'chef/provisioning'
with_driver 'fog:OpenStack'

...

# PRE ALLOCATE
machine_batch do
  %w(chef web repo ci).each do |m|
    machine "#{m}.lab.ci" do
      action :allocate
      add_machine_options bootstrap_options: PER_MACHINE_BOOTSTRAP_OPTIONS[m] || {}
      retries 1
    end
  end
end

# BOOTSTRAP/CONVERGE NODE
machine 'chef.lab.ci' do
  ohai_hints 'openstack' => '{}'
  attribute %w(apt compile_time_update), true
  attribute %w(resolver nameservers), search(:node, 'role:dns').map \
    { |x| x.automatic.ipaddress } unless search(:node, 'role:dns').empty?
  recipe 'lab_base::default'
  role 'chef'
  add_machine_options bootstrap_options: PER_MACHINE_BOOTSTRAP_OPTIONS['chef'] || {}
  files('/etc/chef/encrypted_data_bag_secret' => "#{dtbgsec}")
  action [:ready, :converge]
  converge true
end

​Chef-Zero

Simple, easy-run, fast-start in-memory Chef server for testing and solo purposes

chef-provisioning, chef-zero

​Chef-Provisioning

A collection of resources that enable the creation of machines and machine infrastructures using the chef-client.

Ecosystem

Analytics & audits

# in audit recipe
control_group 'ssh' do
  control 'password authentication' do
    it 'is disabled' do
      expect(file('/etc/ssh/sshd_config')).to_not \
        match(/^PasswordAuthentication\s+yes/i)
    end
  end
end

# in analytics rule
rules 'failed-audit'
  rule on run_control_group
  when
    status != 'success'
  then
    notify('slack', "{
      'username': 'Audit Alarm', 'text': "{{message.name}} (cookbook {{message.cookbook_name}})\n
      had '{{message.number_failed}}' failed audit test(s) on node '{{message.run.node_name}}'"
    }")
  end
end

Chef WebUI

Tools

Preffered languages and tools that fit the job

  • Consul
  • ELK stack
  • Grafana
  • Kubernetes
  • Bluemix

Backup slides

Vagrant

Kibana (ELK stack)

Grafana

Consul.io

Consul.io

Chef workflow

"simple" web app

PartA, develpop cookbooks/repo/recipes

https://asciinema.org/a/31362

 

PartB - execute serverspec/inspec audit

https://asciinema.org/a/31360

Break

(~ 5 min)

Chef Lab I.

Cookbooks & deployment

Cookbook components

  • attributes
  • definitions
  • files
  • libraries
  • recipes
  • resources
  • templates
  • cookbook
    • metadata
    • tests
  • chef repository
    • data bags
    • environment
    • roles

Cookbook types

  • base, environment
  • application
  • library
  • wrapper

Chef repo

  • Monolitic, chef repository with all cookbooks in "cookbooks" directory
  • Berkshelf way, cookbooks has own repos. Berkshelf solve dependency, download cookbooks and manage Berksfile.

 

Attribute types

  • default
  • force_default
  • normal
  • override
  • force_override
  • automatic (ohai)

Attribute source

  • environment
  • role
  • cookbook
  • automatic
  • node
    • attributes (current state)

Env. + Roles

  • to define
    • attributes
    • run_list
  • takes advantage of attribute precedence
  • example roles: base, web, database, mail, ...
  • example environments
    • dev, test, staging, prod
    • per application

Policyfiles

  • new approach
  • out of beta, approaching completion of the "1.0" feature set
  • classic env + roles will remain
  • policy
    • group
    • name
  • named run_lists
  • multiple sources

Deployment

  • bootstrap node
    • install chef-client
    • (register w/chef-server)
  • chef-client
    • apply run_list
    • synchronise cookboks
    • merge attributes
    • run converge
    • update chef-server

run_list

  • role
    • run_list
    • env_run_list
  • policy
    • run_list
  • node
    • run_list
    • expanded_run_list

Cookbook lifecycle

Cookbook TDD

Training, docs

Chef cookbook testing

Continuous Integration test Continuous Deployment cycle

  • ​Cookbooks
    • Unit test (generated by ChefDK using ChefSpec)
    • Integration tests (KitchenCI, ServerSpec, InSpec)
    • Functional tests (ServerSpec, InSpec)
  • Chef-Repo
    • Functional tests (Rake, ServerSpec, InfraSpec, InSpec)
    • Smoke tests (Integration tests, Audit profiles)

Spec dir. structure

$ cd <cookbook>

$ tree

/spec
├── spec_helper.rb
└── unit
    └── recipes
        ├── default_spec.rb
        ├── sysctl_spec.rb
        └── users_spec.rb

2 directories, 4 files

Chef-Spec

# Cookbook Name:: base-linux
# Spec:: default
#
# Copyright (c) 2016 The Authors, All Rights Reserved.

require 'spec_helper'

describe 'base-linux::default' do                                           
  context 'When all attributes are default, on an unspecified platform' do
    before do
      stub_command('which sudo').and_return('/usr/bin/sudo')
    end

    let(:chef_run) do
      runner = ChefSpec::ServerRunner.new
      runner.node.set['virtualization']['system'] = 'vmware'
      runner.converge(described_recipe)
    end

    it 'converges successfully' do
      expect { chef_run }.to_not raise_error
    end
  end
end

Chef-Spec

require 'chefspec'

describe 'cookbook_name::install' do
  let(:chef_run) { ChefSpec::Runner.new.converge(described_recipe) }

  it 'creates a template with attributes' do
    expect(chef_run).to create_template('/tmp/with_attributes').with(
      user: 'user',
      group: 'group',
      backup: false,
    )
  end

  it 'includes resource that have guards that evalute to true' do
    expect(chef_run).to start_service('true_guard')
  end

  it 'includes the `other` recipe' do
    expect(chef_run).to include_recipe('include_recipe::other')
  end

  it 'removes a package with an explicit action' do
    expect(chef_run).to remove_package('explicit_action')
  end

end

Test dir. structure

$ cd <cookbook>

$ tree

/text
├── chef
│   ├── data_bags
│   │   └── users
│   |       └── testuser.json
│   └── encrypted_data_bag_secret
└── integration
    ├── default
    │   ├── serverspec
    │   │   └── default_spec.rb
    │   └── ssh_spec.rb
    └── helpers
        └── serverspec
            └── spec_helper.rb

8 directories, 4 files

Integration tests

# Serverspec
$ cat default/serverspec/default_spec.rb 

require 'spec_helper'

describe 'base-linux::default' do
  # Serverspec examples can be found at
  # http://serverspec.org/resource_types.html

  describe port(80) do
    it { should be_listening }
  end

  describe command('curl http://localhost') do
    its(:stdout) { should match /Hello, world!/ }
  end

  describe file('/etc/services') do
    its(:selinux_label) { should eq 'system_u:object_r:etc_t:s0' } 
  end 

  describe host('serverspec.org') do
    it { should be_resolvable.by('dns') }
  end

end

Integration tests

# InSpec
$ cat test/integration/default/ssh_spec.rb 

only_if do
  command('sshd').exist?
end

describe service('ssh') do
  it { should be_enabled }
  it { should be_running }
end

control 'sshd-11' do
  impact 1.0
  title 'Server: Set protocol version to SSHv2'
  desc 'Set the SSH protocol version to 2'
  describe sshd_config do
    its('Protocol') { should eq('2') }
  end
end

control 'sshd-7' do
  impact 1.0
  title 'Server: Do not permit root-based login with password.'
  desc "To reduce the potential to gain full privileges, do not allow login as root with password"
  describe sshd_config do
    its('PermitRootLogin') { should match(/no|without-password/) }
  end
end

.kitchen.yml

---
# https://docs.chef.io/config_yml_kitchen.html

driver:
    name: vagrant
    use_sudo: false

provisioner:
  name: chef_zero
  cookbook_path: ['.','cookbooks', 'test/cookbooks']

platforms:
    - name: centos-7.1
      run_list:
        - recipe[selinux::disabled]
    - name: ubuntu-14.04

suites:
  - name: default
    run_list:
        - recipe[lab_base]
        - recipe[lab_gitlab::db]
        - recipe[lab_gitlab::default]
    attributes:
        apt:
            compile_time_update: true
        mysql:
          server_root_password: test
        gitlab:
          database:
            host: 127.0.0.1
            password: test

# vim: sw=2 ts=2 sts=2

You are what you eat

Foodcritic.io

http://www.foodcritic.io

 

Chef Style Guide

https://docs.chef.io/ruby.html

Chef workflow

"simple" web app

PartA, develpop cookbooks/repo/recipes

https://asciinema.org/a/31362

 

PartB - execute serverspec/inspec audit

https://asciinema.org/a/31360

Chef Lab II.

Dont forget DevOps?

Break

(~ 5 min)

Chef workflow

For cookbooks

  • Develop
  • Lint
  • Syntax
  • Verify locally
  • Upload code to feature branch in repository
  • If passing CI, merge with master / tag

Guess Buzzword ?

Continuous *

  • Integration

  • Deployment

  • Delivery

  • Improve

  • Validation

Continuous

That's topic for another workshop

Best practice in general

  • Orchestration, Configuration, Releases, Provisioning
  • The same process and deployment everywhere
  • Build and publish artifacts, one for all environments
  • Manage releases
    • Handle "cookbook" versions cross environments
  • Handle configuration cross environments
  • Fully automated deployment
  • Fully automated CI tests & compliance verification
  • Hybrid cloud options available

CI / CD Pipeline

  • Developers check out code into their private workspaces. When done, the commit changes to the repository.

  • The CI server monitors the repository, checks out on changes.

  • The CI server builds the system and runs unit and integration tests.

  • The CI server releases deployable artefacts for testing.

  • The CI server assigns a build label to the version of the code it just built.

  • The CI server informs the team of the successful build.

  • If the build or tests fail, the CI server alerts the team.

  • The team fix the issue at the earliest opportunity.
    Continue to continually integrate and test throughout the project.

Continuous Integration

Team Responsibilities

  • Check in frequently
  • Don’t check in broken code
  • Don’t check in untested code
  • Don’t check in when the build is broken
  • Don’t go home after checking in until the system builds

CI Best Practices

Chef.io

  • Chef-Delivery

Alternatives

  • Build a CI pipeline around other in house hosted tools (Jazz.net, Jenkins, TravisCI, ...)
  • Build a CD in Chef-provisioning, UrbanCode, IBM Cloud Orchestrator)
  • Run a custom solution

CI/CD pipeline

CI/CD pipeline

CI/CD pipeline

App A

App C

App B

ENTERPR

I

SE

Example: Drone.io

CI Overview

.drone.yml

cache:
  mount:
  - .berkshelf/cookbooks

env:
  - CI=drone
  - LABREPO=https://git.vums.blueit/chef-cookbooks

build:
  image: epcim/drone-chef-ci
  commands:

# RUN TESTS
  # Rspec
  - chef exec rspec spec
  # Kitchen soap
  - KITCHEN_LOCAL_YAML=.kitchen.cloud.yml chef exec kitchen test    || E=$?; test -n $E && /bin/sleep 300  #--concurrency 2
  - KITCHEN_LOCAL_YAML=.kitchen.cloud.yml chef exec kitchen destroy
  # Exit on fail
  - test ${E:-0} -ne 0 && exit $E

# RUN LINT CHECKERS
  - chef exec foodcritic "-f any -f ~FC014 -f ~FC015 -f ~FC024 -f ~FC043" .
  - chef exec rubocop -a || chef exec rubocop . #|| echo "ignoring failed rubocop check"

# TAG SUCCESSFUL VERSION
  - VERSION=`ruby -r 'chef/cookbook/metadata' -e "m=Chef::Cookbook::Metadata.new;m.from_file('metadata.rb');puts m.version"`
  - git config remote.origin.url `git config --get remote.origin.url | sed "s/https:\/\//https:\/\/$GIT_USERNAME:$GIT_PASSWORD@/"`
  - git tag -a v${VERSION} -m 'CI tagged successful build'
  - git push origin v${VERSION} 2> /dev/null || echo $?


notify:
  slack:
    webhook_url: 'https:#hooks.slack.com/services/T0SDDASD43/BSDFD84QY/buISDF436dsfjsdlEoo57qO'
    username: 'drone.io'
    channel: '#lab-cookboks'
    on_started: false
    on_success: true
    on_failure: false

Drone.io

Drone.io addons

.drone.yml - full

cache:
  mount:
  - .berkshelf/cookbooks

clone:
  skip_verify: true
  environment:
  - DRONE_GITLAB_SKIP_VERIFY=true
  - GIT_CURL_VERBOSE=1
  - GIT_SSL_NO_VERIFY=1

env:
  - CI=drone
  - LABREPO=https://git.vums.blueit/chef-cookbooks

build:
  image: epcim/drone-chef-ci
  privileged: true
  environment:
  - OS_USERNAME=$$OS_USERNAME
  - OS_PASSWORD=$$OS_PASSWORD
  - OS_TENANT_NAME=$$OS_TENANT_NAME
  - OS_AUTH_URL=$$OS_AUTH_URL
  - BOOTSTRAP_SSH_KEY=$$BOOTSTRAP_SSH_KEY
  - BOOTSTRAP_SSH_PUB=$$BOOTSTRAP_SSH_PUB
  - GIT_USERNAME=$$GIT_USERNAME
  - GIT_PASSWORD=$$GIT_PASSWORD
  - GIT_COMMITTER_NAME=$$GIT_COMMITTER_NAME
  - GIT_COMMITTER_EMAIL=$$GIT_COMMITTER_EMAIL
  commands:
### INITIALIZE CI ENV  ###
  - . /root/.bash_profile
  - eval "$(chef shell-init sh)"
  - export LANG="en_US.UTF-8"
  - export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem
  - chef exec berks install

### INITIALIZE REMOTE CI ENV ###
  - BOOTSTRAP_KEYFILE=/root/.ssh/bootstrap_insecure
  - echo "$BOOTSTRAP_SSH_KEY" >  $BOOTSTRAP_KEYFILE; chmod 0600 $BOOTSTRAP_KEYFILE
  - eval `ssh-agent`
  - ssh-add $BOOTSTRAP_KEYFILE
  - ssh-add ./id_rsa_bootstrap_insecure

  #- chef exec bundle install
  - chef gem install kitchen-openstack

# RUN TESTS
  # Rspec
  - chef exec rspec spec
  # Kitchen soap
  - KITCHEN_LOCAL_YAML=.kitchen.cloud.yml chef exec kitchen test    || E=$?; test -n $E && /bin/sleep 300  #--concurrency 2
  - KITCHEN_LOCAL_YAML=.kitchen.cloud.yml chef exec kitchen destroy
  # Exit on fail
  - test ${E:-0} -ne 0 && exit $E

# RUN LINT CHECKERS
  - chef exec foodcritic "-f any -f ~FC014 -f ~FC015 -f ~FC024 -f ~FC043" .
  - chef exec rubocop -a || chef exec rubocop . #|| echo "ignoring failed rubocop check"

# TAG SUCCESSFUL VERSION
  - VERSION=`ruby -r 'chef/cookbook/metadata' -e "m=Chef::Cookbook::Metadata.new;m.from_file('metadata.rb');puts m.version"`
  - git config remote.origin.url `git config --get remote.origin.url | sed "s/https:\/\//https:\/\/$GIT_USERNAME:$GIT_PASSWORD@/"`
  - git tag -a v${VERSION} -m 'CI tagged successful build'
  - git push origin v${VERSION} 2> /dev/null || echo $?


notify:
  slack:
    webhook_url: 'https:#hooks.slack.com/services/T0SDDASD43/BSDFD84QY/buISDF436dsfjsdlEoo57qO'
    username: 'drone.io'
    channel: '#lab-cookboks'
    on_started: false
    on_success: true
    on_failure: false

.drone.yml - backup

image: epcim/drone-chef-ci:dind

env:
    - CI=drone
    - LABREPO=https://gitlab.lab.local/chef-cookbooks

script:
    - source /root/.bash_profile
    - eval "$(chef shell-init bash)"
    - export LANG="en_US.UTF-8"
    - export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem

    - chef exec berks install
    - chef gem install chef-rewind

    - wrapdocker

    - chef exec rspec spec
    - chef exec kitchen test

    - chef exec foodcritic "-f any -f ~FC014 -f ~FC015 -f ~FC024 -f ~FC043" .
    - chef exec rubocop -a || chef exec rubocop .

notify:
  slack:
    webhook_url: 'https:#hooks.slack.com/services/T0SDDASD43/BSDFD84QY/buISDF436dsfjsdlEoo57qO'
    username: 'drone.io'
    channel: '#lab-cookboks'
    on_started: false
    on_success: true
    on_failure: false

Review


Questions


Feedback

Day 2

Docker, Chef, Hacking

Recap

  • DevOps is "buzzword"
  • Create a cookbook (chef generate cookbook NAME)
  • update medatada.rb (dependencies)
  • if anything from ./path or git:/ update Berksfile
  • write a recipe, add some attributes
  • write a .kitchen.yml (test with "kitchen test")
  • write unit tests in "spec" folder
  • write integration tests in InSpec lang. in "test" folder

Tools mentioned

  • packer.io to build images
  • Vagrant (an virtualization layer abstraction)
  • KitchenCI is your local, friendly,  CI engine
  • InSpec/ServerSpec libraries to check infrastructure
  • ChefSpec library for cookbook unit tests

Remember

  • Presentation and Lab material are
    • written as refrence with links to proper place if you want to lear more and go deep
    • there are chef webinar available for cookbook essential stuff, TDD, node compliance (InSpec), etc..

If shit happens

  • RTFM
  • Open Browser against Github and source code
    • (forget time and read)
  • IRC, Gitter, email author

Orace cookboos

  • 15 cookbooks on supermarket

WAS cookboos

wlp_server "myInstance" do
  config ({
            "featureManager" => {
              "feature" => [ "jsp-2.2", "jaxws-2.1" ]
            },
            "httpEndpoint" => {
              "id" => "defaultHttpEndpoint",
              "host" => "*",
              "httpPort" => "9080",
              "httpsPort" => "9443"
            },
            "application" => {
              "id" => "example",
              "name" => "example",
              "type" => "war",
              "location" => "/apps/example.war"
            }
          })
  jvmOptions [ "-Djava.net.ipv4=true" ]
  serverEnv "JAVA_HOME" => "/usr/lib/j2sdk1.7-ibm/"
  bootstrapProperties "default.http.port" => "9080", "default.https.port" => "9443"
  action :create
end

wlp_download_feature "mongodb" do
  name "mongodb-2.0"
  directory "/opt/ibm/wlp/features"
  accept_license true
end

WAS cookboos

ci.chef.wlp
Chef cookbook for installing and managing WebSphere Application Server Liberty Profile #devops
Updated 21 days ago

ci.chef.iim
Chef Cookbook to install, configure and run IBM Installation Manager
Updated on Nov 3, 2015

ci.chef.ihs
Chef cookbook for installing and managing IBM HTTP Server #devops
Updated on Jun 18, 2015

ci.chef.wxs
Chef Cookbook to install and configure WebSphere eXtreme Scale
Updated on Sep 12, 2014

ci.chef.was
Chef cookbook for installing and managing WebSphere Application Server with wsadmin #devops
Updated on Sep 12, 2014

ci.chef.ihs.config
Chef cookbook for configuring IHS WebSphere Plugin
Updated on Sep 12, 2014

ci.chef.wlp.application
Chef cookbook for deploying applications to the WebSphere Application Server Liberty Profile #devops
Updated on Sep 12, 2014

ci.chef.wlp.samples

UCD plugins

Debug RSPEC

# cd <repo>/cookbooks/base-linux

# cat spec/spec_helper.rb

require 'chefspec'
require 'chefspec/berkshelf'

RSpec.configure do |config|

  # Specify the Chef log_level (default: :warn)
  config.log_level = :debug

end

Debug RSPEC

# rspec spec |wc -l

348945

Debug RSPEC

System cookbook in place, still passing UNIT tests

Chef DK versions

# chef exec chef -v

Chef Development Kit Version: 0.10.0
chef-client version: 12.6.0
berks version: 4.0.1
kitchen version: 1.5.0

# berks

Resolving cookbook dependencies...
Fetching 'base-linux' from source at .
Using apparmor (2.0.1)
Using apt (3.0.0)
Using base-linux (0.1.0) from source at .
Using chef-sugar (3.3.0)
Using chef_handler (1.3.0)
Using compat_resource (12.7.3)
Using cron (1.7.5)
Using firewall (2.4.0)
Using hostsfile (2.4.5)
Using motd (0.6.3)
Using ntp (1.10.1)
Using ohai (2.1.0)
Using selinux (0.9.0)
Using sudo (2.9.0)
Using sysctl (0.7.0) from https://github.com/svanzoest-cookbooks/sysctl (at master)
Using system (0.10.1)
Using users (2.0.3)
Using windows (1.39.2)
Using yum (3.10.0)
Using yum-epel (0.6.6)

Where we failed in LABs

  • cooperation?
  • bug visibility?
  • team work?

Get * R * DONE

  • upload your git code to repo
  • compare with my repo
  • make my repo "upstream" and merge ?
  • fork from somebody who did it right?
  • fix issues, make a PR to upstream repo?
  • git init
  • git add .
  • git commit -m "layz boy on comments"
  • #go to https://github.rtp.raleigh.ibm.com/groups/devops-workshop and create your repo
  • git remote add origin "XYZ"
  • git push --set-upstream origin master
  • # browse, fork from somebody else?

everybody else

  • fork this repo to your namespace
  • clone it locally (git clone "https://xxxx")
  • git checkout -b "my-feature-branch"
  • # work here
  • # if you will succed make PR
  • # if you will fail your develpment
    • git checkout -b mergeorigin
    • git merge "<branch name>"
  • promote to slack

hard way, dont't fork, merge with me

  • git remote add upstream git@github.rtp.raleigh.ibm.com:devops-workshop/pmichalec-base-linux.git
  • git fetch upstream
  • git checkout -b test-merge-with-peter
  • git merge -s theirs --no-commit upstream/master # ???
  • # resolve conflicts, commit
  • git commit -m "merged"
  • # merge with your master
  • git checkout master
  • git merge test-merge-with-peter
  • git push

what else

whate about to check commits history at my repo 

 

Fix Lab I/II.

Cooperation

https://ibm-design.slack.com/messages/devops-workshop/

 

 

  • Login with your company email
  • Promote your progress, ask others, share code

Docker

Docker

Products

 

Crash intro (slides 1-16)

  • Docker Engine
  • Docker Hub
  • Docker Machine
  • Docker Compose
  • Docker Toolbox
  • Docker Swarm
  • Docker Registry
  • . . .

Docker

Filesystem

Docker

VM vs Container

Docker

Architecture

Dockerfile

FROM jpetazzo/dind
MAINTAINER Petr Michalec <epcim@apealive.net>

RUN apt-get update
RUN apt-get install -qqy    curl \
                            sudo \
                            git \
                            mercurial \
                            subversion \
                            ca-certificates \
                            locales \
                            jq

RUN echo 'en_US.UTF-8 UTF-8'>>/etc/locale.gen
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV DEBIAN_FRONTEND noninteractive


## CHEF DK ###########################
RUN curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -P chefdk
ENV PATH /opt/chefdk/bin:/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Make Chef DK the primary Ruby/Chef development environment.
RUN echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
RUN eval "$(chef shell-init bash)"


RUN chef gem install kitchen-docker
RUN chef gem install kitchen-openstack
RUN chef gem install chef-sugar
RUN chef gem install chef-rewind
RUN chef gem install serverspec
RUN chef gem install infratester


# berks pre-fetch some common soup of cookbooks
RUN mkdir /tmp/fake_cookbook; cd $_
RUN echo "name 'fake_cookbook'\nmaintainer 'fake_cookbook'\nlicense 'fake_cookbook'\ndescription 'fake_cookbook'\nversion '0.0.1'" > metadata.rb
RUN echo "source 'https://supermarket.chef.io'\nmetadata\n\n" > Berksfile
RUN echo "cookbook '7-zip'\ncookbook 'apache2'\ncookbook 'apt'\ncookbook 'ark'\ncookbook 'bluepill'\ncookbook 'build-essential'\ncookbook 'certificate'\ncookbook 'chef-client'\ncookbook 'chef_handler'\ncookbook 'chef_ruby'\ncookbook 'chef-sugar'\ncookbook 'chef-vault'\ncookbook 'cron'\ncookbook 'database'\ncookbook 'device-mapper'\ncookbook 'git'\ncookbook 'minitest-handler'\ncookbook 'modules'\ncookbook 'ncurses'\ncookbook 'nginx'\ncookbook 'ntp'\ncookbook 'ohai'\ncookbook 'openssh'\ncookbook 'openssl'\ncookbook 'packagecloud'\ncookbook 'pacman'\ncookbook 'perl'\ncookbook 'rbenv'\ncookbook 'readline'\ncookbook 'resolver'\ncookbook 'resource-control'\ncookbook 'rsyslog'\ncookbook 'ruby'\ncookbook 'ruby_build'\ncookbook 'runit'\ncookbook 'subversion'\ncookbook 'sudo'\ncookbook 'sysctl'\ncookbook 'system'\ncookbook 'ulimit'\ncookbook 'users'\ncookbook 'windows'\ncookbook 'xml'\ncookbook 'yum'\ncookbook 'yum-epel'\ncookbook 'zlib'\n" >> Berksfile
RUN chef exec berks install
RUN cd -


## FIX UP'S ##########################
RUN chmod -R 0440 /etc/sudoers
RUN chmod -R 0440 /etc/sudoers.d
# workaround (drone.io has no way yet to modify this image before git clone happens)
RUN git config --global http.sslverify false




VOLUME /var/lib/docker
CMD ["wrapdocker"]

Docker Orchestration

Docker Orchestration

  • Kubernetes
  • Swarm
  • Messos framewok

Docker practice

FIXME

Docker practice

Cross host dependencies

  • Ambassador pattern
    • link to local container, that act as forwarder
    • changing backend service = restarting ambasador
  • Environment pattern
    • pass the details of remote service as environment variable
    • docker -e DB=<IP ADDRESS>
    • --env-file <file>

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker practice

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker Lab I.

Review


Questions


Feedback

Before Chef Lab III.

Preliminary info about Chef Repo & InSpec profiles

Chef Repository

chef-repo

  • Monolitic, chef repository with all cookbooks in "cookbooks" directory
  • Berkshelf way, cookbooks has own repos. Berkshelf solve dependency, download cookbooks and manage Berksfile.

 

  • to upload / download artifacts to chef server
  • place from where you manage nodes
  • env+roles OR completely Policy based

chef-repo-#{ENV}

  • Chef cookbook as many others  **
  • Contain definition for roles, environments, data bags
  • May contain chef provisioning recipes
    • for applications, environments, etc ...
  • May contain chef audit recipes (InSpec)
  • Integration tests ServerSpec/InSpec  
  • Rakefile, scripts, core configuration

knife-zero-repo

  • For one-time / simple deployments
  • Runs Chef-Zero in memory
  • Filesystem as persistent storage
  • Commands
    • bootstrap
    • converge
    • chef_client
    • diagnose 

rake -T

rake berks                     # Berks install/update/vendor
rake bootstrap:chefonly        # Boostrap CHEF ONLY with SSH
rake bootstrap:ci              # Boostrap CI against OpenStack env
rake bootstrap:gtshub          # Boostrap gtshub using SSH
rake bootstrap:setup           # Bootstrap local cfg for chef-provisioning setup
rake bootstrap:test            # Boostrap TEST against OpenStack env
rake bootstrap:vagrant         # Bootstrap Vagrant env
rake chef:hosts                # Create a hosts.yml file based on knife information
rake chef:hosts_local          # Create a hosts.yml file based on local ChefZero information
rake chef:nodes_local          # Print ip + fqdn from nodes records
rake convert                   # Convert ruby classes to json
rake convert:env_to_json       # Convert ruby environments from ruby to json, creating/overwriting json files
rake convert:metadata_to_json  # Convert all metadata from ruby to json
rake convert:role_to_json      # Convert ruby roles from ruby to json, creating/overwriting json files
rake destroy:ci                # Destroy CI environment
rake dev:bootstrap             # Run task bootstrap:ci in local DEV environment
rake dev:cls                   # Clean up local chef-zero instances
rake dev:setup                 # Setup local development ENV variables
rake foodcritic                # Foodcritic
rake knife:bootstrap           # Bootstrap new node using knife
rake lint                      # Lint & syntax checks
rake provision:chefonly        # Provision CHEF server with SSH
rake provision:ci              # Provision CI environment against Openstack
rake provision:gtshub          # Provision gtshub environment with SSH
rake provision:test            # Provision TEST environment with Openstack
rake rubocop                   # Run RuboCop style and lint checks
rake rubocop:auto_correct      # Auto-correct RuboCop offenses
rake spec                      # Run serverspec to all hosts
rake vagrant:create            # Create the Vagrant machine
rake vagrant:destroy           # Stop and delete the Vagrant machine
rake vagrant:login             # Log into the Vagrant machine via SSH
rake vagrant:provision         # Provision the Vagrant machine with Chef

Chef Provisioning

Chef Provisioning

Amazon Web Services A Chef provisioning driver for Amazon Web Services (AWS).
Docker A Chef provisioning driver for Docker.
Fog A Chef provisioning driver for Fog.
Hanlon A Chef provisioning driver for Hanlon.
LXC A Chef provisioning driver for LXC.
Microsoft Azure A Chef provisioning driver for Microsoft Azure.
OpenNebula A Chef provisioning driver for OpenNebula.
SSH A Chef provisioning driver for SSH.
Vagrant A Chef provisioning driver for Vagrant.
vSphere A Chef provisioning driver for VMware vSphere.

Chef Provisioning

machine 'server_a' do
  recipe 'base_recipes'
end

machine 'server_b' do
  recipe 'base_recipes'
  recipe 'theserver'
end


machine_batch do
  machine 'db' do
    recipe 'mysql'
  end
  1.upto(50) do |i|
    machine "#{web}#{i}" do
      recipe 'apache'
    end
  end
end

Chef Provisioning

  • aws_auto_scaling_group
  • aws_cache_cluster
  • aws_cache_replication_group
  • aws_cache_subnet_group
  • aws_cloudsearch_domain
  • aws_dhcp_options
  • aws_ebs_volume
  • aws_eip_address
  • aws_image
  • aws_instance
  • aws_internet_gateway
  • aws_key_pair
  • aws_launch_configuration
  • aws_load_balancer
  • aws_network_acl
  • aws_network_interface
  • aws_rds_instance
  • aws_rds_subnet_group
  • aws_route_table
  • aws_s3_bucket
  • aws_security_group
  • aws_server_certificate
  • aws_sns_topic
  • aws_sqs_queue
  • aws_subnet
  • aws_vpc

Visibility

  • Monitoring
  • Audit & Compliance
  • Reporting
  • Analytics

Monitoring

  • Healt checks
  • Notification
  • Metrics

Alert on what is actionable

  • Get the attention of the right humans
  • As few alerts as possible
  • Routed to the people who can take action
  • Start with the is it up alert

Chef Audit


# it's just a "serverspec"...

control_group 'SSH Service compliance' do

  control 'ssh service' do
    it 'should be listening on port 22' do
      expect(port(22)).to be_listening
    end
  end

  control 'ssh configuration' do
    it 'ssh disables root logins' do
      expect(file('/etc/ssh/sshd_config')).to contain('PermitRootLogin no')
    end

    it 'is disabled' do
      expect(file('/etc/ssh/sshd_config')).to_not \
        match(/^PasswordAuthentication\s+yes/i)
    end
  end
end


Chef InSpec

# use basic tests
describe package('nginx') do
  it { should be_installed }
end

# extend tests with metadata
control '01' do
  impact 0.7
  title 'Verify nginx service'
  desc 'Ensures nginx service is up and running'
  describe service('nginx') do
    it { should be_enabled }
    it { should be_installed }
    it { should be_running }
  end
end

# implement os dependent tests
web_user = 'www-data'
web_user = 'nginx' if os[:family] == 'centos'

describe user(web_user) do
  it { should exist }

Sensu

Grafana dashboards

Grafana & Sensu

ELK

Chef Lab III.

Let's rock

Chef Delivery

Chef Delivery

Chef Delivery

Chef Delivery

Chef Delivery

Behind the scene is actually a cookbook

https://github.com/chef-cookbooks/delivery-truck 

Chef Compliance

Urban Code

Presentation and materials in workshop folder

https://developer.ibm.com/urbancode

Hack Session

Recap


Questions


Feedback

Resources used

TBD

  • http://chef.github.io/devops-kungfu
  • https://ianunruh.com/
  • https://www.brianchristner.io

 

Backup slides

lab_ cookbooks

  • lab_gitlab
  • lab_audit
  • lab_backup
  • lab_base
  • lab_openvpn
  • lab_chef-server
  • lab_bind
  • lab_drone
  • lab_nagios
  • lab_firewall
  • lab_trac
  • lab_redmine
  • lab_openldap
  • lab_artifactory
  • lab_maven
  • lab_docker
  • lab_subversion
  • lab_postfix
  • lab_jenkins
  • lab_sensu
  • lab_oracle
  • lab_websphere
  • lab_owncloud
  • ...

wrappers

  • percona ( MySQL XtraDB Percona cluster )
  • sensu ( Sensu monitoring + Graphite, Grafana )
  • gitzone ( git managed BIND zones )

applications

Made with Slides.com