Install a kubernetes cluster and deploy apps

Who we are ?

Smaïne Kahlouch

Technical lead and Devops

 

Antoine Legrand

Lead software engineer

@ Arkena

Agenda

  • Project motivations

  • What is kubernetes ?

  • Architecture

  • Work units

  • Networking

  • Deploying a cluster

  • Installing new services

Project motivations

Heroku is the current hosting platform

 

Better cost-control

 

Internalize operations for a better service delivery

 

Performances fine-tunning

I'll assume you already know the basics of Docker and Ansible

 

I'll try to Focus on Ansible but ...

Before starting ...

What is kubernetes ?

 

  • A system for container management in a clustered environment (open sourced by Google)

 

  • Based on Docker container system (work in progress to support Rocket)

 

  • Provides grouping, load balancing, scaling and scheduling features

 

    Current version v1.2

Master components

  • API Server : The main management endpoint for the cluster (RESTful interface)

 

  • Controller Manager: Handles replication management

 

  • Scheduler Server : Assigns workloads to specific nodes

 

  • etcd : A distributed key-value store for sharing configuration

Node components

  • Docker : A Container system which runs on a dedicated network

 

  • Kubelet : Is responsible for the communication with the master server

 

  • Proxy : Used for network forwarding and load balancing

Architecture : Components

Work units

  • Pod:

A colocated group of containers (one-to-many) with shared resources. e.g. network, volumes.

It can be viewed as a "logical host".

 

  • Service:

An interface to a group of containers, which acts as load-balancer and provides an abstraction layer - no need to worry about containers location.

Work units

  • Deployments :

Declarative way to describe the desired state of the application (pods, replica sets).

 

  • Config maps :

Volumes used to store the config files to be used within the pods.

 

  • Secrets :

A custom volumes to store passwords, keys etc.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx
  labels:
    k8s-app: nginx
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 3
  selector:
    matchLabels:
      k8s-app: nginx
  template:
    metadata:
      labels:
        k8s-app: nginx
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 80

Workload example

kubectl create -f nginx-rc.yaml

Logical Architecture

Networking

 

  • The pods are scheduled on a flat shared network accross all nodes.

 

  • Each pod can communicate without proxies and translations (NAT) with other pods within the cluster.

 

  • Several implementations are available: L2 networks, Flannel, Weave, OpenVswitch, Calico

Calico is a layer 3 approach to virtual networking.

• The config is saved into an etcd storage

• The routes are distrubuted using BGP

• Distributed ACL's policy

• allows communication between pods/services

• common ip troubleshooting tools (ping, traceroute, netcat ...)

Here comes Kubespray !

The Github organization has two main projects :

 

  • Kargo:  A playbook which installs and configures a kubernetes cluster

 

  • Kpm: Tool to deploy and manage applications stack on kubernetes.

Setup the cluster

 

Currently supported OS : Debian, Ubuntu, CentOS/RHEL, CoreOS

 

Download and install binaries

 

Configure every components (Docker, etcd,  dnsmasq...)

 

Choose the network plugin:

Flannel, Calico or Weave

---
- hosts: k8s-cluster
  roles:
    - { role: adduser, tags: adduser }
    - { role: download, tags: download }
    - { role: kubernetes/preinstall, tags: preinstall }
    - { role: etcd, tags: etcd }
    - { role: docker, tags: docker, when: ansible_os_family != "CoreOS" }
    - { role: kubernetes/node, tags: node }
    - { role: network_plugin, tags: network }

- hosts: kube-master
  roles:
    - { role: kubernetes/master, tags: master }

- hosts: k8s-cluster
  roles:
    - { role: dnsmasq, tags: dnsmasq }

cluster playbook

ansible-playbook -i inventory/inventory.cfg -u root cluster.yml
node1 ansible_ssh_host=95.54.0.12  # ip=10.3.0.1
node2 ansible_ssh_host=95.54.0.13  # ip=10.3.0.2
node3 ansible_ssh_host=95.54.0.14  # ip=10.3.0.3
node4 ansible_ssh_host=95.54.0.15  # ip=10.3.0.4
node5 ansible_ssh_host=95.54.0.16  # ip=10.3.0.5
node6 ansible_ssh_host=95.54.0.17  # ip=10.3.0.6

[kube-master]
node1
node2

[etcd]
node1
node2
node3

[kube-node]
node2
node3
node4
node5
node6

[k8s-cluster:children]
kube-node
kube-master

Inventory

bin_dir: /usr/local/bin
local_release_dir: "/tmp/releases"
kube_cert_group: kube-cert
kube_log_level: 2

kube_users:
  kube:
    pass: changeme
    role: admin

cluster_name: cluster.local

kube_network_plugin: calico

kube_service_addresses: 10.233.0.0/18

kube_pods_subnet: 10.233.64.0/18

kube_network_node_prefix: 24

peer_with_router: false

kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
kube_apiserver_port: 443 # (https)
kube_apiserver_insecure_port: 8080 # (http)

upstream_dns_servers:
  - 8.8.8.8
  - 4.4.8.8
dns_setup: true
dns_domain: "{{ cluster_name }}"
dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"

group_vars

KPM

 

Outdated, to be done with 

https://github.com/kubespray/kpm

 

Links

Kubernetes documentation :

http://kubernetes.io/v1.1/index.html

 

kubespray repositories :

https://github.com/kubespray

 

kubespray tests :

http://travis.kubespray.io

http://ci.kubespray.io

 

ProjectCalico website :

http://www.projectcalico.org/

 

Thank you

Kubespray

By Smaine Kahlouch

Kubespray

Install a kubernetes cluster and deploy new applications.

  • 2,009