Automatic Deployment of virtual environments on Open Telekom Cloud

Kurt Garloff

OpenStack Cloud Architect

kurt.garloff@t-systems.com

Deutsche OpenStack Tage 21./22.6.2016, Köln

https://slides.com/kgarloff/automation-with-otc-api-1

Why public cloud?

Classical Deployment

Cloud Installation

And the result is ...

What went wrong?

Culture

• Manual approvals

Technology

• Manual installations

Automation

Automating virtual Infrastructure

• DIY REST (curl, jq)
• libcloud
• heat
• scalr
• terraform
• cliqr
• bosh
• juju
• ansible
• ...

OpenStack REST API

cloud-init

- controlled by meta-data src

- run upon first boot

- user accounts, ssh keys

- hostname, timezone

- networks, mounts

- package updates & installation

- arbitrary scripts

- phone home

- OTC: Inject files into /etc/cloud/cloud.cfd.d/

Config Mgmt Tools

- puppet, chef, ansible, saltstack

- run repeatedly to ensure compliance or changes

- configuration properties

- software packages

- can control infra as well

Application Management

OTC hierarchy

Open Telekom Cloud

API

Text

https://cloud.telekom.de/blog

API documentation

https://docs.otc.t-systems.com/?locale=en-us

Getting the parameters

Environment ~/.ostackrc

# Adapt the first two lines ...
export OS_PASSWORD="XXXXX"
export OS_USERNAME="14610698 OTC00000000001000000205"
export OS_TENANT_NAME=eu-de
export OS_PROJECT_NAME=eu-de
export OS_USER_DOMAIN_NAME="${OS_USERNAME##* }"
export OS_AUTH_URL=https://iam.eu-de.otc.t-systems.com:443/v3
export OS_PROJECT_DOMAIN_NAME=
export OS_IDENTITY_API_VERSION=3
export OS_VOLUME_API_VERSION=2
export OS_IMAGE_API_VERSION=2
export OS_ENDPOINT_TYPE=publicURL
export NOVA_ENDPOINT_TYPE=publicURL
export CINDER_ENDPOINT_TYPE=publicURL
# no longer needed
#export OS_CACERT=/etc/ssl/OTC-API-GW-CA-Bundle.pem    

Template preinstalled in our images

Example using otc-tools

Simple shell/curl/jq code

github https://github.com/OpenTelekomCloud/, OBS home:garloff:OTC

To be superceded by python-otcclient

garloff@ImgFact-JumpHost2:~ [0]$ otc ecs list
04213851-9ce9-4fbe-b2eb-42f950822e1c   ecs-8430
3902c40e-8f63-49e7-9a46-3147834c9a7d   test-os421-longssh
garloff@ImgFact-JumpHost2:~ [1]$ otc vpc list
b4690bfa-6d61-40b4-86b5-e8ab903ef11e   vpc-32   OK   192.168.32.0/20
garloff@ImgFact-JumpHost2:~ [0]$ otc subnet list
[
  {
    "id": "7cbe489c-cf80-4f40-a117-a8946714a001",
    "name": "subnet-33",
    "cidr": "192.168.33.0/24",
    "status": "ACTIVE",
    "vpc_id": "b4690bfa-6d61-40b4-86b5-e8ab903ef11e",
    "gateway_ip": "192.168.33.1",
    "dhcp_enable": true,
    "primary_dns": "100.125.4.25",
    "secondary_dns": "217.150.148.148",
    "availability_zone": "eu-de-02"
  },
  {
    "id": "c095a6be-0d22-4393-8e46-05660e32da3f",
    "name": "subnet-32",
    "cidr": "192.168.32.0/24",
    "status": "ACTIVE",
    "vpc_id": "b4690bfa-6d61-40b4-86b5-e8ab903ef11e",
    "gateway_ip": "192.168.32.1",
    "dhcp_enable": true,
    "primary_dns": "100.125.4.25",
    "secondary_dns": "195.244.235.14",
    "availability_zone": "eu-de-01"
  }
]

Creating a VM with otc-tools

otc ecs create \
--wait --instance-type computev1-1 --instance-name TEST_VM \
--image-name Standard_openSUSE_42.1_JeOS_latest --subnet-name subnet-32 \
--vpc-name vpc-32 --security-group-name sg-ssh --admin-pass "Cloud.4321" \
--key-name SSHkey-205a --public false --disksize 6
{
"server": {
 "availability_zone": "eu-de-02",  "name": "TEST_VM",
 "imageRef": "1cec66fa-036a-4192-835d-75f1568f2821",
 "root_volume": { "volumetype": "SATA", "size": "6" },
 "flavorRef": "computev1-1",  "vpcid": "b4690bfa-6d61-40b4-86b5-e8ab903ef11e",
 "security_groups": [ { "id": "fa96c994-b43c-4793-b763-7b624e690511" } ],
 "nics": [ { "subnet_id": "c095a6be-0d22-4393-8e46-05660e32da3f" } ],
 "key_name": "SSHkey-205a",  "adminPass": "Cloud.4321",  "count": "1"
 }
}
2c9eb2c15487316901549c98469a7e9e
JOBID:   https://ecs.eu-de.otc.t-systems.com/v1/a90ddf531f934a9aac8acebf1abdf19a/jobs/2c9eb2c15487316901549c98469a7e9e
{"status":"INIT","entities":{},"job_id":"2c9eb2c15487316901549c98469a7e9e","job_type":"createServer","begin_time":"2016-05-10T21:35:05.879Z","end_time":"","error_code":null,"fail_reason":null}
#{"status":"RUNNING","entities":{},"job_id":"2c9eb2c15487316901549c98469a7e9e","job_type":"createServer","begin_time":"2016-05-10T21:35:05.879Z","end_time":"","error_code":null,"fail_reason":null}
#{"status":"RUNNING","entities":{"sub_jobs_total":1,"sub_jobs":[]},"job_id":"2c9eb2c15487316901549c98469a7e9e","job_type":"createServer","begin_time":"2016-05-10T21:35:05.879Z","end_time":"","error_code":null,"fail_reason":null}
…
#{"status":"RUNNING","entities":{"sub_jobs_total":1,"sub_jobs":[{"status":"RUNNING","entities":{},"job_id":"2c9eb2c15487316901549c9864af7ea1","job_type":"createSingleServer","begin_time":"2016-05-10T21:35:13.583Z","end_time":"","error_code":null,"fail_reason":null}]},"job_id":"2c9eb2c15487316901549c98469a7e9e","job_type":"createServer","begin_time":"2016-05-10T21:35:05.879Z","end_time":"","error_code":null,"fail_reason":null}
#{"status":"SUCCESS","entities":{"sub_jobs_total":1,"sub_jobs":[{"status":"SUCCESS","entities":{"server_id":"03df968c-72d5-40bf-8f7a-0ac2914c864b"},"job_id":"2c9eb2c15487316901549c9864af7ea1","job_type":"createSingleServer","begin_time":"2016-05-10T21:35:13.583Z","end_time":"2016-05-10T21:38:25.672Z","error_code":null,"fail_reason":null}]},"job_id":"2c9eb2c15487316901549c98469a7e9e","job_type":"createServer","begin_time":"2016-05-10T21:35:05.879Z","end_time":"2016-05-10T21:38:34.569Z","error_code":null,"fail_reason":null}
#ECS Creation status: SUCCESS 

Using OpenStack CLI tools

•     Compute (nova v2) - Nova: 2.22.0 – 2.23.3 (Kilo)
•     Images (glance v2) - Glance: 0.15.0 – 0.16.0 (0.17.x does not work, 1.1.0 does)
•     Block Storage (cinder v2) - Cinder: 1.1.1 – 1.3.1 (all versions tested worked)
•     Networking (neutron v2) - Neutron: 2.3.11/2.3.12 (newest versions fail)
•     IAM (keystone v3) - Keystone: 1.3.4 (only libs needed, client itself deprecated)
•     OpenStack: 1.0.4 – latest

OTC is based on OpenStack Juno plus backports plus extensions ...

Recommended client tools

Preinstalled in openSUSE42.x images (OBS)

docker container (Ubuntu) - docker pull tsiotc/otc-client/

Use pip otherwise

OpenStack CLI example

garloff@ImgFact-JumpHost2:~ [0]$ nova list
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
+--------------------------------------+--------------------+--------+------------+-------------+---------------------------------------------------+
| ID                                   | Name               | Status | Task State | Power State | Networks                                          |
+--------------------------------------+--------------------+--------+------------+-------------+---------------------------------------------------+
| 04213851-9ce9-4fbe-b2eb-42f950822e1c | ecs-8430           | ACTIVE | -          | Running     | b4690bfa-6d61-40b4-86b5-e8ab903ef11e=192.168.33.5 |
| 3902c40e-8f63-49e7-9a46-3147834c9a7d | test-os421-longssh | ACTIVE | -          | Running     | b4690bfa-6d61-40b4-86b5-e8ab903ef11e=192.168.33.4 |
+--------------------------------------+--------------------+--------+------------+-------------+---------------------------------------------------+
garloff@ImgFact-JumpHost2:~ [0]$ glance image-list
+--------------------------------------+------------------------------------------------+
| ID                                   | Name                                           |
+--------------------------------------+------------------------------------------------+
| d5fd5821-c8df-4517-9f5c-ab1be9c2f21e | Enterprise_Windows_ENT_2008R2SP1_20160617-0    |
| b66fb2d3-2fc1-426a-a0af-c9dbc7210612 | Enterprise_Windows_STD_2012R2_20160617-0       |
| f6939c3e-d49b-410d-b62c-caa017e4c0c1 | Standard_openSUSE_42.1_JeOS_latest             |
| be2fdeeb-5b90-4649-a30c-0ede922ba1ec | Standard_openSUSE_42.1_Docker_latest           |
| 59c0757e-dbef-4061-baca-7de06247de57 | Enterprise_SLES12_SP1_latest                   |
[...]
garloff@ImgFact-JumpHost2:~ [0]$ neutron subnet-list
+--------------------------------------+--------------+-----------------+----------------------------------------------------+
| id                                   | name         | cidr            | allocation_pools                                   |
+--------------------------------------+--------------+-----------------+----------------------------------------------------+
| 3a3baa38-f977-4824-89a2-cadd3cadce97 | subnet-33$$0 | 192.168.33.0/24 | {"start": "192.168.33.1", "end": "192.168.33.254"} |
| c7950de4-fbb9-4d7d-a5df-68a295a60fa2 | subnet-32    | 192.168.32.0/24 | {"start": "192.168.32.1", "end": "192.168.32.254"} |
+--------------------------------------+--------------+-----------------+----------------------------------------------------+
garloff@ImgFact-JumpHost2:~ [2]$ cinder list
+--------------------------------------+-----------+--------------------------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  |              Name              | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------------------------+------+-------------+----------+--------------------------------------+
| 16a138a9-57a0-444c-9b61-99b0e84cb313 | available |     CentOS-67-volume-0000      |  4   |     SATA    |   true   |                                      |
| 3bd6bd9e-4399-48e9-be4d-67d390042d51 |   in-use  |      ecs-8430-volume-0000      |  4   |     SATA    |   true   | 04213851-9ce9-4fbe-b2eb-42f950822e1c |
| 4084c3b8-bb90-4ba3-b9ee-76236a4a4070 |   in-use  | test-os421-longssh-volume-0000 |  4   |     SATA    |   true   | 3902c40e-8f63-49e7-9a46-3147834c9a7d |
[...]
garloff@ImgFact-JumpHost2:~ [0]$ openstack server list
+--------------------------------------+--------------------+--------+---------------------------------------------------+
| ID                                   | Name               | Status | Networks                                          |
+--------------------------------------+--------------------+--------+---------------------------------------------------+
| 04213851-9ce9-4fbe-b2eb-42f950822e1c | ecs-8430           | ACTIVE | b4690bfa-6d61-40b4-86b5-e8ab903ef11e=192.168.33.5 |
| 3902c40e-8f63-49e7-9a46-3147834c9a7d | test-os421-longssh | ACTIVE | b4690bfa-6d61-40b4-86b5-e8ab903ef11e=192.168.33.4 |
+--------------------------------------+--------------------+--------+---------------------------------------------------+

neutron support significantly enhanced recently

OpenStack: Create VM

garloff@ImgFact-JumpHost2:~ [0]$ cinder create --image-id bdf485be-8f6d-416e-aedd-2f4df02897bf \
    --availability-zone eu-de-01 --volume-type SATA --name OSTEST_VOLUME 10
+---------------------------------------+--------------------------------------------------------------------------------------------------+
|                Property               |                                              Value                                               |
+---------------------------------------+--------------------------------------------------------------------------------------------------+
|              attachments              |                                                []                                                |
|           availability_zone           |                                             eu-de-01      
[...]
|                   id                  |                               1b98ad0e-60eb-40aa-bb3c-5dcb3bbbd28d                        
[...]
|                 status                |                                             creating                                      
[...]
# Wait for availability (polling, sigh!)
garloff@ImgFact-JumpHost2:~ [0]$ cinder show 1b98ad0e-60eb-40aa-bb3c-5dcb3bbbd28d
[...]
|                 status                |                                             available
[...]
garloff@ImgFact-JumpHost2:~ [0]$ nova boot --flavor computev1-1 --boot-volume 1b98ad0e-60eb-40aa-bb3c-5dcb3bbbd28d --nic net-id=c095a6be-0d22-4393-8e46-05660e32da3f --availability-zone eu-de-01 --key-name SSHkey-205a --security_groups fa96c994-b43c-4793-b763-7b624e690511 OSTEST_VM
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
+--------------------------------------+--------------------------------------------------+
| Property                             | Value                                            |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                           |
| OS-EXT-AZ:availability_zone          | eu-de-01                                         |
[...]
| id                                   | f38b4383-0007-46de-9791-7ecd11357895             |
[...]
| status                               | BUILD                                            |
[...]
garloff@ImgFact-JumpHost2:~ [0]$ nova show f38b4383-0007-46de-9791-7ecd11357895
No handlers could be found for logger "keystoneclient.auth.identity.generic.base"
+----------------------------------------------+----------------------------------------------------------+
| Property                                     | Value                                                    |
+----------------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                            | MANUAL                                                   |
[...]
| status                                       | ACTIVE                                                   |
[...]

   

Previously a third step (port create) would have been needed

CLI summary

We cheated:

• Network(VPC), subnet, keypair preexisting
• Can be created using otc or openstack CLI tools
  (the latter since shortly, thanks to neutron update)
• Complete example will be shown in next blog ...

Real world example:

• Two AZ setup
• JumpHosts, Loadbalancers, Database, BT, Web Tier
   

Somewhat tedious ...

Loadbalancers need non-native calls

Resources

Open Build Service

RPMs / DEBs (uvp-tools, otc-tools, openstackclient, ...)

home:garloff:OTC

OpenTelekomCloud Blog

https://cloud.telekom.de/blog

ImageFactory

https://imagefactory.otc.t-systems.com/

Cheat Sheet

API abstraction & orchestration

ansible

Configuration Management Tool

• Suitable for application management
• But is prepared for REST calls
• PoC: ansible-otc on github

# secrets.yml: account data to adjust
USERNAME: "14610140 OTC00000000001000000210"
PASSWORD: "XXXXXXXXXXXXXX"
DOMAIN:            "OTC00000000001000000210"
PROJECT_ID: "72007ea3e3324e42ad508c278c33e1f5"
[...]
garloff@ImgFact-JumpHost2:~/Ansible/ansible-otc [0]$ ansible-playbook -i hosts subnet.yml 

PLAY [all] *********************************************************************
[...]
TASK [subnet : debug] **********************************************************
ok: [localhost] => {
    "msg": {
        "subnets": [
            {
                "availability_zone": "eu-de-02", 
                "cidr": "192.168.70.0/23", 
                "dhcp_enable": true, 
                "gateway_ip": "192.168.70.1", 
                "id": "8dbf6df6-e904-4824-87df-96594a5a86d7", 
                "name": "subnet-70", 
                "primary_dns": "100.125.4.25", 
                "secondary_dns": "217.150.148.148", 
                "status": "ACTIVE", 
                "vpc_id": "a9ab8cde-1e99-40af-830c-d826f09fe475"
            }, 
            {
                "availability_zone": "eu-de-01", 
                "cidr": "192.168.68.0/23", 
                "dhcp_enable": true, 
                "gateway_ip": "192.168.68.1", 
                "id": "d83f62df-a36e-4bec-bfc1-84937ee0bd02", 
                "name": "subnet-68", 
                "primary_dns": "100.125.4.25", 
                "secondary_dns": "217.150.148.148", 
                "status": "ACTIVE", 
                "vpc_id": "a9ab8cde-1e99-40af-830c-d826f09fe475"
            }, 
            {
                "availability_zone": "eu-de-02", 
                "cidr": "192.168.4.0/24", 

HEAT

Pros

• Native OpenStack solution - fits the OTC brand
• Supports AWS
• Open source

Cons

• No support for Google, Azure or VM Ware
• No Web GUI

Planned for OTC 1.1 (2016Q4)

heat_template_version: 2015-04-30

description: Simple template to deploy a single compute instance

resources:
  my_instance:
    type: OS::Nova::Server
    properties:
      key_name: SSHkey-210
      image: Standard_openSUSE_42.1_latest
      flavor: m1.small

Scalr

Pros

• Simple Paradigm: “If this than that” event based configuration
• Multi platform support including AWS, Google, Azure and OpenStack
• Open source

Cons

• Script callbacks contains logic – low abstraction, implementation can be crude
• No SuSE guests are supported
• No support for application tenant design

OTC support planned 7/2016

Libcloud

Pros

• Multi platform support including AWS, Google, Azure and OpenStack
• Configuration of container and support services (~30 Envs and Services)
• Configuration of Kubernetis Cluster
• Open source (apache foundation)

Cons

• Only python program library as interface
• No support for application tenant design

bosh

Pros

• Unique: Configuration of warden container (Cloudfoundry)
• Multi platform support including AWS, Azure, OpenStack
• Configuration of docker containers
• Open source

Cons

• Currently no support for Google platform, but is planned
• No Web GUI
• Looks “bottom up” – acceptance has to be checked

Support in OTC mostly done

juju

Pros

• Graphical tenant design (CLI is offered too)
• Multi platform support including AWS, Google, Azure and OpenStack
• Management of container* and support services
• Open source, hub for “Bundles” and “Charms”

Cons

• (*) Juju focus on LXC and LXD container, docker container is encapsulated in LXD container

OTC: lack of keystone v2 and nova-network

terraform

Pros

• Multi platform support including AWS, Google, Azure and OpenStack
• Management of container and support services
• Open source

Cons

• No support for application tenant design or templates
• No GUI, unclear if RESTful API
• No Support of VM Ware

OTC: support progressing ...

CliQr

Pros

• Unique: Graphical tenant design (CLI and RESTful API is available too)
• Multi platform support including AWS, Google, Azure, VMware, OpenStack
• Management of container and support services
• Application Templates including market place to share them
• Application Monitoring

Cons

• Not open source, product of Cisco

More ...

Which ones?

You tell us ...
 

Rundeck (CERN)

GooCloud?

...

Outlook: Containers

... will be the unit of management for many next generation applications

VMs beneath expected to be managed automatically?

Deployment models still emerging

Technologies still emerging

Docker, CoreOS, Kubernetes, Mesos, ...

OTC: Cloud Container Engine (Docker/Kubernetes) in 1.0.1 (7/2016)

Conclusions

API support for Automation:
Needed for Agility and Scaling out

Clear commitment to provide native OpenStack APIs on Open Telekom Cloud is progressing

Demonstrated otc-tools, openstack native and ansible

Vast amount of API abstraction tools available

Broad tool support requires more than DefCore APIs

Questions &

Call to Action

Use Open Telekom Cloud

Use the API and Tools

Feedback welcome - where are we lacking?

Work with us

- or FOR us!

Contact: /me

kurt.garloff@t-systems.com

t-systems@garloff.de

Credits

Hui-Bin Ma: Great support with API enablement

Christian Kortwich: API abstration tools survey

Zsolt Nagy: otc-tools (also python+Java tools)

Frank K / Reik K: ansible-otc

Bernd R & Anthony C: API testing ...