Cloud is mostly useless without API usage ...

Kurt Garloff

OpenStack Cloud Architect

kurt.garloff@t-systems.com

3rd Cloud Camp, Darmstadt, 21.-22.6.2017

Introduction to API usage on OTC

Mode 1 Enterprise IT

Reliable

Highly available Infrastructure

Carefully planned

Avoiding
risks (by avoiding changes)

Evolving slowly

Managed by experts

Expensive
€€€€€

Software Companies
set the bar higher

Customer experience

Dynamic Adaptation

Scale

Cost

Multi-
Device

Multi-
Channel

Innovation

Service

oriented

Experimentation

Data
driven

Agile Development

Waterfall Software Development has painful, risky phases

  • Integration
  • Releases
  • Integration & User-Acceptance tests
  • Customer feedback & customer issues
  • Mistakes early-on are uncovered late,
    consuming a lot of time & money

 

Rather than avoiding pain, do it all the time continuously
"Move left" - Do risky things early & often

  • Build MVP, iterate to improve
  • Continuous integration (CI)
  • Continuous testing
  • Releases after each sprint (SCRUM)
  • Customer feedback for each release

Everything great?

So we successfully implemented

Agile Development ...

Business

  • Marketing can not communicate constant improvements
  • Fail to gather feedback
  • Reliance on OSS/BSS that mandates slow releases
  • Product planning & approval processes designed for static world

Infrastructure

  • Infrastructure
    needs to be
    ordered, installed, deployed, changed
  • Software deployment to Production risky and slow
  • Test environment very different from Production
  • Operations hates the constant risk

Non agile environment

What if?

  • we could create infrastructure as needed?
  • we could manage infrastructure like software?
  • we could easily change infrastructure
    • responding to application architecture changes?
    • or even just to changed workload?
    • or failures?

 

We could use it to ...

  • Setting up realistic test environments
  • Adapting to market needs
  • Create understanding for Production needs in Dev

This is the cloud

(hyperscale cloud - public cloud architecture

not: enterprise virtualization)

Infrastructure with a standardized API

Infrastructure as code

On-demand Self-service
w/ rapid elasticity
and a large resource pool
Broadly Networked
Metered (pay per use)

AgileDev + CloudInfra

enables DevOps

DevOps is a software delivery process that emphasizes communication and collaboration across the end-to-end value stream from concept to market, including product management, software development, and operations professionals; while automating the process of software integration, testing, deployment and infrastructure changes. It aims to establish a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

Cloud for classical envs

Cloud Installation

And the result is ...

What went wrong?

Culture

  • Manual approvals

Technology

  • Manual installations

AUTOMATION

Text

CI/CD

Test driven

Version Control

Redeployment

Microservices

Modular APIs

Autoscaling

Building Blocks

Open Source

Automation

Automating virtual Infrastructure

  • DIY REST (curl, jq)
  • libcloud
  • ansible
  • heat
  • terraform
  • juju
  • bosh
  • ...

OpenStack REST API

cloud-init

- controlled by meta-data src

- run upon boot

- user accounts, ssh keys

- hostname, timezone

- networks, mounts

- package updates & installation

- arbitrary scripts

- phone home

- subject VM to config mgmt

Config Mgmt Tools

- ansible, chef, saltstack, puppet

- run repeatedly to ensure compliance or changes

- configuration properties

- software packages

 

- can control infra as well

Application Management

Open Telekom Cloud

Tools & API

Text

OpenStack REST API

  • REST: https requests (GET/HEAD/POST/PUT/PATCH/DELETE/...)
    • JSON objects { "tag": "value",
      "struct": { "stag1": "string", "stag2": number },
      "array": [ "val1", "val2", "val3" ] }
  • It all starts with keystone:
    • Authenticate with username, password (POST)
    • You get 401 (unauthorized) or
      • The OpenStack service catalog (names, endpoints)
      • A token (valid max. 24h)
  • Use the token in the header to talk to the services

Environment ~/.ostackrc

# Adapt the first three lines ...
export OS_PASSWORD="XXXXXXXXXXXXXXXXXX"
export OS_USERNAME="14610698 OTC00000000001000XXXXXX"
export OS_USER_DOMAIN_NAME="OTC00000000001000XXXXXX"
export OS_TENANT_NAME=eu-de
export OS_PROJECT_NAME=eu-de
export OS_AUTH_URL=https://iam.eu-de.otc.t-systems.com/v3
export OS_PROJECT_DOMAIN_NAME=
export OS_IDENTITY_API_VERSION=3
export OS_VOLUME_API_VERSION=2
export OS_IMAGE_API_VERSION=2
export OS_ENDPOINT_TYPE=publicURL
export NOVA_ENDPOINT_TYPE=publicURL
export CINDER_ENDPOINT_TYPE=publicURL

Template preinstalled in our images

OS_USERNAME example for ICU,
freely chosable for local users/IdP federation

Keep it SECURE!

Alternative: os-client-config: ~/.config/openstack/clouds.yaml

ap-sg
-AP-SG-

Simple example

garloff@ImgFact-Build2:~ [1]$ otc.sh debug custom GET \$NOVA_URL/servers
DEBUG: curl -i -sS -H Content-Type: application/json -d {
   "auth": {
    "identity": { "methods": [ "password" ],
     "password": {
      "user": { "name": "garloff", "password": "SECRET", 
                "domain": { "name": "OTC00000000001000000205" } } } },
    "scope": { "project": { "name": "eu-de" } }
 } }   https://iam.eu-de.otc.t-systems.com:443/v3/auth/tokens
DEBUG: (0) HTTP/1.1 201 Created
Server: nginx
Date: Wed, 21 Jun 2017 17:29:30 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Iam-Trace-Id: 658ec75a-3e93-49dc-b4cd-ebb84cc09656
X-Subject-Token: MIIsecretsecret
[...]

{"token":{"issued_at":"2017-06-21T17:29:30.738000Z","expires_at":"2017-06-22T17:29:30.738000Z",
"methods":["password"],"project":{"name":"eu-de","id":"a90ddf531f934a9aac8acebf1abdf19a",
"domain: {"name":"OTC00000000001000000205","id":"167dd7366b734d789bdaf448050d7966",
"xdomain_type":"TSI","xdomain_id":"00000000001000000205"}},
"user":{"domain":{"name":"OTC00000000001000000205","id":"167dd7366b734d789bdaf448050d7966", "xdomain_type":"TSI","xdomain_id":"00000000001000000205"},"id":"7f689f85708f4f8c886cf8be44bd6ccb","name":"garloff"},
"catalog":[{"type":"dns","id":"56cd81db1f8445d98652479afe07c5ba","name":"", 
"endpoints":[{"url":"https://dns.eu-de.otc.t-systems.com","region":"eu-de","region_id":"eu-de", "interface":"public","id":"0047a06690484d86afe04877074efddf"}]},{"type":"image","id":"90095f474f054b4ba6e029bc398ccb59","name":"", 
"endpoints":[{"url":"https://ims.eu-de.otc.t-systems.com","region":"eu-de","region_id":"eu-de", "interface":"public","id":"1360c0dc80654c5790c1b6d210f34746"}]},{"type":"orchestration","id":"233635b49a5b4ba48f6615f84ffc6694","name":"", "endpoints":[{"url":"https://rts.eu-de.otc.t-systems.com/v1/a90ddf531f934a9aac8acebf1abdf19a","region":"eu-de","region_id":"eu-de","interface":"public","id":"2bbfa8fc6ea246eb8556d2e2525c46d8"}]}, ... ], 
"roles":[{"name":"te_agency","id":"41ce5857c94c4775b6cc53d5feeba445"},{"name":"te_admin","id":"699bd62cda304d2cad03fd2fb190b8cf"},{"id":"0","name":"op_gated_tag2"}]}}
DEBUG: curl -sS -X GET -H Content-Type: application/json -H Accept: application/json
 -H X-Auth-Token: MIIsecretsecret -H X-Language: en-us 
https://ecs.eu-de.otc.t-systems.com/v2/a90ddf531f934a9aac8acebf1abdf19a/servers
{"servers":[{"name":"Kurt-Private-Server","links":[{"rel":"self", "href":"https://ecs.eu-de.otc.t-systems.com:443/v2/a90ddf531f934a9aac8acebf1abdf19a/servers/d70cb6f7-de84-44fb-8723-c56de3077013"}, {"rel":"bookmark","href":"https://ecs.eu-de.otc.t-systems.com:443/a90ddf531f934a9aac8acebf1abdf19a/servers/d70cb6f7-de84-44fb-8723-c56de3077013"}],"id":"d70cb6f7-de84-44fb-8723-c56de3077013"}, 
{"name":"JumpHost--SAP","links":[{ ... ]} ]}

TOKEN

Catalog

Service

otc.sh (otc-tools)

  • Uses the OS_ environment for config (like OpenStack CLI tools)
  • Test & Demo tool using bash, curl, jq
  • Supports a number of OpenStack APIs: keystone, nova, neutron, cinder, glance, heat, designate
  • Plus some OTC specific ones:
    CCE (the non k8s pieces), ELB, RDS, Cloud-Eye, SMN, DMS
  • Does not target completeness (patches + feature requests welcome!)
  • otc.sh mds (query metadata service)
  • otc.sh custom GET/POST/...  for custom commands (API doc)

python-otcclient

  • CLI tool otc.py like otc-tools
  • Can use the OS_ environment for config (normally own config)
  • AWS style syntax (for migrations)
  • Broader coverage of OTC specific functions
  • Extensible (plugins), python
  • Eventually will replace otc-tools
    • more maintainable
  • Used as reference implementation from OTC QA team

Using OpenStack CLI tools

  • Compute (nova v2) - novaclient-6.0.x (broad version support since end of May)
  • Images (glance v2) - glanceclient-2.5.0
  • Block Storage (cinder v2) - cinderclient-1.9.0
  • Networking (neutron v2) - neutronclient-6.0.x (backport needed for --enable-snat, LBaasv2 and VPNaaS)
  • IAM (keystone v3) - keystoneclient-3.5.1, keystoneauth1-2.12.3 (only libs)
  • DNS (designate v2) - designateclient-2.3.0 (CLI support via openstackclient)
  • Orchestration (heat) - heatclient-1.5.0
  • Unified CLI client: openstackclient-3.2.1 [*], openstacksdk-0.9.5
  • Near future: ironicclient, troveclient

 

OTC is based on OpenStack Juno Mitaka + backports + extensions ...

Recommended client tools: NEWTON

Preinstalled in openSUSE42 images (OBS) - soon CentOS7

docker container (Ubuntu) - docker pull tsiotc/otc-client/

Use pip otherwise

NEW!

Broad support ...

Example

# Helper functions
$ getid() { FIELD=${1:-id}; grep "^| $FIELD " | sed -e 's/^|[^|]*| \([^|]*\) |.*$/"\1"/' -e 's/ *$//'; }
$ listid() { grep $1 | tail -n1 | sed 's/^| \([0-9a-f-]*\) .*$/\1/'; }
# Create VPC - Router
$ neutron router-create vpc-192 | getid
"da73d52a-82e7-48a7-9e00-5a524ea8bd2c"
# Net (L2) and subnet (L3), connect subnet to router
$ neutron net-create net-192 | getid
"2a3830b7-b17e-4493-a647-b1e21db181ea"
$ neutron subnet-create --dns-nameserver 100.125.4.25 --dns-nameserver 8.8.8.8 --name subnet-192 net-192 192.168.192.0/20 | getid
"d929a764-effc-4110-8da9-84d288f6fcc0"
$ neutron router-interface-add vpc-192 subnet-192
Added interface 814f1189-ebcc-467c-bf8d-950040870497 to router vpc-192.
# Enable SNAT
$ neutron router-gateway-set vpc-192 admin_external_net --enable-snat
Set gateway for router vpc-192
# We could boot from image, but we want a larger root volume, poll manually to wait (ugly)
$ cinder create --image Standard_openSUSE_42_JeOS_latest --name OS42-BootVol 10 | getid
"e4c3e6ae-93da-4876-bc8f-70a933ef7656"
$ while true; do cinder show e4c3e6ae-93da-4876-bc8f-70a933ef7656 | getid status | grep available && break; sleep 5; done
available
# Security group setup (output shortened)
$ neutron security-group-create sg-ssh | getid
"118f7bc0-5d9c-4c71-9681-eeefc8ac71ac"
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --remote-group-id 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 22 \
     --port-range-max 22 --remote-ip-prefix 0/0 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol icmp --port-range-min 8 --port-range-max 0 --remote-ip-prefix 0/0 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
# SSH keypair, key gets injected via meta-data server to cloud-init in image
$ nova keypair-add SSHkey-192 > SSHkey-192.pem
# Finally create a VM
$ nova boot --flavor computev1-1 --boot-volume e4c3e6ae-93da-4876-bc8f-70a933ef7656 \
    --nic net-id=2a3830b7-b17e-4493-a647-b1e21db181ea --security-groups 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac \
    --key-name SSHkey-192 --poll TESTVM_OS42_192 | getid
"ed7f4e13-08ef-4951-9c57-fee147b9590f"
$ nova interface-list ed7f4e13-08ef-4951-9c57-fee147b9590f
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
| Port State | Port ID                              | Net ID                               | IP addresses    | MAC Addr          |
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
| ACTIVE     | 44972f58-b753-445e-9932-d08a26661aa5 | 2a3830b7-b17e-4493-a647-b1e21db181ea | 192.168.192.153 | fa:16:3e:20:07:ba |
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
$ ssh -i SSHkey-192.pem linux@192.168.192.153
# Will only work from a VM on the same VPC, as we don't have a floating IP assigned




Manual 10GB volume

Outgoing internet

Internal DNS recommended

Img: Standard user name

Cross-AZ network

openstackclient CLI

Same can be done with openstackclient

 

- Recommended for the future over old single clients

- Not yet covering all aspects, e.g. in neutron

(but being addressed quickly -- check out later versions ...)

- Plugin architecture

- Plugins for OTC extensions under development

Discussion

Can maintain scripts in git to setup test & prod envs

Infrastructure as code - yeah!

Very flexible
 

Will get a bit complex when adding error handling

Even more complex when trying to deal with half-setup envs

Declarative description/model of wanted infra

Better approach

Infra modeling

OpenStack heat

+ CloudFormation for O/S

+ Good coverage

+ Easy to learn
- Only supports O/S

Ansible

+ Standard on O/S

+ Works well with OTC

+ Integr w/ AppMgmt

Terraform

+ Multi-Cloud support

o Reasonable support for OTC

Cloudify

+- TOSCA

+ GUI

o Quality concerns

Common workflow

Keep infra description in code repository
(declarative, yaml or similar)
Versions, branches, history, comments, ...

Consider it part of the code base - DevOps!

Roll out to test environment - CD

Tests against test env uncover infra errors

Additional reviews / testing guarding production

Infra Tests reusable for monitoring

Perf tests possibly even for auto-scaling


E2E automation

HEAT overview

Resources: Virtual objects (e.g. network, subnet, server, disk, image, ...)

Heat Orchestration Templates (HOT):
List of resources with parameters and relationships / dependencies

Templates are described with YAML files

Stack: Deployable set of connected resources (as described in template)

HEAT Engine: The receiver of the templates talking to all services of the resource types to create / destroy the resources to instantiate (or remove)  a stack

HEAT template

heat_template_version: '2014-10-16'

description: >
  HOT template to create single Wordpress instance on OTC
  Creates new router and private subnet, new security group
  Downloads and installs MariaDB, PHP, Apache and Wordpress

parameters:

  key_name:
    type: string
    description: Name of a KeyPair to enable SSH access
    default: wp_demo_key
  flavor:
    type: string
    description: Instance type for WordPress server
    default: c1.medium
  image:
    type: string
    description: OTC Images
    default: Standard_CentOS_7_latest
  public_net_id:
    type: string
    description: ID of OTC public network
    default: 0a2228f2-7f8a-45f1-8e09-9039e1d09975
[...]

resources:
  private_net:
    type: OS::Neutron::Net
    properties:
      name: { get_param: private_net_name }

  private_subnet:
    type: OS::Neutron::Subnet
    properties:
      network_id: { get_resource: private_net }
      cidr: { get_param: private_net_cidr }
      gateway_ip: { get_param: private_net_gateway }
      dns_nameservers: { get_param: private_net_dns }
      allocation_pools:
        - start: { get_param: private_net_pool_start }
          end: { get_param: private_net_pool_end }

  router:
    type: OS::Neutron::Router
    properties:
      external_gateway_info:
        network: { get_param: public_net_id }

  router_interface:
    type: OS::Neutron::RouterInterface
    properties:
      router_id: { get_resource: router }
      subnet_id: { get_resource: private_subnet }
[...]
  wp_demo_instance:
    type: OS::Nova::Server
    depends_on: router_interface
    properties:
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }
      networks:
        - port: {get_resource: wp_demo_port}
      user_data_format: RAW
      user_data:
[...]

outputs:
  PublicIP:
    description: IP address in public network
    value:
      get_attr: [wp_demo_floating_ip, floating_ip_address]
  PrivateIp:
    description: IP address in private network
    value:
      get_attr: [wp_demo_instance, first_address]
  WebsiteURL:
    description: URL for Wordpress wiki

Variable inputs

Objects to create

Reference inputs

& dependencies

Dependency

Output parms

git clone https://github.com/OpenTelekomCloud/heat-templates

Deployment

]$ openstack stack create --template WordPress_Single_instance.yaml CloudCampStack
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                        |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                  | 7d644f68-f685-40a9-9abf-25493cb658f1                                                                                                                                         |
| stack_name          | CloudCampStack                                                                                                                                                               |
| description         | HOT template to create single Wordpress instance on OTC Creates new router and private subnet, new security group Downloads and installs MariaDB, PHP, Apache and Wordpress  |
|                     |                                                                                                                                                                              |
| creation_time       | 2017-06-21T22:27:16.609094                                                                                                                                                   |
| updated_time        | None                                                                                                                                                                         |
| stack_status        | CREATE_IN_PROGRESS                                                                                                                                                           |
| stack_status_reason | Stack CREATE started                                                                                                                                                         |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack stack show CloudCampStack
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                 | Value                                                                                                                                                                        |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                    | 7d644f68-f685-40a9-9abf-25493cb658f1                                                                                                                                         |
| stack_name            | CloudCampStack                                                                                                                                                               |
| description           | HOT template to create single Wordpress instance on OTC Creates new router and private subnet, new security group Downloads and installs MariaDB, PHP, Apache and Wordpress  |
|                       |                                                                                                                                                                              |
| creation_time         | 2017-06-21T22:27:16.609094                                                                                                                                                   |
| updated_time          | None                                                                                                                                                                         |
| stack_status          | CREATE_FAILED                                                                                                                                                                |
| stack_status_reason   | Resource CREATE failed: BadRequest: resources.wp_demo_instance: Invalid key_name provided. (HTTP 400)                                                                        |
| parameters            | OS::project_id: a90ddf531f934a9aac8acebf1abdf19a                                                              
[...]
| outputs               | - description: IP address in public network                                                                                                                                  |
|                       |   output_key: PublicIP                                                                                                                                                       |
|                       |   output_value: null                                                                                          
[...]

Template assumes non-existing key

Output params

Daniela Ebert @ OpenStack Summit:

Ansible

ConfigManagement solution

Good REST support (2.1+)

Used in InterOp challenge in Barcelona
(16 companies showing live deployments on their cloud using the same automation)

python-shade abstracts away some of the differences b/w OpenStack clouds

YAML again

ansible-playbook -e "action=apply env=ENVIRON password=XXXXX" site.yml

Resources

Open Build Service

RPMs / DEBs (uvp-tools, otc-tools, openstackclient, ...)

home:garloff:OTC

OpenTelekomCloud Blog

https://cloud.telekom.de/blog

Slides

https://slides.com/kgarloff/

ImageFactory

https://imagefactory.otc.t-systems.com/

Questions &

Call to Action

Understand why you adopt Cloud

Public Cloud Architecture w/o automation
leaves most benefits on the table

Use OpenStack APIs to automate

Manage your Infrastructure as Code

Use declarative description (heat or similar)

Contact: /me or other OTC engineers

kurt.garloff@t-systems.com

t-systems@garloff.de

Cheat Sheet

Service openstack OTC
compute (VMs) nova v2 (server) ECS
block storage cinder v2 (volume) EVS
object storage [SWIFT] OBS (S3)
network
neutron (net, router, floating IP, VPN) VPC  (EIP)
images glance v2 IMS
authentication
(keystone v3) IAM / MyWorkPl
LoadBalancer
neutron LBaaSv2 ELB / LBaaSv2
Relat Database
(trove) RDS (1.0.1)
Containers (magnum / k8s) CCE (1.0.1) / k8s
BareMetal ironic BMS (2.1)
Orchestration heat RTS (1.1*)

Another API use case

Made with Slides.com