Cloud is mostly useless without API usage ...
Kurt Garloff
OpenStack Cloud Architect
kurt.garloff@t-systems.com
3rd Cloud Camp, Darmstadt, 21.-22.6.2017
Introduction to API usage on OTC
Mode 1 Enterprise IT
Reliable
Highly available Infrastructure
Carefully planned
Avoiding
risks (by avoiding changes)
Evolving slowly
Managed by experts
Expensive
€€€€€
Software Companies
set the bar higher
Customer experience
Dynamic Adaptation
Scale
Cost
Multi-
Device
Multi-
Channel
Innovation
Service
oriented
Experimentation
Data
driven
Agile Development
Waterfall Software Development has painful, risky phases
- Integration
- Releases
- Integration & User-Acceptance tests
- Customer feedback & customer issues
- Mistakes early-on are uncovered late,
consuming a lot of time & money
Rather than avoiding pain, do it all the time continuously
"Move left" - Do risky things early & often
- Build MVP, iterate to improve
- Continuous integration (CI)
- Continuous testing
- Releases after each sprint (SCRUM)
- Customer feedback for each release
Everything great?
So we successfully implemented
Agile Development ...
Business
- Marketing can not communicate constant improvements
- Fail to gather feedback
- Reliance on OSS/BSS that mandates slow releases
- Product planning & approval processes designed for static world
Infrastructure
- Infrastructure
needs to be
ordered, installed, deployed, changed - Software deployment to Production risky and slow
- Test environment very different from Production
- Operations hates the constant risk
Non agile environment
What if?
- we could create infrastructure as needed?
- we could manage infrastructure like software?
- we could easily change infrastructure
- responding to application architecture changes?
- or even just to changed workload?
- or failures?
We could use it to ...
- Setting up realistic test environments
- Adapting to market needs
- Create understanding for Production needs in Dev
This is the cloud
(hyperscale cloud - public cloud architecture
not: enterprise virtualization)
Infrastructure with a standardized API
Infrastructure as code
On-demand Self-service
w/ rapid elasticity
and a large resource pool
Broadly Networked
Metered (pay per use)
AgileDev + CloudInfra
enables DevOps
DevOps is a software delivery process that emphasizes communication and collaboration across the end-to-end value stream from concept to market, including product management, software development, and operations professionals; while automating the process of software integration, testing, deployment and infrastructure changes. It aims to establish a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.
Cloud for classical envs
Cloud Installation
And the result is ...
What went wrong?
Culture
- Manual approvals
Technology
- Manual installations
AUTOMATION
Text
CI/CD
Test driven
Version Control
Redeployment
Microservices
Modular APIs
Autoscaling
Building Blocks
Open Source
Automation
Automating virtual Infrastructure
- DIY REST (curl, jq)
- libcloud
- ansible
- heat
- terraform
- juju
- bosh
- ...
OpenStack REST API
cloud-init
- controlled by meta-data src
- run upon boot
- user accounts, ssh keys
- hostname, timezone
- networks, mounts
- package updates & installation
- arbitrary scripts
- phone home
- subject VM to config mgmt
Config Mgmt Tools
- ansible, chef, saltstack, puppet
- run repeatedly to ensure compliance or changes
- configuration properties
- software packages
- can control infra as well
Application Management
Open Telekom Cloud
Tools & API
Text
OpenStack REST API
- REST: https requests (GET/HEAD/POST/PUT/PATCH/DELETE/...)
- JSON objects { "tag": "value",
"struct": { "stag1": "string", "stag2": number },
"array": [ "val1", "val2", "val3" ] }
- JSON objects { "tag": "value",
- It all starts with keystone:
- Authenticate with username, password (POST)
- You get 401 (unauthorized) or
- The OpenStack service catalog (names, endpoints)
- A token (valid max. 24h)
- Use the token in the header to talk to the services
Environment ~/.ostackrc
# Adapt the first three lines ...
export OS_PASSWORD="XXXXXXXXXXXXXXXXXX"
export OS_USERNAME="14610698 OTC00000000001000XXXXXX"
export OS_USER_DOMAIN_NAME="OTC00000000001000XXXXXX"
export OS_TENANT_NAME=eu-de
export OS_PROJECT_NAME=eu-de
export OS_AUTH_URL=https://iam.eu-de.otc.t-systems.com/v3
export OS_PROJECT_DOMAIN_NAME=
export OS_IDENTITY_API_VERSION=3
export OS_VOLUME_API_VERSION=2
export OS_IMAGE_API_VERSION=2
export OS_ENDPOINT_TYPE=publicURL
export NOVA_ENDPOINT_TYPE=publicURL
export CINDER_ENDPOINT_TYPE=publicURL
Template preinstalled in our images
OS_USERNAME example for ICU,
freely chosable for local users/IdP federation
Keep it SECURE!
Alternative: os-client-config: ~/.config/openstack/clouds.yaml
ap-sg
-AP-SG-
Simple example
garloff@ImgFact-Build2:~ [1]$ otc.sh debug custom GET \$NOVA_URL/servers
DEBUG: curl -i -sS -H Content-Type: application/json -d {
"auth": {
"identity": { "methods": [ "password" ],
"password": {
"user": { "name": "garloff", "password": "SECRET",
"domain": { "name": "OTC00000000001000000205" } } } },
"scope": { "project": { "name": "eu-de" } }
} } https://iam.eu-de.otc.t-systems.com:443/v3/auth/tokens
DEBUG: (0) HTTP/1.1 201 Created
Server: nginx
Date: Wed, 21 Jun 2017 17:29:30 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
X-Iam-Trace-Id: 658ec75a-3e93-49dc-b4cd-ebb84cc09656
X-Subject-Token: MIIsecretsecret
[...]
{"token":{"issued_at":"2017-06-21T17:29:30.738000Z","expires_at":"2017-06-22T17:29:30.738000Z",
"methods":["password"],"project":{"name":"eu-de","id":"a90ddf531f934a9aac8acebf1abdf19a",
"domain: {"name":"OTC00000000001000000205","id":"167dd7366b734d789bdaf448050d7966",
"xdomain_type":"TSI","xdomain_id":"00000000001000000205"}},
"user":{"domain":{"name":"OTC00000000001000000205","id":"167dd7366b734d789bdaf448050d7966", "xdomain_type":"TSI","xdomain_id":"00000000001000000205"},"id":"7f689f85708f4f8c886cf8be44bd6ccb","name":"garloff"},
"catalog":[{"type":"dns","id":"56cd81db1f8445d98652479afe07c5ba","name":"",
"endpoints":[{"url":"https://dns.eu-de.otc.t-systems.com","region":"eu-de","region_id":"eu-de", "interface":"public","id":"0047a06690484d86afe04877074efddf"}]},{"type":"image","id":"90095f474f054b4ba6e029bc398ccb59","name":"",
"endpoints":[{"url":"https://ims.eu-de.otc.t-systems.com","region":"eu-de","region_id":"eu-de", "interface":"public","id":"1360c0dc80654c5790c1b6d210f34746"}]},{"type":"orchestration","id":"233635b49a5b4ba48f6615f84ffc6694","name":"", "endpoints":[{"url":"https://rts.eu-de.otc.t-systems.com/v1/a90ddf531f934a9aac8acebf1abdf19a","region":"eu-de","region_id":"eu-de","interface":"public","id":"2bbfa8fc6ea246eb8556d2e2525c46d8"}]}, ... ],
"roles":[{"name":"te_agency","id":"41ce5857c94c4775b6cc53d5feeba445"},{"name":"te_admin","id":"699bd62cda304d2cad03fd2fb190b8cf"},{"id":"0","name":"op_gated_tag2"}]}}
DEBUG: curl -sS -X GET -H Content-Type: application/json -H Accept: application/json
-H X-Auth-Token: MIIsecretsecret -H X-Language: en-us
https://ecs.eu-de.otc.t-systems.com/v2/a90ddf531f934a9aac8acebf1abdf19a/servers
{"servers":[{"name":"Kurt-Private-Server","links":[{"rel":"self", "href":"https://ecs.eu-de.otc.t-systems.com:443/v2/a90ddf531f934a9aac8acebf1abdf19a/servers/d70cb6f7-de84-44fb-8723-c56de3077013"}, {"rel":"bookmark","href":"https://ecs.eu-de.otc.t-systems.com:443/a90ddf531f934a9aac8acebf1abdf19a/servers/d70cb6f7-de84-44fb-8723-c56de3077013"}],"id":"d70cb6f7-de84-44fb-8723-c56de3077013"},
{"name":"JumpHost--SAP","links":[{ ... ]} ]}
TOKEN
Catalog
Service
otc.sh (otc-tools)
- Uses the OS_ environment for config (like OpenStack CLI tools)
- Test & Demo tool using bash, curl, jq
- Supports a number of OpenStack APIs: keystone, nova, neutron, cinder, glance, heat, designate
- Plus some OTC specific ones:
CCE (the non k8s pieces), ELB, RDS, Cloud-Eye, SMN, DMS - Does not target completeness (patches + feature requests welcome!)
- otc.sh mds (query metadata service)
- otc.sh custom GET/POST/... for custom commands (API doc)
python-otcclient
- CLI tool otc.py like otc-tools
- Can use the OS_ environment for config (normally own config)
- AWS style syntax (for migrations)
- Broader coverage of OTC specific functions
- Extensible (plugins), python
- Eventually will replace otc-tools
- more maintainable
- Used as reference implementation from OTC QA team
Using OpenStack CLI tools
- Compute (nova v2) - novaclient-6.0.x (broad version support since end of May)
- Images (glance v2) - glanceclient-2.5.0
- Block Storage (cinder v2) - cinderclient-1.9.0
- Networking (neutron v2) - neutronclient-6.0.x (backport needed for --enable-snat, LBaasv2 and VPNaaS)
- IAM (keystone v3) - keystoneclient-3.5.1, keystoneauth1-2.12.3 (only libs)
- DNS (designate v2) - designateclient-2.3.0 (CLI support via openstackclient)
- Orchestration (heat) - heatclient-1.5.0
- Unified CLI client: openstackclient-3.2.1 [*], openstacksdk-0.9.5
- Near future: ironicclient, troveclient
OTC is based on OpenStack Juno Mitaka + backports + extensions ...
Recommended client tools: NEWTON
Preinstalled in openSUSE42 images (OBS) - soon CentOS7
docker container (Ubuntu) - docker pull tsiotc/otc-client/
Use pip otherwise
NEW!
Broad support ...
Example
# Helper functions
$ getid() { FIELD=${1:-id}; grep "^| $FIELD " | sed -e 's/^|[^|]*| \([^|]*\) |.*$/"\1"/' -e 's/ *$//'; }
$ listid() { grep $1 | tail -n1 | sed 's/^| \([0-9a-f-]*\) .*$/\1/'; }
# Create VPC - Router
$ neutron router-create vpc-192 | getid
"da73d52a-82e7-48a7-9e00-5a524ea8bd2c"
# Net (L2) and subnet (L3), connect subnet to router
$ neutron net-create net-192 | getid
"2a3830b7-b17e-4493-a647-b1e21db181ea"
$ neutron subnet-create --dns-nameserver 100.125.4.25 --dns-nameserver 8.8.8.8 --name subnet-192 net-192 192.168.192.0/20 | getid
"d929a764-effc-4110-8da9-84d288f6fcc0"
$ neutron router-interface-add vpc-192 subnet-192
Added interface 814f1189-ebcc-467c-bf8d-950040870497 to router vpc-192.
# Enable SNAT
$ neutron router-gateway-set vpc-192 admin_external_net --enable-snat
Set gateway for router vpc-192
# We could boot from image, but we want a larger root volume, poll manually to wait (ugly)
$ cinder create --image Standard_openSUSE_42_JeOS_latest --name OS42-BootVol 10 | getid
"e4c3e6ae-93da-4876-bc8f-70a933ef7656"
$ while true; do cinder show e4c3e6ae-93da-4876-bc8f-70a933ef7656 | getid status | grep available && break; sleep 5; done
available
# Security group setup (output shortened)
$ neutron security-group-create sg-ssh | getid
"118f7bc0-5d9c-4c71-9681-eeefc8ac71ac"
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --remote-group-id 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 22 \
--port-range-max 22 --remote-ip-prefix 0/0 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
$ neutron security-group-rule-create --direction ingress --ethertype IPv4 --protocol icmp --port-range-min 8 --port-range-max 0 --remote-ip-prefix 0/0 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac
# SSH keypair, key gets injected via meta-data server to cloud-init in image
$ nova keypair-add SSHkey-192 > SSHkey-192.pem
# Finally create a VM
$ nova boot --flavor computev1-1 --boot-volume e4c3e6ae-93da-4876-bc8f-70a933ef7656 \
--nic net-id=2a3830b7-b17e-4493-a647-b1e21db181ea --security-groups 118f7bc0-5d9c-4c71-9681-eeefc8ac71ac \
--key-name SSHkey-192 --poll TESTVM_OS42_192 | getid
"ed7f4e13-08ef-4951-9c57-fee147b9590f"
$ nova interface-list ed7f4e13-08ef-4951-9c57-fee147b9590f
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
| ACTIVE | 44972f58-b753-445e-9932-d08a26661aa5 | 2a3830b7-b17e-4493-a647-b1e21db181ea | 192.168.192.153 | fa:16:3e:20:07:ba |
+------------+--------------------------------------+--------------------------------------+-----------------+-------------------+
$ ssh -i SSHkey-192.pem linux@192.168.192.153
# Will only work from a VM on the same VPC, as we don't have a floating IP assigned
Manual 10GB volume
Outgoing internet
Internal DNS recommended
Img: Standard user name
Cross-AZ network
openstackclient CLI
Same can be done with openstackclient
- Recommended for the future over old single clients
- Not yet covering all aspects, e.g. in neutron
(but being addressed quickly -- check out later versions ...)
- Plugin architecture
- Plugins for OTC extensions under development
Discussion
Can maintain scripts in git to setup test & prod envs
Infrastructure as code - yeah!
Very flexible
Will get a bit complex when adding error handling
Even more complex when trying to deal with half-setup envs
Declarative description/model of wanted infra
Better approach
Infra modeling
OpenStack heat
+ CloudFormation for O/S
+ Good coverage
+ Easy to learn
- Only supports O/S
Ansible
+ Standard on O/S
+ Works well with OTC
+ Integr w/ AppMgmt
Terraform
+ Multi-Cloud support
o Reasonable support for OTC
Cloudify
+- TOSCA
+ GUI
o Quality concerns
Common workflow
Keep infra description in code repository
(declarative, yaml or similar)
Versions, branches, history, comments, ...
Consider it part of the code base - DevOps!
Roll out to test environment - CD
Tests against test env uncover infra errors
Additional reviews / testing guarding production
Infra Tests reusable for monitoring
Perf tests possibly even for auto-scaling
E2E automation
HEAT overview
Resources: Virtual objects (e.g. network, subnet, server, disk, image, ...)
Heat Orchestration Templates (HOT):
List of resources with parameters and relationships / dependencies
Templates are described with YAML files
Stack: Deployable set of connected resources (as described in template)
HEAT Engine: The receiver of the templates talking to all services of the resource types to create / destroy the resources to instantiate (or remove) a stack
HEAT template
heat_template_version: '2014-10-16'
description: >
HOT template to create single Wordpress instance on OTC
Creates new router and private subnet, new security group
Downloads and installs MariaDB, PHP, Apache and Wordpress
parameters:
key_name:
type: string
description: Name of a KeyPair to enable SSH access
default: wp_demo_key
flavor:
type: string
description: Instance type for WordPress server
default: c1.medium
image:
type: string
description: OTC Images
default: Standard_CentOS_7_latest
public_net_id:
type: string
description: ID of OTC public network
default: 0a2228f2-7f8a-45f1-8e09-9039e1d09975
[...]
resources:
private_net:
type: OS::Neutron::Net
properties:
name: { get_param: private_net_name }
private_subnet:
type: OS::Neutron::Subnet
properties:
network_id: { get_resource: private_net }
cidr: { get_param: private_net_cidr }
gateway_ip: { get_param: private_net_gateway }
dns_nameservers: { get_param: private_net_dns }
allocation_pools:
- start: { get_param: private_net_pool_start }
end: { get_param: private_net_pool_end }
router:
type: OS::Neutron::Router
properties:
external_gateway_info:
network: { get_param: public_net_id }
router_interface:
type: OS::Neutron::RouterInterface
properties:
router_id: { get_resource: router }
subnet_id: { get_resource: private_subnet }
[...]
wp_demo_instance:
type: OS::Nova::Server
depends_on: router_interface
properties:
image: { get_param: image }
flavor: { get_param: flavor }
key_name: { get_param: key_name }
networks:
- port: {get_resource: wp_demo_port}
user_data_format: RAW
user_data:
[...]
outputs:
PublicIP:
description: IP address in public network
value:
get_attr: [wp_demo_floating_ip, floating_ip_address]
PrivateIp:
description: IP address in private network
value:
get_attr: [wp_demo_instance, first_address]
WebsiteURL:
description: URL for Wordpress wiki
Variable inputs
Objects to create
Reference inputs
& dependencies
Dependency
Output parms
git clone https://github.com/OpenTelekomCloud/heat-templates
Deployment
]$ openstack stack create --template WordPress_Single_instance.yaml CloudCampStack
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 7d644f68-f685-40a9-9abf-25493cb658f1 |
| stack_name | CloudCampStack |
| description | HOT template to create single Wordpress instance on OTC Creates new router and private subnet, new security group Downloads and installs MariaDB, PHP, Apache and Wordpress |
| | |
| creation_time | 2017-06-21T22:27:16.609094 |
| updated_time | None |
| stack_status | CREATE_IN_PROGRESS |
| stack_status_reason | Stack CREATE started |
+---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
$ openstack stack show CloudCampStack
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 7d644f68-f685-40a9-9abf-25493cb658f1 |
| stack_name | CloudCampStack |
| description | HOT template to create single Wordpress instance on OTC Creates new router and private subnet, new security group Downloads and installs MariaDB, PHP, Apache and Wordpress |
| | |
| creation_time | 2017-06-21T22:27:16.609094 |
| updated_time | None |
| stack_status | CREATE_FAILED |
| stack_status_reason | Resource CREATE failed: BadRequest: resources.wp_demo_instance: Invalid key_name provided. (HTTP 400) |
| parameters | OS::project_id: a90ddf531f934a9aac8acebf1abdf19a
[...]
| outputs | - description: IP address in public network |
| | output_key: PublicIP |
| | output_value: null
[...]
Template assumes non-existing key
Output params
Daniela Ebert @ OpenStack Summit:
Ansible
ConfigManagement solution
Good REST support (2.1+)
Used in InterOp challenge in Barcelona
(16 companies showing live deployments on their cloud using the same automation)
python-shade abstracts away some of the differences b/w OpenStack clouds
YAML again
ansible-playbook -e "action=apply env=ENVIRON password=XXXXX" site.yml
Resources
Open Build Service
RPMs / DEBs (uvp-tools, otc-tools, openstackclient, ...)
OpenTelekomCloud Blog
Slides
ImageFactory
Questions &
Call to Action
Understand why you adopt Cloud
Public Cloud Architecture w/o automation
leaves most benefits on the table
Use OpenStack APIs to automate
Manage your Infrastructure as Code
Use declarative description (heat or similar)
Cheat Sheet
| Service | openstack | OTC |
|---|---|---|
| compute (VMs) | nova v2 (server) | ECS |
| block storage | cinder v2 (volume) | EVS |
| object storage | [SWIFT] | OBS (S3) |
| network |
neutron (net, router, floating IP, VPN) | VPC (EIP) |
| images | glance v2 | IMS |
| authentication |
(keystone v3) | IAM / MyWorkPl |
| LoadBalancer |
neutron LBaaSv2 | ELB / LBaaSv2 |
| Relat Database |
(trove) | RDS (1.0.1) |
| Containers | (magnum / k8s) | CCE (1.0.1) / k8s |
| BareMetal | ironic | BMS (2.1) |
| Orchestration | heat | RTS (1.1*) |