18 Jan 2017

Product Manager, 이해승

, 서버 운영의 변화

Virtual Machine, Hypervisor, ... more deeply

Docker

Docker Lifecycle

docker create
docker rename
docker run
docker rm
docker update

Cheat Sheet

https://github.com/wsargent/docker-cheat-sheet

이런 내용은 다들

아실테니까 .....

오늘 진짜로 이야기하려는 내용은,

Deployment & Server Operation
Infrastructure as a Service
Platform as a Service
Containerized Application Virtualization
Re-definition of Deployment by Docker

1. Deployment & Server Operation

State

상태

사물, 현상이 놓여 있는 모양이나 형편

stateful/stateless - protocol

Authorization

Add User

Env Vars

Firewall

Dependencies

Network

Proxy Setting

Run App!!

Bundle Install

Configuration

Install GO or Python

Git Clone

Deployment Process

Authorization

Add User

Env Vars

Firewall

Dependencies

Network

Proxy Setting

Run App!!

Bundle Install

Configuration

Install GO or Python

Git Clone

State Change during Deployment

Practically,

numerous Applications

on server

New Application - D

Deployment Process - Normalization

therefore,

Deployment means making all applications
to Running or Runnable State

Server Operation

... means "maintaining the 'state' that all applications are running or runnable.

Why Server Operation is Hard?

maintaining

global

state

management

Why Server Operation is Hard?

2. Infra as a Service

Instance

Virtualized Resources with same environment

Infrastructure as a Service?

Instance

Virtualized Resources with same environment
Usable Computing Resource which is wanted/managed.

Infrastructure as a Service?

Disposable Component

Server = physical resources
Instance = disposable resources

Important Concept

Image, or Imagination

App: A/B/D

App: C

App: A/B/D

App: C

App: A/B/D

App: C

Instance

Imagination & Scale In/Out

Scale Out

App: A/B/D

App: C

App: A/B/D

App: C

Instance

Auto Scaling

Resource Monitoring & Auto Scaling

App: C

Instance

Image Management

AMI

이미지 수의 증가: 수에 비례해서 관리가 어려워짐 / 애플리케이션 수 x 시간

AMI

...

Image Management

이미지 수의 증가: 수에 비례해서 관리가 어려워짐 / 애플리케이션 수 x 시간
상태에 기반한 이미지: 특정 상태의 '상태'를 저장한 것일 뿐

Instance

Image Management

이미지 수의 증가: 수에 비례해서 관리가 어려워짐 / 애플리케이션 수 x 시간
상태에 기반한 이미지: 특정 상태의 '상태'를 저장한 것일 뿐
최종적으로 남은 이미지: 어떻게 만들었는지 아무도 모름 --> 복원 불가능

Image Management

이미지 수의 증가: 수에 비례해서 관리가 어려워짐 / 애플리케이션 수 x 시간
상태에 기반한 이미지: 특정 상태의 '상태'를 저장한 것일 뿐
최종적으로 남은 이미지: 어떻게 만들었는지 아무도 모름 --> 복원 불가능

==> 단순화된 상태 관리!

==> 어떻게 하면 상태를 더 잘 관리할 수 있을까?

==> 이미지/인스턴스 활용의 극대화

Image Management, How To?

Configuration Management

Old-Fashioned

명령어를 통한 상태 변화

CMD

Install Nginx

Setup Firewall

Install Python

Cookbook을 통한 상태 변화(CHEF)

BaaS service based on CHEF(KTHitel)

Cookbook1

Cookbook2

Install Nginx

Setup Firewall

Install Python

Idempotence

Cookbook1

Cookbook2

Install Nginx

Setup Firewall

Install Python

Cookbook1

Cookbook2

Install Nginx

Setup Firewall

Install Python

Infrastructure as a Code(IaC)

Cookbook

Install Nginx

Setup Firewall

Install Python

Apache Cookbook example

CMD

$ apt -get install apache2

Chef Cookbook

package 'httpd'

service 'httpd' do

action [ :start, :enable ]

end

Effects of Configuration Management

초기 적용이 어려움

학습비용
많은 시행착오
작업 속도가 느림

서버 운영을 단순화

서버의 상태를 코드로 관리
어떤 상태를 재현 가능하게 만들어 줌
다수의 서버를 운용 중에 특정 상태로 변화시킬 수 있음

Effects of Configuration Management

초기 적용이 어려움

학습비용
많은 시행착오
작업 속도가 느림

서버 운영을 단순화

서버의 상태를 코드로 관리
어떤 상태를 재현 가능하게 만들어 줌
다수의 서버를 운용 중에 특정 상태로 변화시킬 수 있음

"상태"를 관리하는 도구!

3. Platform as a Service

Platform as a Service

Elements of Deployment(past)

Application

Server

(ver x.xxx)

Application

(ver x.xxx)

Elements of Deployment(PaaS)

Application 배포 단위

배포 환경은 Heroku가 제공

Push & Build

(ver x.xxx)

Deployment

애플리케이션과 배포 단위의 1:1 매칭

애플리케이션별 독립된 배포 환경 제공

1:1

(ver x.xxx)

1:1

(ver x.xxx)

Scale-In/Out 기본 지원

애플리케이션 업데이트

- conventional

Update App

xx1

xx2

xx3

애플리케이션 업데이트

- Rollback

Update

xx1

xx2

xx3

xx1

Rollback

애플리케이션 업데이트

- on Heroku

Push & Build

(ver x.xx1)

xx1

Push & Build

xx2

Push & Build

xx3

(ver x.xx2)

(ver x.xx3)

애플리케이션 업데이트

- Rollback on Heroku

배포의 단순화

애플리케이션 개발

환경구축

배포

Instance

Build

확장의 단순화

애플리케이션 개발

환경구축

배포

Instance

이미지화

확장

그럼에도 PaaS를

사용하지 않는 이유?

비용

PaaS에 적합하지 않은 애플리케이션

Build Error!

(ver x.xxx)

PaaS != IaaS

IaaS에 비해서 큰 학습 비용
원격 접속 시스템이 없거나 제한
파일 시스템 이용에 제한
Site 패키지 설치 제한
로그 수집 제한적 허용

4. Containerized Application Virtualization

PaaS의 단점?

IaaS에 비해서 큰 학습 비용
원격 접속 시스템이 없거나 제한
파일 시스템 이용에 제한
Site 패키지 설치 제한
로그 수집 제한적 허용

상태 변화에 의존한 작업

State X

State X+1

Command

Collect Logs

Modify Files

Install Library

PaaS는?

Immutable(Stateless)

Command

Collect Logs

Modify Files

Install Library

State X

State X+1

PaaS는?

Immutable(Stateless)

Command

Collect Logs

Modify Files

Install Library

State X

State X+1

==> 컨테이너형 가상화의 특징!!!

Immutable(Stateless)

Important Concept

PaaS는?

Immutable(Stateless)

Command

Collect Logs

Modify Files

Install Library

State X

State X+1

==> 컨테이너형 가상화의 특징!!!

Heroku Dyno

= Linux Container(LXC),

Chroot on Steroid

Chroot

Pivot root 기능
파일, 라이브러리는 직접 준비
사용이 까다로움
프로세스 격리의 초기버전 정도?

Linux Container(LXC)

Kernel Namespace
Apparmor and SELinux profiles
Seccomp policies
Chroots(using pivot_root)
Kernel capabilities
Control groups(cgroups)

Linux Container(LXC)

Host Kernel 공유

Linux Container(LXC)

프로세스 실행 환경 분리

Host

Container

unique Namespace
unique Root directory
unique network env resource

unique Namespace
unique Root directory
unique network env resource

unique Namespace
unique Root directory
unique network env resource

unique Namespace
unique Root directory
unique network env resource

Process Isolation

격리된 환경에서 특정 프로세스만 실행

Linux Container

격리된환경, 즉 내부적인 의미의 컨테이너에 집중

Docker

LXC

AuFS

Docker Index

프로세스 격리

계층화된 저장 장치

이미지 공유

Docker

LXC

AuFS

Docker Index

프로세스 격리

계층화된 저장 장치

이미지 공유

.˙. Linux Container의 확장 툴셋

Heroku Dynos

≈

Heroku Dynos

≈

Almost same? Docker는 LXC의 Wrapper?

Docker의 구조

현재의 Docker(latest 1.13)

LXC

AuFS

프로세스 격리

계층화된 저장 장치

이미지 공유

Libcontainer

DeviceMapper

BTRFS

Docker Hub

Registry

Private

3rd Party

Repository

Boot2Docker

Docker 지원 시스템

Ubuntu, Red Hat Enterprise Linux, Oracle Linux, CentOS, Debian, Gentoo, Google Cloud Platform, Rackspace Cloud(Carina), Amazon EC2, IBM Softlayer, Microsoft Azure, Arch Linux, Fedora, OpenSUSE, CRUX linux, Microsoft Windows/Windows server, Mac OS X

Docker만의 차별성

이미지(Image) 기반의 애플리케이션 가상화

이미지: 특정 애플리케이션이 실행 가능한 고유 환경

컨테이너: 특정 이미지 기반으로 실행되는 격리된 프로세스

Build Image

Read Only & Writable Layer

rootfs / Base Image

상태 변화

Git 설치하기

Git 설치 후 상태 변화

Diff를 통해서 Base Image와 컨테이너의 차이를 파악

Commit으로 새로운 이미지 생성

새로운 상태를 이미지로 저장

Dockerfile

이미지생성과정을기술한Docker 전용 DSL

Dockerfile을통한이미지빌드

빌드된 이미지

호스트 환경과 무관하게 실행 가능!

빌드된 이미지

호스트 환경과 무관하게 실행 가능!

==> 이미 애플리케이션 실행 가능 상태? 배포 완료?

Docker를 활용한 오픈소스 배포 문화

StriderCD - Node.js web application

IHaskell - Python, Haskell, Zeromq 기반 utility/Web UI

Docker Hub

Docker Priavate Registry

Build once, Run anywhere

표준화된 이미지

이미지 생성/공유 기능

공식 Registry 서비스 지원

Private Registry 애플리케이션

Container vs. Docker

: 표준화된 컨테이너의 이동성에 집중

LXC vs. Docker

: LXC == 프로세스 격리를 위한 도구

: Docker == 컨테이너 수송을 위한 도구

Docker의 핵심

IaaS의 자유

PaaS의 단순함

--> Container on CDN?

5. Re-Definition of Deployment by Docker

배포단위

Docker Images

애플리케이션 실행 환경

애플리케이션

배포 대상

Docker Servers

Remember

Conventional Deployment & Operation

배포와 서버 운영의 새로운 정의

Development

push

CI Server

Docker Servers

Docker Regitry

애플리케이션 업데이트

Development

push

CI Server

Docker Regitry

애플리케이션 롤백

Docker Regitry

CoreOS

클러스터링 가능한 도커 전용 OS

물류 시스템?

Docker로 모든 애플리케이션을 컨테이너로 만든다면, 서버는 Docker를 돌리기 위한 존재일뿐 --> CoreOS concept

CDN 3.0?

... easier to manage is to use Openshift 3.0
as the infrastructure of the CDN.

REF: https://www.linkedin.com/pulse/new-cdn-architecture-leveraging-docker-containers-glenn-west

Feature Toggle - launch darkly

Support for diverse kinds of workloads such as big data, cloud native apps, etc.	Types of Workloads	Cloud Native Application
"Application Group" models dependencies as a tree of groups. Components are started in dependency order. Colocation of group's containers on same Mesos slave is not supported. A Pod abstraction is on roadmap, but not yet available.	Application Definition	A combination of Pods, Replication Controllers, Replica Sets, Services and Deployments. As explained in the overview above, a pod is a group of co-located containers; the atomic unit of deployment. Pods do not express dependencies among individual containers within them. Containers in a single Pod are guaranteed to run on a single Kubernetes node.
Possible to scale an individual group, its dependents in the tree are also scaled.	Application Scalability constructs	Each application tier is defined as a pod and can be scaled when managed by a Deployment or Replication Controller. The scaling can be manual or automated.
Applications are distributed among Slave Nodes.	High Availability	Pods are distributed among Worker Nodes. Services also HA by detecting unhealthy pods and removing them.
Application can be reached via Mesos-DNS, which can act as a rudimentary load balancer.	Load Balancing	Pods are exposed through a Service, which can be a load balancer.
Load-sensitive autoscaling available as a proof-of-concept application. Rate-sensitive autoscaling available for Mesosphere’s enterprise customers. Rich metric-based scaling policy.	Auto-Scaling for the Application	Auto-scaling using a simple number-of-pods target is defined declaratively with the API exposed by Replication Controllers. CPU-utilization-per-pod target is available as of v1.1 in the Scale sub resource. Other targets are on the roadmap.

"Rolling restarts" model uses application-defined minimumHealthCapacity (ratio of nodes serving new/old application) "Health check" hooks consume a "health" API provided by the application itself	Rolling Application Upgrades and Rollback	"Deployment" model supports strategies, but one similar to Mesos is planned for the future Health checks test for liveness i.e. is app responsive
Logging: Can use ELK Monitoring: Can use external tools	Logging and Monitoring	Health checks of two kinds: liveness (is app responsive) and readiness (is app responsive, but busy preparing and not yet able to serve) Logging: Container logs shipped to Elasticsearch/Kibana (ELK) service deployed in cluster Resource usage monitoring: Heapster/Grafana/Influx service deployed in cluster Logging/monitoring add-ons are part of official project Sysdig Cloud integration
A Marathon container can use persistent volumes, but such volumes are local to the node where they are created, so the container must always run on that node. An experimental flocker integration supports persistent volumes that are not local to one node.	Storage	Two storage APIs: The first provides abstractions for individual storage backends (e.g. NFS, AWS EBS, ceph,flocker). The second provides an abstraction for a storage resource request (e.g. 8 Gb), which can be fulfilled with different storage backends. Modifying the storage resource used by the Docker daemon on a cluster node requires temporarily removing the node from the cluster

Marathon's docker integration facilitates mapping container ports to host ports, which are a limited resource. A container does not get its own IP by default, but it can if Mesos is integrated with Calico. Even so, multiple containers cannot share a network namespace (i.e. cannot talk to one another on localhost).	Networking	The networking model lets any pod can communicate with other pods and with any service. The model requires two networks (one for pods, the other for services) Neither network is assumed to be (or needs to be) reachable from outside the cluster. The most common way of meeting this requirement is to deploy an overlay network on the cluster nodes.
Containers can discover services using DNS or reverse proxy.	Service Discovery	Pods discover services using intra-cluster DNS
Mesos has been simulated to scale to 50,000 nodes, although it is not clear how far scale has been pushed in production environments. Mesos can run LXC or Docker containers directly from the Marathon framework or it can fire up Kubernetes or Docker Swarm (the Docker-branded container manager) and let them do it.	Performance and Scalability	With the release of 1.2, Kubernetes now supports 1000-node clusters. Kubernetes scalability is benchmarked against the following Service Level Objectives (SLOs): API responsiveness: 99% of all API calls return in less than 1s Pod startup time: 99% of pods and their containers (with pre-pulled images) start within 5s.

REF: https://platform9.com/blog/compare-kubernetes-vs-mesos/

Mesos+Marathon is slightly better and larger than kubernetes.
Mesos can work with multiple frameworks and Kubernetes is one of them.
Kubernetes on Mesos is working!

K8s-Mesos architecture

Disposable Component

Immutable Infrastructure

Important Concepts

갑자기, CDN topology

Scattered CDN

Consolidated CDN

갑자기, CDN topology

Scattered CDN

Consolidated CDN

physical proximity minimizes latency
effective in low-connectivity regions
smaller POPs are easier to deploy

Higher maintenance costs
RTT prolonged by multiple connection points
Cumbersome to deploy new configurations

High-capacity servers are better for DDoS mitigation
Enables agile configuration deployment
Lower maintenance costs

Less effective in low-connectivity regions
High-capacity POPs harder to deploy

갑자기, CDN topology

Scattered CDN

Consolidated CDN

physical proximity minimizes latency
effective in low-connectivity regions
smaller POPs are easier to deploy

Higher maintenance costs
RTT prolonged by multiple connection points
Cumbersome to deploy new configurations

High-capacity servers are better for DDoS mitigation
Enables agile configuration deployment
Lower maintenance costs

Less effective in low-connectivity regions
High-capacity POPs harder to deploy

Akamai, CDNW

CloudFlare, LimeLight, and

AWS, GCP, Azure, AliYun

CDN strategy?

Scattered CDN

Consolidated CDN

CDNW CS

CDNW NGP

vs.?

CS와 NGP가 계속 따로 운영될 수 밖에 없는 구조.

통합은 어렵다. 새로 만드는게 더 빠르다 --> PJ WE?

일부라도 좀 더 고쳐 보자. MSA로 변경해 보자 --> 통합은 여전히?

더 큰 그림(architecture)가 필요하다. 하지만 누가?

컨테이너화의 대상은? docker는 기술일 뿐? 우리에게 맞추려면?

CAPEX & OPEX를 줄이기 위해서 SD(Software-Defined)를 적극적으로 적용?

Docker Lifecycle

Cheat Sheet

1. Deployment & Server Operation

Server Operation

Why Server Operation is Hard?

Why Server Operation is Hard?

Why Server Operation is Hard?

2. Infra as a Service

Infrastructure as a Service?

Infrastructure as a Service?

Disposable Component

Important Concept

Image, or Imagination

Imagination & Scale In/Out

Auto Scaling

Image Management

Image Management

Image Management

Image Management

Image Management, How To?

Configuration Management

Old-Fashioned

Cookbook을 통한 상태 변화(CHEF)

Idempotence

Infrastructure as a Code(IaC)

Apache Cookbook example

Effects of Configuration Management

Effects of Configuration Management

"상태"를 관리하는 도구!

3. Platform as a Service

Platform as a Service

Elements of Deployment(past)

Elements of Deployment(PaaS)

그럼에도 PaaS를

사용하지 않는 이유?

PaaS != IaaS

4. Containerized Application Virtualization

PaaS의 단점?

Immutable(Stateless)

Important Concept

표준화된 이미지

Docker의 핵심

5. Re-Definition of Deployment by Docker

CoreOS

CDN 3.0?

Disposable Component

Immutable Infrastructure

Important Concepts

CDNW CS

CDNW NGP

Questions?