Microservice with Functional Gateway
Jonghoon Kim
- Serial entrepreneur
- Experienced rapid growth in various startups
- Experienced in full project life cycle from design to implementation to integration
Work experience
- TicketMonster - SW Engineer
- Resty - CTO
- Goodoc - CTO
- Kakao - SW Engineer
- Kakao Mobility
SW Engineer, Technical Leader
- Hyundai Motor Company
Senior Research Engineer@AI Research Lab
Motivation
Kakao T
다양한 서비스들을 하나의 앱으로 합쳐서 '슈퍼앱' 을 런칭해야 하는 프로젝트
Large
Complex
Monolithic
Taxi 1.0
중형, 대형, 모범, 블랙,
멀티콜, 시승, 서프라이즈,
업무택시, 택시 얼라이언스,
화물, 합승, 예약, 콜전용택시
카드사제휴, 광고1, 광고2
…
자율주행택시
해외진출
승객과 기사
사업구역, 행정구역,
약관, 공지사항, 배차,
미터기, 서징, 할인, 쿠폰
자동결제, 현장결제, 미수, 돈통,
정산1, 정산2, 정산3
…
통계
Andorid 승객앱
iOS 승객앱
기사앱
블랙기사앱
시승기사앱
SDK
…
Web
톡연동
API
Admin
Partner Admin
Web View
...
Monolithic Hell
-
복잡도 증가, 한명의 개발자가 이해할 수 있는 수준을 넘어감
- 한부분의 문제가 전체에 영향을 줌
- 스케일링 이슈 - 일반적인 방법은 로드밸런서 뒤에 여러카피를 두는 것, 그러나 각 컴포넌트별로 요구사항이 다름(IO, Memory, CPU...)
- 각 문제별로 필요한 기술스택이 다름
- 새로운 멤버의 부담감
- 코드 변경의 두려움
- 개발속도 둔화
- 새로운 시도의 두려움
- 일을 나눠서 하기 어려움
New Architecture
-
서비스간 시너지 창출
-
개별 기능이 조합 가능해야 함
-
개별 기능이 조합 가능해야 함
-
외부 서비스 포함하여 확장이 쉬워야 함
-
기능간 의존성이 없어야 함
-
기능간 의존성이 없어야 함
-
개발자를 쉽게 충원하고 늘릴수 있어야 함
- 기술셋을 자유롭게 선택해야 함
-
기존 도메인 지식을 몰라도 쉽게 적응할 수 있어야 함
-
클라 통합만이 목표는 아님
- 기존의 많은 문제점을 해결해야 함
Microservice
- 기능별로 분리된 작은 서비스들로 구성, 작고 한 가지 일을 잘하는데 주력
- 서비스 목적에 맞춰 효율적인 언어와 플랫폼 선택
- 표준 인터페이스로 해당 기능을 외부로 제공
- 독립적으로 배포와 확장이 가능
Api Gateway
- 각 마이크로 서비스들을 조합해서 클라가 필요한 API를 제공
- 다양한 서비스를 조합해야하므로 비동기 처리가 필수
- 높은 확장성, 고가용성을 위한 클라이언트 필요
(service discovery, circuit breaker, load balancer, retry…)
- 다수의 비동기 API 호출 합성을 위한 적절한 추상화가 필요했음
Candidate
- ruby + Sinatra
- ruby + celluloid
- ruby + concurrent-ruby
- jruby + Sinatra
- jruby + Finagle
- Scala + Finagle
- Scala + Finch
- Scala + Finatra
- Go + Gokit
- Elixir
- Closure + Finagle Wrapper
- Scala + Akka
- java + Netflix Stack
- C#
- python
Factor
-
Performance
-
Dynamic vs Static
-
Ecosystem
-
Production ready
- Learning curve
Marius Eriksen
Monolithic System
- Limited to a single language & runtime
- Can only support few developers
- Deploys get more dangerous
Distribute!!!
- Each service written in appropriate language
- Can only support many developers
- Granular deploys
But...
Challenges
- Dozens of network services (different language, different protocol)
- Highly concurrent
- Unreliable network, machines
- Partial failures
- Failures can cascade
Building a Functional Gateway
A little scala
Scala
- Hybrid FP/OO language
- Expressive
- Statically typed
- Runs in the JVM
- Interoperates with Java
Values (Immutable)
val i: Int = 1234
val foo: String = "bar"
Type inference
val i = 1234
val foo = "bar"
Functions are values
val addOne: Int => Int = { x: Int =>
x + 1
}
addOne(1) == 2
Pure OO
Every value is an object
val i = 123
i.toString == "123"
i.<(333) == true
i < 333 == true
Collections
Array(1, 2, 3, 4)
List(1, 2, 3, 4)
Set(1, 1, 2)
Tuple
val hostPort = ("localhost", 80)
hostPort._1 == "localhost"
hostPort._2 == 80
1 -> 2 == (1,2)
Maps
Map(1 -> 2)
Map(("foo", "bar"))
Map(1 -> Map("foo" -> "bar"))
Option
trait Option[T] {
def isDefined: Boolean
def get: T
def getOrElse(t: T): T
}
val numbers = Map("one" -> 1, "two" -> 2)
numbers.get("two") == Some(2)
numbers.get("three") == None
Option is a container that may or may not hold something.
Pattern matching
hostPort match {
case ("localhost", port) => ...
case (host, 80) => ...
}
val result = res1 match {
case Some(n) => n * 2
case None => 0
}
Map
val numbers = List(1, 2, 3, 4)
numbers.map {(n: Int) => n * 2}
numbers.map(_ * 2)
Filter
numbers.filter(_ % 2 == 0)
val isEven: Int => Boolean = {
n: Int => n % 2 == 0
}
numbers.filter(isEven)
Flatten
List(List(1, 2), List(3, 4)).flatten
=> List(1, 2, 3, 4)
List(Some(1), Some(2), None, Some(4)).flatten
=> List(1, 2, 4)
Flatmap
val nestedNumbers = List(List(1, 2), List(3, 4))
nestedNumbers.flatMap(x => x.map(_ * 2))
=> List(2, 4, 6, 8)
nestedNumbers.map((x: List[Int]) => x.map(_ * 2)).flatten
For comprehension
val numbers = List("123", "456", "789")
for {
str <- numbers
num <- str
} yield num
=> List(1, 2, 3, 4, 5, 6, 7, 8, 9)
Function compoistion
def addOne(x: Int) = x + 1
def addTwo(x: Int) = x + 2
val addThree = addOne _ compose addTwo _
// f compose g => f(g(x))
val addThree2 = addOne _ andThen addTwo _
// f andThen g => g(f(x))
Marius Eriksen
Building blocks
- Futures
- Services
- Filters
Futures
- A Future is a placeholder for a result that may not yet exist
- Futures are how we represent concurrent execution
-
Any sort of asynchronous operation
- Long computation
- Network call
- Disk I/O
- Operations can fail
- Div by zero
- Connection failure
- Timeout
Futures
- Future are a kind of container
-
A Future[T] occupy one of three states:
- Empty (pending)
- Succeeded (with a result of type T)
- Failed (with a Throwable)
trait Future[A]
val f: Future[Int]
// an integer valued future
Futures as containers
- Only one element container
- You can transform them
Future[T].map[U](f: T => U): Future[U]
// Converts a Future[T] to a Future[U] by applying f
Future[T].filter(p: T => Boolean): Future[T]
// Convert a successful Future[T] to a failed Future[T]
// by applying predicate
Use callback
val f: Future[Int] = ???
Await.result(f) // directly query
// you use callback functions instead
f onSuccess { res =>
println("The result is " + res)
} onFailure { exc =>
println("f failed with " + exc)
}
Promises
- Futures are read-only
- A Promise is a writable future
val p: Promise[Int]
// Success
p.setValue(1)
// Failure
p.setException(new Exception)
Thumbnail extractor
trait Webpage {
def imageLinks: Seq[String]
def links: Seq[String]
}
def fetch(url: String): Future[Webpage]
def getThumbnail(url: String): Future[Webpage] = ???
Thumbnail extractor
def getThumbnail(url: String): Future[Webpage] = {
val promise = new Promise[Webpage]
fetch(url) onSuccess { page =>
fetch(page.imageLinks(0)) onSuccess { p =>
promise.setValue(p)
} onFailure { exc =>
promise.setException(exc)
}
} onFailure { exc =>
promise.setException(exc)
}
promise
}
Callback-hell...
Sequential composition
- Fetching the webpage
- Parsing that page to find the first image link
- Fetching the image link
Use flatmap
def flatMap[B](f: A => Future[B]): Future[B]
- The most important Future combinator
- flatMap sequences two futures
- It takes a Future and an asynchronous function and returns another Future
- If either Future fails, the given Future will also fail
Use flatmap
def getThumbnail(url: String): Future[Webpage] =
fetch(url) flatMap { page =>
fetch(page.imageLinks(0))
}
Concurrent composition
- Future provides some concurrent combinators
- Convert a sequence of Future into a Future of sequence
object Future {
def collect[A](fs: Seq[Future[A]]): Future[Seq[A]]
def join(fs: Seq[Future[_]]): Future[Unit]
def select(fs: Seq[Future[A]]): Future[(Try[A], Seq[Future[A]])]
}
Concurrent composition
def getThumbnails(url: String): Future[Seq[Webpage]] =
fetch(url) flatMap { page =>
Future.collect(
page.imageLinks map { u => fetch(u) }
)
}
Useful for fan-out operations
Recovering from failure
- Futures must be composable and recoverable
- flatMap operates over values
- rescue operates over exceptions
def rescue[B](f: Exception => Future[B]): Future[B]
// Recovering errors...
val f = fetch(url) rescue {
case ConnectionFailed =>
fetch(url)
}
Services
- A service is a kind of asynchronous function
- ex. RPC
- Dispatch a request
- Wait a while
- Succeeds or fails
- It's a Function!
trait Service[Req, Rep] extends (Req => Future[Rep])
val http: Service[HttpReq, HttpRep]
val redis: Service[RedisCmd, RedisRep]
val thrift: Service[TFrame, TFrame]
Services are symmetric
Servers implement these, clients make use of them
// A simple service
// server
val multiplier = { i =>
Future.value(i*2)
}
// client
multiplier(123) onSuccess { res =>
println("result", r)
}
Services are symmetric
// server
Http.serve(":8080", new Service[HttpReq, HttpRep] {
def apply(req: HttpReq): Future[HttpRep] = ???
})
// client
val http = Http.newService("kakao.com:80")
// proxy
Http.serve(":8080", Http.newService("kakao.com:80"))
Filters
- Services : Logical endpoint
- Filters : Service agnostic behavior
- Timeouts
- Retries
- Statistics
- Authentication
- Logging
- Filters compose over services
- It's a Function!
trait Filter[ReqIn, ReqOut, RepIn, RepOut] extends
((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])
Timeout filter
val timeout = { (req, service) =>
service(req).within(1.second)
}
class TimeoutFilter[Req, Rep](to: Duration)
extends Filter[Req, Rep, Req, Rep] {
def apply(req: Req, service: Service[Req, Rep]) =
service(req).within(to)
}
Filters are stackable
val timeout: Filter[…]
val auth: Filter[…]
val authAndTimeout = auth andThen timeout
val service: Service[…]
val authAndTimeoutService =
authAndTimeout andThen service
Finagle
Asynchronous
Non-Blocking
Protocol Agnostic
Fullstack RPC
Adopters
- Foursquare
- ING Bank
- Soundcloud
- Tumblr
Servers
- Concurrency Limit
- Request Timeout
- Metrics and Tracing
- HTTP admin interface
Concurrency Limit
import com.twitter.finagle.Http
val server = Http.server
.withAdmissionControl.concurrencyLimit(
maxConcurrentRequests = 10,
maxWaiters = 0
)
Request Timeout
import com.twitter.conversions.time._
import com.twitter.finagle.Http
val server = Http.server
.withRequestTimeout(42.seconds)
Metrics
// define your stats
val counter = statsReceiver.counter("requests_counter")
// update the value
counter.incr()
// The value of this counter will be exported
// by the HTTP server and accessible at /admin/metrics.json
Tracing
- Zipkin is a distributed tracing system
- It helps gather timing data needed to troubleshoot latency problems in microservice architectures
- Zipkin’s design is based on the Google Dapper paper.
Zipkin
Clients
- Retries
- Naming / Service Discovery
- Timeouts and Expirations
- Load Balancing
- Rate Limiting
- Connection Pooling
- Circuit Breaking
- Failure Detection
- Metrics and Tracing
- Interrupts
- Context Propagation
- ...
Retry Budget
import com.twitter.finagle.Http
import com.twitter.conversions.time._
import com.twitter.finagle.service.RetryBudget
val budget = RetryBudget(
ttl = 10.seconds,
minRetriesPerSec = 5,
percentCanRetry = 0.1
)
val client = Http.client.withRetryBudget(budget)
// Allow retrying 10% of total requests on top of 5 retries per sec
Retry Backoff
import com.twitter.finagle.Http
import com.twitter.conversions.time._
import com.twitter.finagle.service.Backoff
val client = Http.client
.withRetryBackoff(Backoff.exponentialJittered(2.seconds, 32.seconds))
// Backoff for rnd(2s), rnd(4s), rnd(8), ..., rnd(32s), ... rnd(32s)
Timeout
import com.twitter.conversions.time._
import com.twitter.finagle.Http
val client = Http.client
.withTransport.connectTimeout(1.second) // TCP connect
.withSession.acquisitionTimeout(42.seconds)
.withSession.maxLifeTime(20.seconds) // connection max life time
.withSession.maxIdleTime(10.seconds) // connection max idle time
Load Balancing
- client treats its server set as replica set
- Load balancing involves load distributor and load factor
-
Currently available
- Load distributors: heap, p2c, aperture, round robin
- Load factors: least loaded, peak EWMA
LB via Aperture
import com.twitter.conversions.time._
import com.twitter.finagle.Http
import com.twitter.finagle.loadbalancer.Balancers
val balancer = Balancers.aperture(
lowLoad = 1.0, highLoad = 2.0, // the load band adjusting an aperture
minAperture = 10 // min aperture size
)
val client = Http.client.withLoadBalancer(balancer)
Circuit Breaking
- To preemptively disable sessions that will likely fail requests
- Placed under LB to exclude particular nodes from its replica set
-
Policy
- FailFast
- FailureAccrual
- ThresholdFailureDetector
Fail Accrual
import com.twitter.conversions.time._
import com.twitter.finagle.Http
import com.twitter.finagle.service.Backoff
import com.twitter.finagle.service.FailureAccrualFactory.Param
import com.twitter.finagle.service.exp.FailureAccrualPolicy
val twitter = Http.client
.configured(Param(() => FailureAccrualPolicy.successRate(
requiredSuccessRate = 0.95,
window = 100,
markDeadFor = Backoff.const(10.seconds)
)))
// Mark a replica dead if SuccessRate is below 95%
// over the most recent 100 requests and then backoff for 10 sec.
Finatra
- Used in production at Twitter (and many other org)
- Actively developed and maintained
- Fast, testable, Scala services built on TwitterServer and Finagle
- Fully async using Futures
- Powerful JSON support
Q & A
temp
By JongHoon Kim
temp
- 81