Monolithic System

Limited to a single language & runtime
Can only support few developers
Deploys get more dangerous

Distribute!!!

Each service written in appropriate language
Can only support many developers
Granular deploys

But...

Challenges

Dozens of network services (different language, different protocol)
Highly concurrent
Unreliable network, machines
Partial failures
Failures can cascade

Building a Functional Gateway

A little scala

Scala

Hybrid FP/OO language
Expressive
Statically typed
Runs in the JVM
Interoperates with Java

Values (Immutable)


    val i: Int = 1234

    val foo: String = "bar"

Type inference


    val i = 1234
    
    val foo = "bar"

Functions are values


    val addOne: Int => Int = { x: Int =>
        x + 1
    }

    addOne(1) == 2

Pure OO

Every value is an object


    val i = 123

    i.toString == "123"

    i.<(333) == true
    
    i < 333 == true

Collections


    Array(1, 2, 3, 4)

    List(1, 2, 3, 4)
    
    Set(1, 1, 2)

Tuple


    val hostPort = ("localhost", 80)

    hostPort._1 == "localhost"

    hostPort._2 == 80

    1 -> 2 == (1,2)

Maps


    Map(1 -> 2)
    
    Map(("foo", "bar"))

    Map(1 -> Map("foo" -> "bar"))

Option


    trait Option[T] {
      def isDefined: Boolean
      def get: T
      def getOrElse(t: T): T
    }
    
    val numbers = Map("one" -> 1, "two" -> 2)
    
    numbers.get("two") == Some(2)
    
    numbers.get("three") == None

Option is a container that may or may not hold something.

Pattern matching


    hostPort match {
      case ("localhost", port) => ...
      case (host, 80) => ...
    }
    
    val result = res1 match {
      case Some(n) => n * 2
      case None => 0
    }

Map

    
    val numbers = List(1, 2, 3, 4)

    numbers.map {(n: Int) => n * 2}

    numbers.map(_ * 2)

Filter


    numbers.filter(_ % 2 == 0)

    val isEven: Int => Boolean = {
      n: Int => n % 2 == 0
    }

    numbers.filter(isEven)

Flatten


    List(List(1, 2), List(3, 4)).flatten

      => List(1, 2, 3, 4)

    List(Some(1), Some(2), None, Some(4)).flatten

      => List(1, 2, 4)

Flatmap


  val nestedNumbers = List(List(1, 2), List(3, 4))

  nestedNumbers.flatMap(x => x.map(_ * 2))

    => List(2, 4, 6, 8)

  nestedNumbers.map((x: List[Int]) => x.map(_ * 2)).flatten

For comprehension


  val numbers = List("123", "456", "789")

  for {
    str <- numbers
    num <- str
  } yield num

    => List(1, 2, 3, 4, 5, 6, 7, 8, 9)

Function compoistion


  def addOne(x: Int) = x + 1

  def addTwo(x: Int) = x + 2

  val addThree = addOne _ compose addTwo _
  // f compose g => f(g(x))

  val addThree2 = addOne _ andThen addTwo _
  // f andThen g => g(f(x))

Your Server as a Function

Marius Eriksen

Building blocks

Futures
Services
Filters

Futures

A Future is a placeholder for a result that may not yet exist
Futures are how we represent concurrent execution
Any sort of asynchronous operation
- Long computation
- Network call
- Disk I/O
Operations can fail
- Div by zero
- Connection failure
- Timeout

Futures

Future are a kind of container
A Future[T] occupy one of three states:
- Empty (pending)
- Succeeded (with a result of type T)
- Failed (with a Throwable)


  trait Future[A]

  val f: Future[Int]
  // an integer valued future

Futures as containers

Only one element container
You can transform them


  Future[T].map[U](f: T => U): Future[U]
  // Converts a Future[T] to a Future[U] by applying f

  Future[T].filter(p: T => Boolean): Future[T]
  // Convert a successful Future[T] to a failed Future[T]
  // by applying predicate

Use callback


  val f: Future[Int] = ???

  Await.result(f) // directly query

  // you use callback functions instead
  f onSuccess { res =>
    println("The result is " + res)
  } onFailure { exc =>
    println("f failed with " + exc)
  }

Promises

Futures are read-only
A Promise is a writable future


    val p: Promise[Int]
    
    // Success
    p.setValue(1)

    // Failure
    p.setException(new Exception)

Thumbnail extractor


  trait Webpage {
    def imageLinks: Seq[String]
    def links: Seq[String]
  }

  def fetch(url: String): Future[Webpage]

  def getThumbnail(url: String): Future[Webpage] = ???

Thumbnail extractor


  def getThumbnail(url: String): Future[Webpage] = {
    val promise = new Promise[Webpage]
    fetch(url) onSuccess { page =>
      fetch(page.imageLinks(0)) onSuccess { p =>
        promise.setValue(p)
      } onFailure { exc =>
        promise.setException(exc)
      }
    } onFailure { exc =>
      promise.setException(exc)
    }
    promise
  }

Callback-hell...

Sequential composition

Fetching the webpage
Parsing that page to find the first image link
Fetching the image link

Use flatmap


  def flatMap[B](f: A => Future[B]): Future[B]

The most important Future combinator
flatMap sequences two futures
It takes a Future and an asynchronous function and returns another Future
If either Future fails, the given Future will also fail

Use flatmap


  def getThumbnail(url: String): Future[Webpage] =
    fetch(url) flatMap { page =>
      fetch(page.imageLinks(0))
    }

Concurrent composition

Future provides some concurrent combinators
Convert a sequence of Future into a Future of sequence


  object Future {
    def collect[A](fs: Seq[Future[A]]): Future[Seq[A]]
    def join(fs: Seq[Future[_]]): Future[Unit]
    def select(fs: Seq[Future[A]]): Future[(Try[A], Seq[Future[A]])]
  }

Concurrent composition


  def getThumbnails(url: String): Future[Seq[Webpage]] =
    fetch(url) flatMap { page =>
      Future.collect(
        page.imageLinks map { u => fetch(u) }
      )
    }

Useful for fan-out operations

Recovering from failure

Futures must be composable and recoverable
flatMap operates over values
rescue operates over exceptions


  def rescue[B](f: Exception => Future[B]): Future[B]


  // Recovering errors...
  val f = fetch(url) rescue {
    case ConnectionFailed =>
      fetch(url)
  }

Services

A service is a kind of asynchronous function
ex. RPC
- Dispatch a request
- Wait a while
- Succeeds or fails
It's a Function!


  trait Service[Req, Rep] extends (Req => Future[Rep])

  val http: Service[HttpReq, HttpRep]
  val redis: Service[RedisCmd, RedisRep]
  val thrift: Service[TFrame, TFrame]

Services are symmetric

Servers implement these, clients make use of them


  // A simple service

  // server
  val multiplier = { i =>
    Future.value(i*2)
  }

  // client
  multiplier(123) onSuccess { res =>
    println("result", r)
  }

Services are symmetric


  // server
  Http.serve(":8080", new Service[HttpReq, HttpRep] {
    def apply(req: HttpReq): Future[HttpRep] = ???
  })

  // client
  val http = Http.newService("kakao.com:80")

  // proxy
  Http.serve(":8080", Http.newService("kakao.com:80"))

Filters

Services : Logical endpoint
Filters : Service agnostic behavior
- Timeouts
- Retries
- Statistics
- Authentication
- Logging
Filters compose over services
It's a Function!


  trait Filter[ReqIn, ReqOut, RepIn, RepOut] extends
    ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])

Timeout filter


  val timeout = { (req, service) =>
    service(req).within(1.second)
  }

  class TimeoutFilter[Req, Rep](to: Duration)
    extends Filter[Req, Rep, Req, Rep] {
    def apply(req: Req, service: Service[Req, Rep]) =
      service(req).within(to)
    }

Filters are stackable


  val timeout: Filter[…]
  val auth: Filter[…]

  val authAndTimeout = auth andThen timeout

  val service: Service[…]

  val authAndTimeoutService =
    authAndTimeout andThen service

Finagle

Asynchronous

Non-Blocking

Protocol Agnostic

Fullstack RPC

Adopters

Foursquare
ING Bank
Pinterest
Soundcloud
Tumblr
Twitter

Servers

Concurrency Limit
Request Timeout
Metrics and Tracing
HTTP admin interface

Concurrency Limit


  import com.twitter.finagle.Http

  val server = Http.server
  .withAdmissionControl.concurrencyLimit(
    maxConcurrentRequests = 10,
    maxWaiters = 0
  )

Request Timeout


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http

  val server = Http.server
  .withRequestTimeout(42.seconds)

Metrics


  // define your stats
  val counter = statsReceiver.counter("requests_counter")
  // update the value
  counter.incr()
  // The value of this counter will be exported
  // by the HTTP server and accessible at /admin/metrics.json

Admin Interface

Tracing

Zipkin is a distributed tracing system
It helps gather timing data needed to troubleshoot latency problems in microservice architectures
Zipkin’s design is based on the Google Dapper paper.

Zipkin

Clients

Retries
Naming / Service Discovery
Timeouts and Expirations
Load Balancing
Rate Limiting
Connection Pooling
Circuit Breaking
Failure Detection
Metrics and Tracing
Interrupts
Context Propagation
...

Retry Budget


  import com.twitter.finagle.Http
  import com.twitter.conversions.time._
  import com.twitter.finagle.service.RetryBudget
    
  val budget = RetryBudget(
    ttl = 10.seconds,
    minRetriesPerSec = 5,
    percentCanRetry = 0.1
  )
    
  val client = Http.client.withRetryBudget(budget)

  // Allow retrying 10% of total requests on top of 5 retries per sec

Retry Backoff


  import com.twitter.finagle.Http
  import com.twitter.conversions.time._
  import com.twitter.finagle.service.Backoff

  val client = Http.client
  .withRetryBackoff(Backoff.exponentialJittered(2.seconds, 32.seconds))

  // Backoff for rnd(2s), rnd(4s), rnd(8), ..., rnd(32s), ... rnd(32s)

Timeout


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http

  val client = Http.client
  .withTransport.connectTimeout(1.second) // TCP connect
  .withSession.acquisitionTimeout(42.seconds)
  .withSession.maxLifeTime(20.seconds) // connection max life time
  .withSession.maxIdleTime(10.seconds) // connection max idle time

Load Balancing

client treats its server set as replica set
Load balancing involves load distributor and load factor
Currently available
- Load distributors: heap, p2c, aperture, round robin
- Load factors: least loaded, peak EWMA

LB via Aperture


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http
  import com.twitter.finagle.loadbalancer.Balancers

  val balancer = Balancers.aperture(
    lowLoad = 1.0, highLoad = 2.0, // the load band adjusting an aperture
    minAperture = 10 // min aperture size
  )

  val client = Http.client.withLoadBalancer(balancer)

Circuit Breaking

To preemptively disable sessions that will likely fail requests
Placed under LB to exclude particular nodes from its replica set
Policy
- FailFast
- FailureAccrual
- ThresholdFailureDetector

Fail Accrual


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http
  import com.twitter.finagle.service.Backoff
  import com.twitter.finagle.service.FailureAccrualFactory.Param
  import com.twitter.finagle.service.exp.FailureAccrualPolicy

  val twitter = Http.client
  .configured(Param(() => FailureAccrualPolicy.successRate(
    requiredSuccessRate = 0.95,
    window = 100,
    markDeadFor = Backoff.const(10.seconds)
  )))

  // Mark a replica dead if SuccessRate is below 95%
  // over the most recent 100 requests and then backoff for 10 sec.

Finatra

Used in production at Twitter (and many other org)
Actively developed and maintained
Fast, testable, Scala services built on TwitterServer and Finagle
Fully async using Futures
Powerful JSON support

Microservice with Functional Gateway

Microservice with Functional Gateway

More from JongHoon Kim