Microservice with Functional Gateway

Monolithic System

  • Limited to a single language & runtime
  • Can only support few developers
  • Deploys get more dangerous

Distribute!!!

  • Each service written in appropriate language
  • Can only support many developers
  • Granular deploys

But...

Challenges

  • Dozens of network services (different language, different protocol)
  • Highly concurrent
  • Unreliable network, machines
  • Partial failures
  • Failures can cascade

Building a Functional Gateway

A little scala

Scala

  • Hybrid FP/OO language
  • Expressive
  • Statically typed
  • Runs in the JVM
  • Interoperates with Java

Values (Immutable)


    val i: Int = 1234

    val foo: String = "bar"

Type inference


    val i = 1234
    
    val foo = "bar"

Functions are values


    val addOne: Int => Int = { x: Int =>
        x + 1
    }

    addOne(1) == 2

Pure OO

Every value is an object


    val i = 123

    i.toString == "123"

    i.<(333) == true
    
    i < 333 == true

Collections


    Array(1, 2, 3, 4)

    List(1, 2, 3, 4)
    
    Set(1, 1, 2)

Tuple


    val hostPort = ("localhost", 80)

    hostPort._1 == "localhost"

    hostPort._2 == 80

    1 -> 2 == (1,2)

Maps


    Map(1 -> 2)
    
    Map(("foo", "bar"))

    Map(1 -> Map("foo" -> "bar"))

Option


    trait Option[T] {
      def isDefined: Boolean
      def get: T
      def getOrElse(t: T): T
    }
    
    val numbers = Map("one" -> 1, "two" -> 2)
    
    numbers.get("two") == Some(2)
    
    numbers.get("three") == None

Option is a container that may or may not hold something.

Pattern matching


    hostPort match {
      case ("localhost", port) => ...
      case (host, 80) => ...
    }
    
    val result = res1 match {
      case Some(n) => n * 2
      case None => 0
    }

Map

    
    val numbers = List(1, 2, 3, 4)

    numbers.map {(n: Int) => n * 2}

    numbers.map(_ * 2)

Filter


    numbers.filter(_ % 2 == 0)

    val isEven: Int => Boolean = {
      n: Int => n % 2 == 0
    }

    numbers.filter(isEven)

Flatten


    List(List(1, 2), List(3, 4)).flatten

      => List(1, 2, 3, 4)

    List(Some(1), Some(2), None, Some(4)).flatten

      => List(1, 2, 4)

Flatmap


  val nestedNumbers = List(List(1, 2), List(3, 4))

  nestedNumbers.flatMap(x => x.map(_ * 2))

    => List(2, 4, 6, 8)

  nestedNumbers.map((x: List[Int]) => x.map(_ * 2)).flatten

For comprehension


  val numbers = List("123", "456", "789")

  for {
    str <- numbers
    num <- str
  } yield num

    => List(1, 2, 3, 4, 5, 6, 7, 8, 9)

Function compoistion


  def addOne(x: Int) = x + 1

  def addTwo(x: Int) = x + 2

  val addThree = addOne _ compose addTwo _
  // f compose g => f(g(x))

  val addThree2 = addOne _ andThen addTwo _
  // f andThen g => g(f(x))

Your Server as a Function

Marius Eriksen

Building blocks

  • Futures
  • Services
  • Filters

Futures

  • A Future is a placeholder for a result that may not yet exist
  • Futures are how we represent concurrent execution
  • Any sort of asynchronous operation
    • ​Long computation
    • Network call
    • Disk I/O
  • Operations can fail
    • Div by zero
    • Connection failure
    • Timeout

Futures

  • Future are a kind of container
  • A Future[T] occupy one of three states:
    • Empty (pending)
    • Succeeded (with a result of type T)
    • Failed (with a Throwable)

  trait Future[A]

  val f: Future[Int]
  // an integer valued future

Futures as containers

  • Only one element container
  • You can transform them

  Future[T].map[U](f: T => U): Future[U]
  // Converts a Future[T] to a Future[U] by applying f

  Future[T].filter(p: T => Boolean): Future[T]
  // Convert a successful Future[T] to a failed Future[T]
  // by applying predicate

Use callback


  val f: Future[Int] = ???

  Await.result(f) // directly query

  // you use callback functions instead
  f onSuccess { res =>
    println("The result is " + res)
  } onFailure { exc =>
    println("f failed with " + exc)
  }

Promises

  • Futures are read-only
  • A Promise is a writable future

    val p: Promise[Int]
    
    // Success
    p.setValue(1)

    // Failure
    p.setException(new Exception)

Thumbnail extractor


  trait Webpage {
    def imageLinks: Seq[String]
    def links: Seq[String]
  }

  def fetch(url: String): Future[Webpage]

  def getThumbnail(url: String): Future[Webpage] = ???

Thumbnail extractor


  def getThumbnail(url: String): Future[Webpage] = {
    val promise = new Promise[Webpage]
    fetch(url) onSuccess { page =>
      fetch(page.imageLinks(0)) onSuccess { p =>
        promise.setValue(p)
      } onFailure { exc =>
        promise.setException(exc)
      }
    } onFailure { exc =>
      promise.setException(exc)
    }
    promise
  }

Callback-hell...

Sequential composition

  1. Fetching the webpage
  2. Parsing that page to find the first image link
  3. Fetching the image link

Use flatmap


  def flatMap[B](f: A => Future[B]): Future[B]
  • The most important Future combinator
  • flatMap sequences two futures
  • It takes a Future and an asynchronous function and returns another Future
  • If either Future fails, the given Future will also fail

Use flatmap


  def getThumbnail(url: String): Future[Webpage] =
    fetch(url) flatMap { page =>
      fetch(page.imageLinks(0))
    }

Concurrent composition

  • Future provides some concurrent combinators
  • Convert a sequence of Future into a Future of sequence 

  object Future {
    def collect[A](fs: Seq[Future[A]]): Future[Seq[A]]
    def join(fs: Seq[Future[_]]): Future[Unit]
    def select(fs: Seq[Future[A]]): Future[(Try[A], Seq[Future[A]])]
  }

Concurrent composition


  def getThumbnails(url: String): Future[Seq[Webpage]] =
    fetch(url) flatMap { page =>
      Future.collect(
        page.imageLinks map { u => fetch(u) }
      )
    }

Useful for fan-out operations

Recovering from failure

  • Futures must be composable and recoverable
  • flatMap operates over values
  • rescue operates over exceptions 

  def rescue[B](f: Exception => Future[B]): Future[B]

  // Recovering errors...
  val f = fetch(url) rescue {
    case ConnectionFailed =>
      fetch(url)
  }

Services

  • A service is a kind of asynchronous function
  • ex. RPC
    • Dispatch a request
    • Wait a while
    • Succeeds or fails
  • It's a Function!

  trait Service[Req, Rep] extends (Req => Future[Rep])

  val http: Service[HttpReq, HttpRep]
  val redis: Service[RedisCmd, RedisRep]
  val thrift: Service[TFrame, TFrame]

Services are symmetric

Servers implement these, clients make use of them


  // A simple service

  // server
  val multiplier = { i =>
    Future.value(i*2)
  }

  // client
  multiplier(123) onSuccess { res =>
    println("result", r)
  }

Services are symmetric


  // server
  Http.serve(":8080", new Service[HttpReq, HttpRep] {
    def apply(req: HttpReq): Future[HttpRep] = ???
  })

  // client
  val http = Http.newService("kakao.com:80")

  // proxy
  Http.serve(":8080", Http.newService("kakao.com:80"))

Filters

  • Services : Logical endpoint
  • Filters : Service agnostic behavior
    • Timeouts
    • Retries
    • Statistics
    • Authentication
    • Logging
  • Filters compose over services
  • It's a Function!

  trait Filter[ReqIn, ReqOut, RepIn, RepOut] extends
    ((ReqIn, Service[ReqOut, RepIn]) => Future[RepOut])

Timeout filter


  val timeout = { (req, service) =>
    service(req).within(1.second)
  }

  class TimeoutFilter[Req, Rep](to: Duration)
    extends Filter[Req, Rep, Req, Rep] {
    def apply(req: Req, service: Service[Req, Rep]) =
      service(req).within(to)
    }

Filters are stackable


  val timeout: Filter[…]
  val auth: Filter[…]

  val authAndTimeout = auth andThen timeout

  val service: Service[…]

  val authAndTimeoutService =
    authAndTimeout andThen service

Finagle

Asynchronous

Non-Blocking

Protocol Agnostic

Fullstack RPC

Adopters

  • Foursquare
  • ING Bank
  • Pinterest
  • Soundcloud
  • Tumblr
  • Twitter

Servers

  • Concurrency Limit
  • Request Timeout
  • Metrics and Tracing
  • HTTP admin interface

Concurrency Limit


  import com.twitter.finagle.Http

  val server = Http.server
  .withAdmissionControl.concurrencyLimit(
    maxConcurrentRequests = 10,
    maxWaiters = 0
  )

Request Timeout


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http

  val server = Http.server
  .withRequestTimeout(42.seconds)

Metrics


  // define your stats
  val counter = statsReceiver.counter("requests_counter")
  // update the value
  counter.incr()
  // The value of this counter will be exported
  // by the HTTP server and accessible at /admin/metrics.json

Admin Interface

Tracing

  • Zipkin is a distributed tracing system
  • It helps gather timing data needed to troubleshoot latency problems in microservice architectures
  • Zipkin’s design is based on the Google Dapper paper.

Zipkin

Clients

  • Retries
  • Naming / Service Discovery
  • Timeouts and Expirations
  • Load Balancing
  • Rate Limiting
  • Connection Pooling
  • Circuit Breaking
  • Failure Detection
  • Metrics and Tracing
  • Interrupts
  • Context Propagation
  • ...

Retry Budget


  import com.twitter.finagle.Http
  import com.twitter.conversions.time._
  import com.twitter.finagle.service.RetryBudget
    
  val budget = RetryBudget(
    ttl = 10.seconds,
    minRetriesPerSec = 5,
    percentCanRetry = 0.1
  )
    
  val client = Http.client.withRetryBudget(budget)

  // Allow retrying 10% of total requests on top of 5 retries per sec

Retry Backoff


  import com.twitter.finagle.Http
  import com.twitter.conversions.time._
  import com.twitter.finagle.service.Backoff

  val client = Http.client
  .withRetryBackoff(Backoff.exponentialJittered(2.seconds, 32.seconds))

  // Backoff for rnd(2s), rnd(4s), rnd(8), ..., rnd(32s), ... rnd(32s)

Timeout


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http

  val client = Http.client
  .withTransport.connectTimeout(1.second) // TCP connect
  .withSession.acquisitionTimeout(42.seconds)
  .withSession.maxLifeTime(20.seconds) // connection max life time
  .withSession.maxIdleTime(10.seconds) // connection max idle time

Load Balancing

  • client treats its server set as replica set
  • Load balancing involves load distributor and load factor
  • Currently available
    • Load distributors: heap, p2c, aperture, round robin
    • Load factors: least loaded, peak EWMA

LB via Aperture


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http
  import com.twitter.finagle.loadbalancer.Balancers

  val balancer = Balancers.aperture(
    lowLoad = 1.0, highLoad = 2.0, // the load band adjusting an aperture
    minAperture = 10 // min aperture size
  )

  val client = Http.client.withLoadBalancer(balancer)

Circuit Breaking

  • To preemptively disable sessions that will likely fail requests
  • Placed under LB to exclude particular nodes from its replica set
  • Policy
    • FailFast
    • FailureAccrual
    • ThresholdFailureDetector

Fail Accrual


  import com.twitter.conversions.time._
  import com.twitter.finagle.Http
  import com.twitter.finagle.service.Backoff
  import com.twitter.finagle.service.FailureAccrualFactory.Param
  import com.twitter.finagle.service.exp.FailureAccrualPolicy

  val twitter = Http.client
  .configured(Param(() => FailureAccrualPolicy.successRate(
    requiredSuccessRate = 0.95,
    window = 100,
    markDeadFor = Backoff.const(10.seconds)
  )))

  // Mark a replica dead if SuccessRate is below 95%
  // over the most recent 100 requests and then backoff for 10 sec.

Finatra

  • Used in production at Twitter (and many other org)
  • Actively developed and maintained
  • Fast, testable, Scala services built on TwitterServer and Finagle
  • Fully async using Futures
  • Powerful JSON support

Q & A