Anatomy of a twitter microservice

μWURFL Case Study

Cloud happens 💩

Your server as a function

http://monkey.org/~marius/funsrv.pdf

This quality of being able to store strain energy and deﬂect elastically under a load without breaking is called ‘resilience’, and it is a veryvaluable characteristic in a structure. Resilience may be deﬁned as ‘the amount of strain energywhich can be stored in a structure without causing permanent damage to it’.

— Gordon, J.e.“Structures.”

Futures
Services
Filters

Results of asyncronous operations are rapresented as composable futures, expressing the dependencies between operations

Systems boundaries are represented by asynchronous functions called services. Symmetric and uniform API represents both clients and servers

Application-agnostic concerns (e.g. timeouts, retries, authentication) are encapsulated by filters which compose to build services from multiple independent modules

FINAGLE

Finagle is an extensible RPC system for the JVM, used to construct high-concurrency servers. Finagle implements uniform client and server APIs for several protocols, and is designed for high performance and concurrency. Most of Finagle’s code is protocol agnostic, simplifying the implementation of new protocols.

connection pools, with throttling to avoid TCP connection churn
failure detectors, to identify slow or crashed hosts
failover strategies, to direct traffic away from unhealthy hosts
load-balancers, including “least-connections” and other strategies
back-pressure techniques, to defend servers against abusive clients and dogpiling.
standard statistics, logs, and exception reports
distributed tracing (a la Dapper) across protocols
optionally uses ZooKeeper for cluster management
supports common sharding strategies.

server announcing itself to Zookeeper

client

val client = Http.newService("uwurfl.dev.williamhill.plc:10000")
or..
val client = Http.newService("uwurfl1.dev.wh.plc:10000,uwurfl2.dev.wh.plc:10000")

Await.ready(Httpx.serveAndAnnounce(":*", "zk!zk.dev.wh.plc:2181!/zktest!0", foo))

val client = Httpx.newService("zk2!localhost:2181!/zktest")
val req = RequestBuilder().url("http://foo").buildGet

client using namespace for discovery

LOOK MA, NO LOAD BALANCERS & TOTALY DYNAMIC! 😀

Any RPC protocol can be mapped onto Finagle

Thrift
Memcached
MySQL
...

Unified Metrics
makes it easy to
understand what
you are monitorIng

(and they are always the same)

Look Pa, PErcentiles!

ZIPKIN

Trace all the things

register any finagle client/server with zipkin
val zipkinTracer = ZipkinTracer.mk("zipkin.dev.wh.plc", port = 9410, sampleRate = 1f)

Distributed tracing application based on the Google Dapper paper

Trace Identifiers

Trace ID
Span ID
Parent ID

Common RPC annotations

cs - client start

sr - server receive
ss - server send
cr - client receive

http://research.google.com/pubs/pub36356.html

A common Language for TRAcing

http://www.slideshare.net/johanoskarsson/zipkin-strangeloop

https://blog.twitter.com/2013/observability-at-twitter

https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

Twitter Server

TwitterServer defines a template from which servers at Twitter are built. It provides common application components such as an administrative HTTP server, tracing, stats, etc. These features are wired in correctly for use in production at Twitter.

Features

Flags (config)
Logging (standard)
Metrics exposed as JSON(standard)
HTTP admin interface
Lifecycle management

Admin console (built in)

FINATRA

Fast, testable, Scala HTTP services

Features

Flags (config)
Logging (standard)
Metrics (standard)
HTTP admin interface
Lifecycle management

Simple Controllers

Strongly defined Request model & Validation

class Example extends Controller {
  get("/") { request: Request =>
    "hi"
  }
}

http://foo.com/users?max=10&start_date=2014-05-30TZ&verbose=true

case class UsersRequest(
  @Max(100) @QueryParam max: Int,
  @PastDate @QueryParam startDate: Option[DateTime],
  @QueryParam verbose: Boolean = false)

Dependency Injection without XML

class MyController @Inject()(dao: GroupsDAO, service: FooService) extends Controller

WARMUP+AUTOMATIC SMOKE TEST

Send custom stats to local receiver

class WurflWarmupHandler @Inject()(httpWarmup: HttpWarmup) extends Handler {
  override def handle() = {
    httpWarmup.send(
      get("/v1/json"),
      responseCallback = expectOkResponse(_)
    )
  }
}

class MyService @Inject()(
  stats: StatsReceiver)

Injection of environmental variables via flags

object MyModule1 extends TwitterModule {
  val key = flag("key", "defaultkey", "The key to use.")

  @Singleton
  @Provides
  def providesThirdPartyFoo: ThirdPartyFoo = {
    new ThirdPartyFoo(key())
  }
}

Mustache for templated responses

FILTER CHAIN

@Mustache("foo")
case class FooView(
  name: String)

get("/foo") { request: Request =>
  FooView("abc")
}

get("/foo") { request: Request =>
  response.notFound(
    FooView("abc"))
}

class Server extends HttpServer {
  override configureHttp(router: HttpRouter) {
    router.
      filter[CommonFilters].
      filter[LoggingMDCFilter].
      filter[FinagleRequestScopeFilter].
      filter[UserFilter].
      add[MyController1].
      add[MyController2]
  }
}

Logging trait, no boilerplate

class MyClass extends Logging {
  def foo() = {
    info("Calculating...")
    "bar"
  }
}

Test framework included

import com.google.inject.Stage
import com.twitter.finatra.http.test.EmbeddedHttpServer
import com.twitter.inject.server.FeatureTest

class MyServiceStartupTests extends FeatureTest {
  val server = new EmbeddedHttpServer(
    stage = Stage.PRODUCTION,
    twitterServer = new SampleApiServer,
    extraArgs = Seq(
      "-com.twitter.server.resolverMap=myservice=nil!"))

  "SampleApiServer" should {
    "startup" in {
      server.assertHealthy()
    }
  }
}

Access Log (same pattern as apache)

service log (application Log)

0:0:0:0:0:0:0:1 - - [16/Oct/2015:00:59:45 +0000] "GET /v1/json HTTP/1.1" 200 202 5 "curl/7.43.0" 3dad52fbec16da6a
0:0:0:0:0:0:0:1 - - [16/Oct/2015:00:59:46 +0000] "GET /v1/json HTTP/1.1" 200 202 1 "curl/7.43.0" 520d2b910eef9eea

2015-10-16 02:59:26,604 INFO                  HttpWarmup                Warmup Request("GET /v1/json", from 0.0.0.0/0.0.0.0:0) complete with Status(200)
2015-10-16 02:59:27,138 INFO                  DefaultTracer$            Tracer: com.twitter.finagle.zipkin.thrift.SamplingTracer
2015-10-16 02:59:27,145 INFO                  WurflServerMain$          http server started on port: 8888
2015-10-16 02:59:27,146 INFO                  WurflServerMain$          Enabling health endpoint on port 9990
2015-10-16 02:59:27,146 INFO                  WurflServerMain$          App started.
2015-10-16 02:59:27,147 INFO                  WurflServerMain$          Startup complete, server ready.
2015-10-16 02:59:45,185 INFO 3dad52fbec16da6a WurflController           Entering /v1/json with curl/7.43.0
2015-10-16 02:59:45,186 DEBUG 3dad52fbec16da6a WurflServiceImpl          Looking up capabilities.
2015-10-16 02:59:45,187 INFO 3dad52fbec16da6a ScientiaWurflEngine       capabilities=[generic_curl_based_crawler, match=conclusive]
2015-10-16 02:59:45,187 INFO 3dad52fbec16da6a WurflController           Capabilities found DeviceCapabilities(Map(device_os -> , is_wireless_device -> false, physical_screen_height -> 400, is_mobile -> false, is_tablet -> false, device_os_version -> , physical_screen_width -> 400))
2015-10-16 02:59:46,024 INFO 520d2b910eef9eea WurflController           Entering /v1/json with curl/7.43.0
2015-10-16 02:59:46,024 DEBUG 520d2b910eef9eea WurflServiceImpl          Looking up capabilities.
2015-10-16 02:59:46,025 INFO 520d2b910eef9eea ScientiaWurflEngine       capabilities=[generic_curl_based_crawler, match=cached]
2015-10-16 02:59:46,025 INFO 520d2b910eef9eea WurflController           Capabilities found DeviceCapabilities(Map(device_os -> , is_wireless_device -> false, physical_screen_height -> 400, is_mobile -> false, is_tablet -> false, device_os_version -> , physical_screen_width -> 400))
2015-10-16 02:59:46,633 INFO aa0cbfd36849d955 WurflController           Entering /v1/json with curl/7.43.0
2015-10-16 02:59:46,633 DEBUG aa0cbfd36849d955 WurflServiceImpl          Looking up capabilities.
2015-10-16 02:59:46,633 INFO aa0cbfd36849d955 ScientiaWurflEngine       capabilities=[generic_curl_based_crawler, match=cached]
2015-10-16 02:59:46,634 INFO aa0cbfd36849d955 WurflController           Capabilities found DeviceCapabilities(Map(device_os -> , is_wireless_device -> false, physical_screen_height -> 400, is_mobile -> false, is_tablet -> false, device_os_version -> , ph

With MDC in the logs I can follow the requests!

UNIFICATION OF LOGS

UWURFL Customisations

Bridged the Twitter Metrics to Dropwizard metrics and then exported to JMX in order to integrate with Wily

https://github.com/rlazoti/finagle-metrics

https://dropwizard.github.io/

UWURFL Customisations

Exported Dropwizard metrics to Graphite and then Graphana (free)

SBT-native-packager

Universal deployment

μWURFL

All of this in order to get a bit of json

curl -H "user-agent: Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) Mobile/1A543a Safari/419.3" localhost:8888/v1/json

{
   "timestamp" : 1445003791106,
   "capabilities" : {
      "is_mobile" : true,
      "device_os" : "iOS",
      "device_os_version" : 1,
      "is_tablet" : false,
      "is_wireless_device" : true,
      "physical_screen_height" : 74,
      "physical_screen_width" : 50
   }
}

Anatomy of a twitter microservice

Cloud happens 💩

Your server as a function

http://monkey.org/~marius/funsrv.pdf

Futures Services Filters

FINAGLE

server announcing itself to Zookeeper

client

client using namespace for discovery

LOOK MA, NO LOAD BALANCERS & TOTALY DYNAMIC! 😀

Any RPC protocol can be mapped onto Finagle

Unified Metrics makes it easy to understand what you are monitorIng

Look Pa, PErcentiles!

ZIPKIN

Trace all the things

Trace Identifiers

Common RPC annotations

A common Language for TRAcing

Twitter Server

Features

Admin console (built in)

FINATRA

Features

Simple Controllers

Strongly defined Request model & Validation

Dependency Injection without XML

WARMUP+AUTOMATIC SMOKE TEST

Send custom stats to local receiver

Injection of environmental variables via flags

Mustache for templated responses

FILTER CHAIN

Logging trait, no boilerplate

Test framework included

Access Log (same pattern as apache)

service log (application Log)

With MDC in the logs I can follow the requests!

UWURFL Customisations

UWURFL Customisations

SBT-native-packager

Universal deployment

All of this in order to get a bit of json

Futures
Services
Filters

Unified Metrics
makes it easy to
understand what
you are monitorIng