Poweramp

Microservices at iHeartRadio

Kai(luo) Wang @kailuowang

Poweramp Architecture

akka cluster

API gateway

w/ predeployed endpoints

 

distributed endpoints

akka service

 

 

 

distributed endpoints

akka service

 

 

 

 

akka service

 

 

data

stores

external services

message queues

 

 

command

center

 

 

 

 

metrics monitor

 

 

http

tcp

Akka Cluster

  • Out-of-the-box clustering infrastructure

  • Loose coupling through typed binary protocol 

  • Transparent programming model within and across microservices

  • High resiliency, performance and scalability, built-in

  • Strong community and commercial support

Distributed Endpoints

API gateway

(play app)

 

Microservice

(akka app) 

 

register endpoints

request work pipeline

(powered by asobu)

(powered by kanaloa)

Why distributed endpoints

 

1, cross cutting logic in api gateway e.g. authentication, error handling, validation

2, deploy cross cutting logic without redeploying microservices

3, microservices remain independent

Distributed Endpoints

routes file

###
#  summary: Get a user's tests and the groups he/she is in
#  parameters:
#    - name: time
#      description: epoch millis for the time as of when
#  responses:
#    200:
#      description: A map of test ids to group names
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.GroupsResult'
###
GET   /users/:userId/tests/groups    abtest.MainController.getGroups(userId: Int, time: Long)

###
#  summary: create a new a/b test
#  parameters:
#    - name: body
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.CreateTest'
#  responses:
#    200:
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.TestCreated'
###
POST      /tests      poweramp.abtest.MainController.createTest()

Distributed Endpoints

controller DSL

package abtest

import ...

case class MainController(backend: ActorRef)(
  implicit ...
) extends DistributedController {

  handle(
    "getGroups",
    process[GetGroups]()
  )(using(backend).
      expect[GroupsResult] >>
      respondJson(Ok))

  handle(
    "createTest",
    process[CreateTest](
     from(author = authenticatedUserName),
     from(test   = jsonBody[ABTest]))          
  )(using(backend).
      expect[TestCreated] >>
      respondJson(Ok))
}
package api.abtest.service

object Protocol {
  case class GetGroups(
    userId: ProfileId,
    time:   DateTime
  ) 
  
  case class GroupsResult(
     groups: Map[TestId, GroupName]
  )

  case class CreateTest(
    test:    ABTest,
    author:  String
  )
  
  case class ABTest(
    name:                TestName,
    start:               DateTime,
    end:                 DateTime,
    statisticalTestType: StatisticalTestType,
    groups:              Vector[Group]
  ) 

  case class TestCreated(
    test: ABtest
  ) 
}

Reactive Work Pipeline

Smart controlled pressure on backend

  • Little's law inspired back pressure

  • optimal concurrency size exploring 

  • circuit breaker

(powered by kanaloa)

Metrics Monitor

  • throughput

  • failures

  • inbound requests

  • queue length

  • expected wait time

  • process time

  • circuit breaks

  • concurrency size

(powered by kanaloa)

Distributed Endpoints

routes file

###
#  summary: Get a user's tests and the groups he/she is in
#  parameters:
#    - name: time
#      description: epoch millis for the time as of when
#  responses:
#    200:
#      description: A map of test ids to group names
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.GroupsResult'
###
GET   /users/:userId/tests/groups    abtest.MainController.getGroups(userId: Int, time: Long)

###
#  summary: create a new a/b test
#  parameters:
#    - name: body
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.CreateTest'
#  responses:
#    200:
#      schema:
#        $ref: '#/definitions/api.abtest.service.Protocol.TestCreated'
###
POST      /tests      poweramp.abtest.MainController.createTest()

Swagger Integration

(powered by play-swagger)

Command Center

nodes monitor

(adapted from muuki88)

events monitor

job monitor/commander

SBT utils

  • bootstrap cluster

  • start/stop services

  • report services status

  • report remote cluster status 

  • blue-green deployment

What's coming

  • protobuf(or ion) 

    • generate case classes from message types

  • multi-service versioning 

  • auto-scale service instances

  • continues delivery

iHeartRadio OSS Links

https://github.com/iheartradio/...

Other OSS

 

  • cats (abstractions for functional programming)

  • shapeless (generic programming)

  • kittens (combine the above two)

(other than akka and play)

Poweramp is developed by

the iHeartRadio AMP Team

  • Amit Patel 
  • Joseph Price - @joprice
  • Kailuo Wang - @kailuowang
  • Laurent Vauthrin
  • Marcel Walden
  • Matt Fielder 
  • William Ho

Special thanks

  • Adam Denenberg  - @denen
  • James Roper     - @jroper
  • Jim Powers      - @corruptmemory
  • Konrad Malawski - @ktosopl
  • Miles Sabin     - @milessabin
  • Patrik Nordwall - @patriknw
  • Rich Dougherty  - @richdougherty

Functional Building Block

val getUser: Id => User

A => B

type K[A, B] =
  Kleisli[XorT[Future, Reason, ?], A, B] 

val getUser: Kleisli[
              XorT[Future, Reason, ?], Id, User
             ]

Function Composition

case class User(name: Name, age: Age) 

def validateUser: K[(AccountName, Password), Id]

def getUserName: K[Id, Name]

def getUserAge: K[Id, Age]


validateUser andThen getUserName

Monadic Composition

def getUser:  K[Id, User] =
  for {
    name <- getUserName
    age <- getUserAge
  } yield User(name, age)

Generic Composition

val gen = sequenceGeneric[User]

def getUser: K[Id, User] = 
  gen(
    name = getUserName,
    age = getUserAge 
  )

Did you remember to rate the previous
session?

Poweramp_Scaladays

By Kailuo Wang

Made with Slides.com

Poweramp_Scaladays

An overview of Poweramp, a microservice platform at iHeartRadio.

  • 2,388

More from Kailuo Wang