Real-world

functional Scala

Chris Birchall

1st Oct 2016

OOP

time

Java dev

"better Java"

discover

Scala

rediscover

FP

Case study: AMIgo

• Self-serve AMI bakery

• Implemented in Scala + Play

• Very impure/effect-heavy domain

  • File I/O, external processes, DynamoDB, ...

• Event bus

• Not many tests

https://github.com/guardian/amigo

DEMO

Package structure

Ansible stuff

Play controllers

DynamoDB DAOs

Event bus stuff

Domain models

Packer stuff

Guardian-specific stuff

Scheduler stuff

Play view templates

Play DI stuff

What went well?

Case study: AMIgo

What could be improved?

What AMIgo got right

• Functional modelling

• Prefer objects to classes

• Avoid traits

• No lazy vals

• Minimise boilerplate

Functional modelling

Preamble

OOP design

Data

Behaviour

trait Recipe {
  def generatePackerConfig(): File
}
trait ProcessRunner {
  def run(cmd: Seq[String]): Future[Int]
}

class BakeImpl(
  id: BakeId,
  buildNumber: Int,
  recipe: Recipe,
  processRunner: ProcessRunner) extends Bake {

  def execute(): Future[Int] = {
    val packerFile = 
      recipe.generatePackerConfig()
    val packerCommand = 
      buildPackerCommand(packerFile, buildNumber)
    processRunner.run(packerCommand)
  }
  
  ...
}

Data

Behaviour

Decouple data from behaviour

FP design

Model your domain using

Algebraic Data Types (ADTs)

Data

Sum type

• A or B or ...
• Represented in Scala using sealed trait/abstract class

sealed abstract class BakeStatus

case object Running extends BakeStatus
case object Complete extends BakeStatus
case object Failed extends BakeStatus

Product type

• A and B and ...
• Represented in Scala using case class

case class Bake(
    recipe: Recipe,
    buildNumber: Int,
    status: BakeStatus,
    amiId: Option[AmiId],
    startedBy: String,
    startedAt: DateTime
)

Use the type system to your advantage

Data

case class AmiId(value: String) extends AnyVal
case class BaseImageId(value: String) extends AnyVal
case class RecipeId(value: String) extends AnyVal
case class RoleId(value: String) extends AnyVal
case class BakeId(recipeId: RecipeId, buildNumber: Int)

Value class: no runtime overhead

Use the type system to your advantage

Data

object Bakes {

  def updateAmiId(bakeId: BakeId, amiId: AmiId): Unit = ...

}

val bakeId = BakeId(RecipeId("my-recipe"), buildNumber = 123)
val amiId = AmiId("ami-a1b2c3d4")

Bakes.updateAmiId(amiId, bakeId) // yay, compile error!

Model your errors

Data

def loadRoleFromDisk(roleId: RoleId): Role = {
  ...
  if (dirNotFound)
    throw new IOException("Role directory not found")
  ...
  if (invalid)
    throw new InvalidRoleException("check ur role!")
  ...
}

def loadRoleFromDisk(roleId: RoleId):
    Either[IOException,
           Either[InvalidRoleException,
                  Role]] = ...

Nope!

Hmm...

Model your errors

Data

sealed trait LoadRoleResult

object LoadRoleResult {
  case class Success(role: Role) extends LoadRoleResult
  case object RoleDirectoryNotFound extends LoadRoleResult
  case object RoleFormatInvalid extends LoadRoleResult
}

def loadRoleFromDisk(roleId: RoleId): LoadRoleResult = ...

Downside: don't get map/flatMap for free

Model your errors

Data

sealed trait LoadRoleError

object LoadRoleError {
  case object RoleDirectoryNotFound extends LoadRoleError
  case object RoleFormatInvalid extends LoadRoleError
}

def loadRoleFromDisk(roleId: RoleId): Either[LoadRoleError, Role] = ...

Keep an eye out for monoids

Data

TODO

This gives you a bunch of stuff for free

e.g. foldMap

case class Result(successes: Int, failures: Int)

implicit val rm = new Monoid[Result] {
  def empty = Result(0, 0)
  def combine(x: Result, y: Result) = Result(
    successes = x.successes + y.successes,
    failures = x.failures + y.failures
  )
}

Keep an eye out for monoids

Data

TODO

Maps the function over the list and combines the results

def processPageOfItems(page: Page[Item]): Result = ...
val pages: List[Page] = ...

import cats.instances.list._
import cats.syntax.foldable._

val overallResult: Result = 
  pages.foldMap(processPageOfItems)

Behaviour

Data

• ADTs
• Strongly typed
• Model all the things

Model behaviour as plain old functions

Behaviour

object PlaybookGenerator {

  def renderRole(role: CustomisedRole): String = ...

  def generatePlaybook(recipe: Recipe): Yaml = {
    val allRoles = recipe.baseImage.builtinRoles ++ recipe.roles
    val roleStrings = allRoles.map(renderRole)

    Yaml(...)
  }

  ...
}

Larger functions composed of smaller ones

Behaviour

Function hierarchy

Data

• ADTs
• Strongly typed
• Model all the things

Some functions are purer than others

Behaviour

object PackerRunner {

  def createImage(bake: Bake): Future[ExitCode] = {
    val playbookYaml = 
      PlaybookGenerator.generatePlaybook(bake.recipe)
    val playbookFile = writeToTempFile(playbookYaml)

    val packerBuildConfig = PackerBuildConfigGenerator
      .generatePackerBuildConfig(bake)
    val packerConfigFile = writeToTempFile(packerBuildConfig)

    executePacker(bake, playbookFile, packerConfigFile)
  }

  ...
}

Pure 😇

Pure 😇

Impure 🙈

Impure 🙈

Impure 🙈

Behaviour

Pure

functions

Effects

Data

• ADTs
• Strongly typed
• Model all the things

What AMIgo got right

• Functional modelling

• Prefer objects to classes

• Avoid traits

• No lazy vals

• Minimise boilerplate

Prefer objects to classes

vs

object PackerRunner {

  def createImage(bake: Bake): Future[ExitCode] = {
    ...
  }

}

class PackerRunner(bake: Bake) {

  def createImage(): Future[ExitCode] = {
    ...
  }

}

Prefer objects to classes

Benefits of objects

• Singleton
  • No lifecycle to worry about
• Principle of least power
  • No state, just a bunch of functions

Prefer objects to classes

Problem: You end up passing lots of params around

Partial solution: Pass them implicitly

If this gets excessive, bundle into a single context class

(c.f. ScalaCache)

object Recipes {

  def list()(implicit dynamo: Dynamo): Iterable[Recipe] = {
    ...
  }
 
  ...
}

class RecipeController(...)(implicit dynamo: Dynamo) 
    extends Controller {

  def listRecipes = Action {
    Ok(views.html.recipes(Recipes.list()))
  }

  ...
}

Caveat: Daniel Spiewak calls this an anti-pattern

... but it's fine if resource lifecycle == app lifecycle

Prefer objects to classes

Problem: how to do Dependency Injection?

Supplementary question: why do we do DI?

Dependency injection

Why do we do DI?

1. Mocking dependencies so we can test in isolation
2. Separating business logic from resource management

OOP

FP

DAG of dependencies

Function composition

Dependency injection

1. Mocking dependencies

Pass functions as parameters

case class PackerInput(bake: Bake, playbookFile: Path, packerConfigFile: Path)

def createImage(bake: Bake,
  createPlaybook: Recipe => Path,
  createPackerConfig: Bake => Path,
  runPacker: PackerInput => Future[ExitCode]): Future[ExitCode] = {

  val playbookFile = createPlaybook(bake.recipe)
  val packerConfigFile = createPackerConfig(bake)
  runPacker(PackerInput(bake, playbookFile, packerConfigFile))
}

Dependency injection

2. Resource management

Bad:

Better:

def getItem(id: Int): Option[Item] = {
  val conn = new DBConnection()
  try {
    ... 
  } finally { 
    conn.close()
  }
}

def getItem(id: Int, 
            conn: DBConnection): Option[Item] = {
  ...
}

Dependency injection

2. Resource management

Even better:

def getItem(id: Int) = { (conn: DBConnection) => 
  ...
}

val run: DBConnection => Item = getItem(123)
val item: Item = run(new DBConnection())

Curried function

• Pure
• Separates description from execution
• Composable (we'll come back to this)

What AMIgo got right

• Functional modelling

• Prefer objects to classes

• Avoid traits

• No lazy vals

• Minimise boilerplate

Avoid traits

Traits are attractive to OOP devs

• Multiple inheritance, diamond inheritance
• Mixin = exciting inheritance/composition hybrid
• Can be used like a Java/C# interface
• Support fancy patterns, e.g. abstract override

Avoid traits

Problems with traits

• Initialisation order gotchas
  • "NullPointerException?! I thought Scala didn't have nulls!"
  • Hack around with lazy vals (or, shudder, early init)
• Encourages inheritance, overriding, OOP design

Avoid traits

Corollary: no cake!

What AMIgo got right

• Functional modelling

• Prefer objects to classes

• Avoid traits

• No lazy vals

• Minimise boilerplate

No lazy vals

• Runtime overhead
• Risk of deadlock
• Cognitive overhead
  • Initialisation of lazy val ≈ side-effect

object HandyService {
  lazy val usefulData = expensiveComputation()
}

object MyCode {

  def performanceCriticalCode() = {
    val foo = HandyService.usefulData
    ...
  }

}

Is this OK?

Has somebody already taken the performance hit for us?

What AMIgo got right

• Functional modelling

• Prefer objects to classes

• Avoid traits

• No lazy vals

• Minimise boilerplate

Data

• ADTs
• Strongly typed
• Model all the things

Behaviour

Pure

functions

Effects

Minimise boilerplate

Glue, wiring, boilerplate

Minimise boilerplate

Scanamo

• One line to write arbitrary case class to DynamoDB
• Codec is automatically derived using shapeless

object Bakes {

  def create(...)(implicit dynamo: Dynamo): Bake = {
    val bake = Bake(...)
    val dbModel = ...
    Scanamo.put(dynamo.client)(tableName)(dbModel)
    bake
  }

  ...
}

Minimise boilerplate

Automagic

Transforms between domain models and DB models

object Bake {

  case class DbModel(
    recipeId: RecipeId,
    buildNumber: Int,
    status: BakeStatus,
    amiId: Option[AmiId],
    startedBy: String,
    startedAt: DateTime)

  import automagic._

  def domain2db(bake: Bake): DbModel =
    transform[Bake, Bake.DbModel](bake, "recipeId" -> bake.recipe.id)

  def db2domain(dbModel: DbModel, recipe: Recipe): Bake =
    transform[Bake.DbModel, Bake](dbModel, "recipe" -> recipe)

}

Minimise boilerplate

Enumeratum

Makes sealed classes work like enums

sealed abstract class BakeStatus extends EnumEntry

object BakeStatus extends Enum[BakeStatus] {

  val values = findValues

  case object Running extends BakeStatus
  case object Complete extends BakeStatus
  case object Failed extends BakeStatus

}

val running: BakeStatus = BakeStatus.withName("Running")

What could be improved

• Separation of pure and impure code

  • Try the Reader monad for DI

• Separate description from execution

  • Try the Free monad

Measures of success

• More of the code can be tested

• No loss of readability/understandability

  • Who is the target audience?

Reader monad

Wraps a function f: A => B

scala> import cats.data.Reader
import cats.data.Reader

scala> val f: Int => String = _.toString
f: Int => String = <function1>

scala> val reader = Reader(f)
reader: cats.data.Reader[Int,String] = ...

scala> reader.run(123)
res1: cats.Id[String] = 123

umm... why?

Reader monad

Recap: DI via currying

def getItem(id: Int) = { (conn: DBConnection) => 
  ...
}

val run: DBConnection => Item = getItem(123)
val item: Item = run(new DBConnection())

Reader monad

Composition with currying

def getItem(id: Int): DBConnection => Item = { 
  (conn: DBConnection) =>
    // read from DB
    Item(id, "widget")
}

def renameItem(item: Item, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    val item = getItem(id)(conn)
    renameItem(item, newName)(conn)
}

val item: Item = renameItemWithId(123, "hello")(new DBConnection())

Reader monad

Composition with Reader

import cats.data.Reader
 
def getItem(id: Int) = Reader[DBConnection, Item]{ 
  conn =>
    // read from DB
    Item(id, "widget")
}

def renameItem(item: Item, newName: String) = Reader[DBConnection, Item]{ 
  conn =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): Reader[DBConnection, Item] = {
  for {
    item <- getItem(id)
    updatedItem <- renameItem(item, newName)
  } yield updatedItem
}

val result: Item = renameItemWithId(123, "hello").run(new DBConnection())

Reader monad

Slightly more realistic example

def getItem(id: Int): DBConnection => Option[Item] = { 
  (conn: DBConnection) =>
    // read from DB
    Some(Item(id, "widget"))
}

def renameItem(item: Item, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): DBConnection => Option[Item] = { 
  (conn: DBConnection) =>
    val item = getItem(id)(conn)
    item.map(x => renameItem(x, newName)(conn))
}

val item: Option[Item] = renameItemWithId(123, "hello")(new DBConnection())

Reader monad

Again with Reader

import cats.data.Reader
 
def getItem(id: Int) = ReaderT[Option, DBConnection, Item]{ 
  conn =>
    // read from DB
    Some(Item(id, "widget"))
}

def renameItem(item: Item, newName: String) = Reader[DBConnection, Item]{ 
  conn =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): ReaderT[Option, DBConnection, Item] = {
  for {
    item <- getItem(id)
    updatedItem <- renameItem(item, newName).lift[Option]
  } yield updatedItem
}

val result: Option[Item] = renameItemWithId(123, "hello").run(new DBConnection())

Reader monad

How would this look in AMIgo?

def createImage(bake: Bake, prism: Prism, eventBus: EventBus)
               (implicit packerConfig: PackerConfig): Future[ExitCode] = {
  val playbookYaml = PlaybookGenerator.generatePlaybook(bake.recipe)
  val playbookFile = TempFiles.writePlaybookToFile(playbookYaml, bake.recipe.id)

  prism.findAllAWSAccountNumbers() flatMap { awsAccountNumbers =>
    Logger.info(s"AMI will be shared with the following AWS accounts: $awsAccountNumbers")
    val packerBuildConfig = PackerBuildConfigGenerator.generatePackerBuildConfig(
      bake, playbookFile, awsAccountNumbers)
    val packerConfigFile = 
       TempFiles.writePackerConfigToFile(packerBuildConfig, bake.recipe.id)

    PackerRunner.executePacker(bake, playbookFile, packerConfigFile, eventBus)
  }
}

Starting point

Reader monad

def createImage(
  createPlaybook: Recipe => Path,
  findAllAWSAccounts: WSClient => Future[Seq[AWSAccount]],
  createPackerBuildConfig: (Bake, Path, Seq[AWSAccount], PackerConfig) => Path,
  executePacker: (Bake, Path, Path, EventBus) => Future[ExitCode])
  (bake: Bake, 
  eventBus: EventBus, 
  wSClient: WSClient, 
  packerConfig: PackerConfig): Future[ExitCode] = {

  val playbookFile = createPlaybook(bake.recipe)

  findAllAWSAccounts(wSClient) flatMap { awsAccounts =>
    Logger.info(s"AMI will be shared with the following AWS accounts: $awsAccounts")

    val packerConfigFile = 
      createPackerConfigToFile(bake, playbookFile, awsAccounts, packerConfig)

    PackerRunner.executePacker(bake, playbookFile, packerConfigFile, eventBus)
  }
}

Pass functions as arguments

Reader monad

// In AppComponents.scala

val createImage: (Bake, EventBus, WSClient, PackerConfig) = BuildService.createImage(
  createPlaybook = PlaybookGenerator.generatePlaybook andThen TempFiles.writeToFile,
  findAllAWSAccounts = ...,
  ...
) _

Use currying to hide wiring

// In controller

val bake = Bakes.create(recipe, buildNumber, startedBy = request.user.fullName)
createImage(bake, eventBus, wsClient, packerConfig)

Reader monad

class CreateImageContext(
  eventBus: EventBus, 
  wSClient: WSClient, 
  packerConfig: PackerConfig
)

def createImage(
  createPlaybook: Recipe => Path,
  findAllAWSAccounts: WSClient => Future[Seq[AWSAccount]],
  createPackerBuildConfig: (Bake, Path, Seq[AWSAccount], PackerConfig) => Path,
  executePacker: (Bake, Path, Path, EventBus) => Future[ExitCode])
  (bake: Bake)(context: CreateImageContext): Future[ExitCode] = {

  ...
}

Bundle dependencies into a context

Reader monad

def createImage(
  createPlaybook: Recipe => Future[Path],
  findAllAWSAccounts: ReaderT[Future, WSClient, Seq[AWSAccount]],
  createPackerBuildConfig: 
    (Bake, Path, Seq[AWSAccount]) => ReaderT[Future, PackerConfig, Path],
  executePacker: (Bake, Path, Path) => ReaderT[Future, EventBus, ExitCode])
  (bake: Bake): ReaderT[Future, CreateImageContext, ExitCode] = {

  ...
}

Reader-ify

Any function that took one of the dependencies as an argument now returns a Reader (or ReaderT)

Reader monad

def createImage(
  ...
  )
  (bake: Bake): ReaderT[Future, CreateImageContext, ExitCode] = {

  ...
  ... wiring to turn all the params into ReaderT[Future, CreateImageContext, A]
  ...

  for {
    playbookFile <- writePlaybook(playbookYaml)
    accounts <- findAccounts
    _ = Logger.info(s"AMI will be shared with the following AWS accounts: $accounts")
    packerConfigFile <- writePackerBuildConfig(bake, playbookFile, accounts)
    exitCode <- runPacker(PackerInput(bake, playbookFile, packerConfigFile))
  } yield {
    exitCode
  }
}

Compose the readers

Reader monad

type UseContext[A] = ReaderT[Future, CreateImageContext, A]

val writePlaybook: Recipe => UseContext[Path] = recipe =>
  ReaderT.lift[Future, CreateImageContext, Path](createPlaybook(recipe))

val findAccounts: UseContext[Seq[AWSAccount]] =
  findAllAWSAccountNumbers.local[CreateImageContext](_.wsClient)

val writePackerBuildConfig: (Bake, Path, Seq[AWSAccount]) => UseContext[Path] = 
  (bake, path, awsAccounts) =>
    createPackerBuildConfig(bake, path, awsAccounts)
      .local[CreateImageContext](_.packerConfig)

val runPacker: PackerInput => UseContext[ExitCode] = packerInput =>
  executePacker(packerInput).local[CreateImageContext](_.eventBus)

So, about that wiring...

Reader monad

Monad wrangling

What we want: ReaderT[F, Input, Output]

Reader monad

val bake = ... // generate using scalacheck-shapeless!

val reader = BuildService.createImage(
  createPlaybook = _ => Future.successful(Paths.get("playbook.yml")),
  findAllAWSAccounts = ...
  createPackerBuildConfig = ...
  executePacker = _ => ReaderT.pure[Future, EventBus, ExitCode](ExitCode(0))
)(bake)

val future = reader.run(context)

whenReady(future){ exitCode => exitCode should be(ExitCode(0)) }

I can now write a (quite cumbersome) test

Reader monad

Conclusions

• This was not a good use case for Reader
  • Could have injected deps using plain old currying
• In the real world, lift/pure/local wiring is painful
• Start with Reader, don't add it retroactively

Reader monad

by the way...

ReaderT[F, A, B] == Kleisli[F, A, B]

Reader[A, B] == Kleisli[Id, A, B]

So now you know what Kleisli means!

Free monad

... will have to wait for another talk

Summary

• Model your domain functionally
• Minimise boilerplate
• Separate description from execution
• There's more than one way to do so

Learning resources

• ReaderT 101 - Mind Candy blog
• A purely functional approach to building large applications - Noel Markham
• Dead-Simple Dependency Injection - Rúnar Bjarnason

Reader monad

Learning resources

• Compositional Application Architecture With Reasonably Priced Monads - Rúnar Bjarnason
• Cats Free monad documentation
• Why The Free Monad Isn't Free - Kelley Robinson

Free monad

Learning resources

General FP/other

• Functional Programming Patterns v3 - Raúl Raja Martínez
• Functional and Reactive Domain Modelling - Debasish Ghosh
• Functional Programming in Scala - Paul Chiusano and Rúnar Bjarnason