Real-world

functional Scala

Chris Birchall

1st Oct 2016

OOP

time

Java dev

"better Java"

discover

Scala

rediscover

Case study: AMIgo

Self-serve AMI bakery
Implemented in Scala + Play
Very impure/effect-heavy domain
- File I/O, external processes, DynamoDB, ...
Event bus
Not many tests

https://github.com/guardian/amigo

DEMO

Package structure

Ansible stuff

Play controllers

DynamoDB DAOs

Event bus stuff

Domain models

Packer stuff

Guardian-specific stuff

Scheduler stuff

Play view templates

Play DI stuff

What went well?

Case study: AMIgo

What could be improved?

What AMIgo got right

Functional modelling
Prefer objects to classes
Avoid traits
No lazy vals
Minimise boilerplate

Functional modelling

Preamble

OOP design

Data

Behaviour

trait Recipe {
  def generatePackerConfig(): File
}
trait ProcessRunner {
  def run(cmd: Seq[String]): Future[Int]
}

class BakeImpl(
  id: BakeId,
  buildNumber: Int,
  recipe: Recipe,
  processRunner: ProcessRunner) extends Bake {

  def execute(): Future[Int] = {
    val packerFile = 
      recipe.generatePackerConfig()
    val packerCommand = 
      buildPackerCommand(packerFile, buildNumber)
    processRunner.run(packerCommand)
  }
  
  ...
}

Data

Behaviour

Decouple data from behaviour

FP design

Model your domain using

Algebraic Data Types (ADTs)

Data

Sum type

A or B or ...
Represented in Scala using sealed trait/abstract class

sealed abstract class BakeStatus

case object Running extends BakeStatus
case object Complete extends BakeStatus
case object Failed extends BakeStatus

Product type

A and B and ...
Represented in Scala using case class

case class Bake(
    recipe: Recipe,
    buildNumber: Int,
    status: BakeStatus,
    amiId: Option[AmiId],
    startedBy: String,
    startedAt: DateTime
)

Use the type system to your advantage

Data

case class AmiId(value: String) extends AnyVal
case class BaseImageId(value: String) extends AnyVal
case class RecipeId(value: String) extends AnyVal
case class RoleId(value: String) extends AnyVal
case class BakeId(recipeId: RecipeId, buildNumber: Int)

Value class: no runtime overhead

Use the type system to your advantage

Data

object Bakes {

  def updateAmiId(bakeId: BakeId, amiId: AmiId): Unit = ...

}

val bakeId = BakeId(RecipeId("my-recipe"), buildNumber = 123)
val amiId = AmiId("ami-a1b2c3d4")

Bakes.updateAmiId(amiId, bakeId) // yay, compile error!

Model your errors

Data

def loadRoleFromDisk(roleId: RoleId): Role = {
  ...
  if (dirNotFound)
    throw new IOException("Role directory not found")
  ...
  if (invalid)
    throw new InvalidRoleException("check ur role!")
  ...
}

def loadRoleFromDisk(roleId: RoleId):
    Either[IOException,
           Either[InvalidRoleException,
                  Role]] = ...

Nope!

Hmm...

Model your errors

Data

sealed trait LoadRoleResult

object LoadRoleResult {
  case class Success(role: Role) extends LoadRoleResult
  case object RoleDirectoryNotFound extends LoadRoleResult
  case object RoleFormatInvalid extends LoadRoleResult
}

def loadRoleFromDisk(roleId: RoleId): LoadRoleResult = ...

Downside: don't get map/flatMap for free

Model your errors

Data

sealed trait LoadRoleError

object LoadRoleError {
  case object RoleDirectoryNotFound extends LoadRoleError
  case object RoleFormatInvalid extends LoadRoleError
}

def loadRoleFromDisk(roleId: RoleId): Either[LoadRoleError, Role] = ...

Keep an eye out for monoids

Data

TODO

This gives you a bunch of stuff for free

e.g. foldMap

case class Result(successes: Int, failures: Int)

implicit val rm = new Monoid[Result] {
  def empty = Result(0, 0)
  def combine(x: Result, y: Result) = Result(
    successes = x.successes + y.successes,
    failures = x.failures + y.failures
  )
}

Keep an eye out for monoids

Data

TODO

Maps the function over the list and combines the results

def processPageOfItems(page: Page[Item]): Result = ...
val pages: List[Page] = ...

import cats.instances.list._
import cats.syntax.foldable._

val overallResult: Result = 
  pages.foldMap(processPageOfItems)

Behaviour

Data

ADTs
Strongly typed
Model all the things

Model behaviour as plain old functions

Behaviour

object PlaybookGenerator {

  def renderRole(role: CustomisedRole): String = ...

  def generatePlaybook(recipe: Recipe): Yaml = {
    val allRoles = recipe.baseImage.builtinRoles ++ recipe.roles
    val roleStrings = allRoles.map(renderRole)

    Yaml(...)
  }

  ...
}

Larger functions composed of smaller ones

Behaviour

Function hierarchy

Data

ADTs
Strongly typed
Model all the things

Some functions are purer than others

Behaviour

object PackerRunner {

  def createImage(bake: Bake): Future[ExitCode] = {
    val playbookYaml = 
      PlaybookGenerator.generatePlaybook(bake.recipe)
    val playbookFile = writeToTempFile(playbookYaml)

    val packerBuildConfig = PackerBuildConfigGenerator
      .generatePackerBuildConfig(bake)
    val packerConfigFile = writeToTempFile(packerBuildConfig)

    executePacker(bake, playbookFile, packerConfigFile)
  }

  ...
}

Pure 😇

Impure 🙈

Behaviour

Pure

functions

Effects

Data

ADTs
Strongly typed
Model all the things

What AMIgo got right

Functional modelling
Prefer objects to classes
Avoid traits
No lazy vals
Minimise boilerplate

Prefer objects to classes

vs

object PackerRunner {

  def createImage(bake: Bake): Future[ExitCode] = {
    ...
  }

}

class PackerRunner(bake: Bake) {

  def createImage(): Future[ExitCode] = {
    ...
  }

}

Prefer objects to classes

Benefits of objects

Singleton
- No lifecycle to worry about
Principle of least power
- No state, just a bunch of functions

Prefer objects to classes

Problem: You end up passing lots of params around

Partial solution: Pass them implicitly

If this gets excessive, bundle into a single context class

(c.f. ScalaCache)

object Recipes {

  def list()(implicit dynamo: Dynamo): Iterable[Recipe] = {
    ...
  }
 
  ...
}

class RecipeController(...)(implicit dynamo: Dynamo) 
    extends Controller {

  def listRecipes = Action {
    Ok(views.html.recipes(Recipes.list()))
  }

  ...
}

Caveat: Daniel Spiewak calls this an anti-pattern

... but it's fine if resource lifecycle == app lifecycle

Prefer objects to classes

Problem: how to do Dependency Injection?

Supplementary question: why do we do DI?

Dependency injection

Why do we do DI?

Mocking dependencies so we can test in isolation
Separating business logic from resource management

OOP

FP

DAG of dependencies

Function composition

Dependency injection

1. Mocking dependencies

Pass functions as parameters

case class PackerInput(bake: Bake, playbookFile: Path, packerConfigFile: Path)

def createImage(bake: Bake,
  createPlaybook: Recipe => Path,
  createPackerConfig: Bake => Path,
  runPacker: PackerInput => Future[ExitCode]): Future[ExitCode] = {

  val playbookFile = createPlaybook(bake.recipe)
  val packerConfigFile = createPackerConfig(bake)
  runPacker(PackerInput(bake, playbookFile, packerConfigFile))
}

Dependency injection

2. Resource management

Bad:

Better:

def getItem(id: Int): Option[Item] = {
  val conn = new DBConnection()
  try {
    ... 
  } finally { 
    conn.close()
  }
}

def getItem(id: Int, 
            conn: DBConnection): Option[Item] = {
  ...
}

Dependency injection

2. Resource management

Even better:

def getItem(id: Int) = { (conn: DBConnection) => 
  ...
}

val run: DBConnection => Item = getItem(123)
val item: Item = run(new DBConnection())

Curried function

Pure
Separates description from execution
Composable (we'll come back to this)

What AMIgo got right

Functional modelling
Prefer objects to classes
Avoid traits
No lazy vals
Minimise boilerplate

Avoid traits

Traits are attractive to OOP devs

Multiple inheritance, diamond inheritance
Mixin = exciting inheritance/composition hybrid
Can be used like a Java/C# interface
Support fancy patterns, e.g. abstract override

Avoid traits

Problems with traits

Initialisation order gotchas
- "NullPointerException?! I thought Scala didn't have nulls!"
- Hack around with lazy vals (or, shudder, early init)
Encourages inheritance, overriding, OOP design

Avoid traits

Corollary: no cake!

What AMIgo got right

Functional modelling
Prefer objects to classes
Avoid traits
No lazy vals
Minimise boilerplate

No lazy vals

Runtime overhead
Risk of deadlock
Cognitive overhead
- Initialisation of lazy val ≈ side-effect

object HandyService {
  lazy val usefulData = expensiveComputation()
}

object MyCode {

  def performanceCriticalCode() = {
    val foo = HandyService.usefulData
    ...
  }

}

Is this OK?

Has somebody already taken the performance hit for us?

What AMIgo got right

Functional modelling
Prefer objects to classes
Avoid traits
No lazy vals
Minimise boilerplate

Data

ADTs
Strongly typed
Model all the things

Behaviour

Pure

functions

Effects

Minimise boilerplate

Glue, wiring, boilerplate

Minimise boilerplate

Scanamo

One line to write arbitrary case class to DynamoDB
Codec is automatically derived using shapeless

object Bakes {

  def create(...)(implicit dynamo: Dynamo): Bake = {
    val bake = Bake(...)
    val dbModel = ...
    Scanamo.put(dynamo.client)(tableName)(dbModel)
    bake
  }

  ...
}

Minimise boilerplate

Automagic

Transforms between domain models and DB models

object Bake {

  case class DbModel(
    recipeId: RecipeId,
    buildNumber: Int,
    status: BakeStatus,
    amiId: Option[AmiId],
    startedBy: String,
    startedAt: DateTime)

  import automagic._

  def domain2db(bake: Bake): DbModel =
    transform[Bake, Bake.DbModel](bake, "recipeId" -> bake.recipe.id)

  def db2domain(dbModel: DbModel, recipe: Recipe): Bake =
    transform[Bake.DbModel, Bake](dbModel, "recipe" -> recipe)

}

Minimise boilerplate

Enumeratum

Makes sealed classes work like enums

sealed abstract class BakeStatus extends EnumEntry

object BakeStatus extends Enum[BakeStatus] {

  val values = findValues

  case object Running extends BakeStatus
  case object Complete extends BakeStatus
  case object Failed extends BakeStatus

}

val running: BakeStatus = BakeStatus.withName("Running")

What could be improved

Separation of pure and impure code
- Try the Reader monad for DI
Separate description from execution
- Try the Free monad

Measures of success

More of the code can be tested
No loss of readability/understandability
- Who is the target audience?

Reader monad

Wraps a function f: A => B

scala> import cats.data.Reader
import cats.data.Reader

scala> val f: Int => String = _.toString
f: Int => String = <function1>

scala> val reader = Reader(f)
reader: cats.data.Reader[Int,String] = ...

scala> reader.run(123)
res1: cats.Id[String] = 123

umm... why?

Reader monad

Recap: DI via currying

def getItem(id: Int) = { (conn: DBConnection) => 
  ...
}

val run: DBConnection => Item = getItem(123)
val item: Item = run(new DBConnection())

Reader monad

Composition with currying

def getItem(id: Int): DBConnection => Item = { 
  (conn: DBConnection) =>
    // read from DB
    Item(id, "widget")
}

def renameItem(item: Item, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    val item = getItem(id)(conn)
    renameItem(item, newName)(conn)
}

val item: Item = renameItemWithId(123, "hello")(new DBConnection())

Reader monad

Composition with Reader

import cats.data.Reader
 
def getItem(id: Int) = Reader[DBConnection, Item]{ 
  conn =>
    // read from DB
    Item(id, "widget")
}

def renameItem(item: Item, newName: String) = Reader[DBConnection, Item]{ 
  conn =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): Reader[DBConnection, Item] = {
  for {
    item <- getItem(id)
    updatedItem <- renameItem(item, newName)
  } yield updatedItem
}

val result: Item = renameItemWithId(123, "hello").run(new DBConnection())

Reader monad

Slightly more realistic example

def getItem(id: Int): DBConnection => Option[Item] = { 
  (conn: DBConnection) =>
    // read from DB
    Some(Item(id, "widget"))
}

def renameItem(item: Item, newName: String): DBConnection => Item = { 
  (conn: DBConnection) =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): DBConnection => Option[Item] = { 
  (conn: DBConnection) =>
    val item = getItem(id)(conn)
    item.map(x => renameItem(x, newName)(conn))
}

val item: Option[Item] = renameItemWithId(123, "hello")(new DBConnection())

Reader monad

Again with Reader

import cats.data.Reader
 
def getItem(id: Int) = ReaderT[Option, DBConnection, Item]{ 
  conn =>
    // read from DB
    Some(Item(id, "widget"))
}

def renameItem(item: Item, newName: String) = Reader[DBConnection, Item]{ 
  conn =>
    // save to DB
    item.copy(name = newName)
}

def renameItemWithId(id: Int, newName: String): ReaderT[Option, DBConnection, Item] = {
  for {
    item <- getItem(id)
    updatedItem <- renameItem(item, newName).lift[Option]
  } yield updatedItem
}

val result: Option[Item] = renameItemWithId(123, "hello").run(new DBConnection())

Reader monad

How would this look in AMIgo?

def createImage(bake: Bake, prism: Prism, eventBus: EventBus)
               (implicit packerConfig: PackerConfig): Future[ExitCode] = {
  val playbookYaml = PlaybookGenerator.generatePlaybook(bake.recipe)
  val playbookFile = TempFiles.writePlaybookToFile(playbookYaml, bake.recipe.id)

  prism.findAllAWSAccountNumbers() flatMap { awsAccountNumbers =>
    Logger.info(s"AMI will be shared with the following AWS accounts: $awsAccountNumbers")
    val packerBuildConfig = PackerBuildConfigGenerator.generatePackerBuildConfig(
      bake, playbookFile, awsAccountNumbers)
    val packerConfigFile = 
       TempFiles.writePackerConfigToFile(packerBuildConfig, bake.recipe.id)

    PackerRunner.executePacker(bake, playbookFile, packerConfigFile, eventBus)
  }
}

Starting point

Reader monad

def createImage(
  createPlaybook: Recipe => Path,
  findAllAWSAccounts: WSClient => Future[Seq[AWSAccount]],
  createPackerBuildConfig: (Bake, Path, Seq[AWSAccount], PackerConfig) => Path,
  executePacker: (Bake, Path, Path, EventBus) => Future[ExitCode])
  (bake: Bake, 
  eventBus: EventBus, 
  wSClient: WSClient, 
  packerConfig: PackerConfig): Future[ExitCode] = {

  val playbookFile = createPlaybook(bake.recipe)

  findAllAWSAccounts(wSClient) flatMap { awsAccounts =>
    Logger.info(s"AMI will be shared with the following AWS accounts: $awsAccounts")

    val packerConfigFile = 
      createPackerConfigToFile(bake, playbookFile, awsAccounts, packerConfig)

    PackerRunner.executePacker(bake, playbookFile, packerConfigFile, eventBus)
  }
}

Pass functions as arguments

Reader monad

// In AppComponents.scala

val createImage: (Bake, EventBus, WSClient, PackerConfig) = BuildService.createImage(
  createPlaybook = PlaybookGenerator.generatePlaybook andThen TempFiles.writeToFile,
  findAllAWSAccounts = ...,
  ...
) _

Use currying to hide wiring

// In controller

val bake = Bakes.create(recipe, buildNumber, startedBy = request.user.fullName)
createImage(bake, eventBus, wsClient, packerConfig)

Reader monad

class CreateImageContext(
  eventBus: EventBus, 
  wSClient: WSClient, 
  packerConfig: PackerConfig
)

def createImage(
  createPlaybook: Recipe => Path,
  findAllAWSAccounts: WSClient => Future[Seq[AWSAccount]],
  createPackerBuildConfig: (Bake, Path, Seq[AWSAccount], PackerConfig) => Path,
  executePacker: (Bake, Path, Path, EventBus) => Future[ExitCode])
  (bake: Bake)(context: CreateImageContext): Future[ExitCode] = {

  ...
}

Bundle dependencies into a context

Reader monad

def createImage(
  createPlaybook: Recipe => Future[Path],
  findAllAWSAccounts: ReaderT[Future, WSClient, Seq[AWSAccount]],
  createPackerBuildConfig: 
    (Bake, Path, Seq[AWSAccount]) => ReaderT[Future, PackerConfig, Path],
  executePacker: (Bake, Path, Path) => ReaderT[Future, EventBus, ExitCode])
  (bake: Bake): ReaderT[Future, CreateImageContext, ExitCode] = {

  ...
}

Reader-ify

Any function that took one of the dependencies as an argument now returns a Reader (or ReaderT)

Reader monad

def createImage(
  ...
  )
  (bake: Bake): ReaderT[Future, CreateImageContext, ExitCode] = {

  ...
  ... wiring to turn all the params into ReaderT[Future, CreateImageContext, A]
  ...

  for {
    playbookFile <- writePlaybook(playbookYaml)
    accounts <- findAccounts
    _ = Logger.info(s"AMI will be shared with the following AWS accounts: $accounts")
    packerConfigFile <- writePackerBuildConfig(bake, playbookFile, accounts)
    exitCode <- runPacker(PackerInput(bake, playbookFile, packerConfigFile))
  } yield {
    exitCode
  }
}

Compose the readers

Reader monad

type UseContext[A] = ReaderT[Future, CreateImageContext, A]

val writePlaybook: Recipe => UseContext[Path] = recipe =>
  ReaderT.lift[Future, CreateImageContext, Path](createPlaybook(recipe))

val findAccounts: UseContext[Seq[AWSAccount]] =
  findAllAWSAccountNumbers.local[CreateImageContext](_.wsClient)

val writePackerBuildConfig: (Bake, Path, Seq[AWSAccount]) => UseContext[Path] = 
  (bake, path, awsAccounts) =>
    createPackerBuildConfig(bake, path, awsAccounts)
      .local[CreateImageContext](_.packerConfig)

val runPacker: PackerInput => UseContext[ExitCode] = packerInput =>
  executePacker(packerInput).local[CreateImageContext](_.eventBus)

So, about that wiring...

Reader monad

Monad wrangling

What we want: ReaderT[F, Input, Output]

What we have	Necessary wrangling
a: Output	ReaderT.pure[F, Input, Output](a)
fa: F[Output]	ReaderT.lift[F, Input, Output](fa)
r: Reader[Input, Output]	r.lift[F]
r: ReaderT[F, X, Output]	r.local[Input](_.x)

Reader monad

val bake = ... // generate using scalacheck-shapeless!

val reader = BuildService.createImage(
  createPlaybook = _ => Future.successful(Paths.get("playbook.yml")),
  findAllAWSAccounts = ...
  createPackerBuildConfig = ...
  executePacker = _ => ReaderT.pure[Future, EventBus, ExitCode](ExitCode(0))
)(bake)

val future = reader.run(context)

whenReady(future){ exitCode => exitCode should be(ExitCode(0)) }

I can now write a (quite cumbersome) test

Reader monad

Conclusions

This was not a good use case for Reader
- Could have injected deps using plain old currying
In the real world, lift/pure/local wiring is painful
Start with Reader, don't add it retroactively

Reader monad

by the way...

ReaderT[F, A, B] == Kleisli[F, A, B]

Reader[A, B] == Kleisli[Id, A, B]

So now you know what Kleisli means!

Free monad

... will have to wait for another talk

Summary

Model your domain functionally
Minimise boilerplate
Separate description from execution
There's more than one way to do so

Learning resources

ReaderT 101 - Mind Candy blog
A purely functional approach to building large applications - Noel Markham
Dead-Simple Dependency Injection - Rúnar Bjarnason

Reader monad

Learning resources

Compositional Application Architecture With Reasonably Priced Monads - Rúnar Bjarnason
Cats Free monad documentation
Why The Free Monad Isn't Free - Kelley Robinson

Free monad

Learning resources

General FP/other

Functional Programming Patterns v3 - Raúl Raja Martínez
Functional and Reactive Domain Modelling - Debasish Ghosh
Functional Programming in Scala - Paul Chiusano and Rúnar Bjarnason

Real-world

functional Scala

Case study: AMIgo

Self-serve AMI bakery

Implemented in Scala + Play

Very impure/effect-heavy domain

Event bus

Not many tests

https://github.com/guardian/amigo

DEMO

Package structure

What went well?

Case study: AMIgo

What could be improved?

What AMIgo got right

Functional modelling

Prefer objects to classes

Avoid traits

No lazy vals

Minimise boilerplate

Functional modelling

Preamble

OOP design

Decouple data from behaviour

FP design

Model your domain using

Algebraic Data Types (ADTs)

Data

Sum type

Product type

Use the type system to your advantage

Data

Use the type system to your advantage

Data

Model your errors

Data

Model your errors

Data

Model your errors

Data

Keep an eye out for monoids

Data

Keep an eye out for monoids

Data

Model behaviour as plain old functions

Behaviour

Some functions are purer than others

Behaviour

What AMIgo got right

Functional modelling

Prefer objects to classes

Avoid traits

No lazy vals

Minimise boilerplate

Prefer objects to classes

vs

Prefer objects to classes

Benefits of objects

Prefer objects to classes

Prefer objects to classes

Dependency injection

Why do we do DI?

OOP

FP

Dependency injection

1. Mocking dependencies

Dependency injection

2. Resource management

Dependency injection

2. Resource management

What AMIgo got right

Functional modelling

Prefer objects to classes

Avoid traits

No lazy vals

Minimise boilerplate

Avoid traits

Traits are attractive to OOP devs

Avoid traits

Problems with traits