Functional Web Stack:

Http4s + Doobie

Alejandra Holman | Cristian Spinetta | Diego Parra

Outline

 

  • Why functional programming?

  • Doobie

  • Http4s - Server & Client

 

Functional Programming

"As software becomes more and more complex, it is more and more important to structure it well. Well-structured software is easy to write and to debug, and provides a collection of modules that can be reused to reduce future programming costs. In this paper we show that two features of functional languages in particular, higher-order functions and lazy evaluation, can contribute significantly to modularity ... We conclude that since modularity is the key to successful programming, functional programming offers important advantages for software development."

 

Why Functional Programming Matters?, by John Hughes

Side-Effect

It modifies the status of something external to the execution unit

Effect

Data Type with extra features

Functional Programming

def divide(dividend: Int,
           divisor: Int): Int = {
    if (divisor == 0)
      throw new RuntimeException(
        "Divisor can't be 0")
    dividend / divisor
  }
def divide(dividend: Int,
           divisor: Int):
      Either[String, Int] = {
    if (divisor == 0)
      Left("Divisor can't be 0")
    else
      Right(dividend / divisor)
  }

e.g. Exceptions, I/O operations, mutability, etc.

e.g. Option, Either, Task, Validated, etc.

  • Context-dependent
  • No composable
  • Hard to reason, refactor, test, mantain and so forth...
  • Referential transparency
  • Composable
  • Type safe
  • Separation of concerns

How can we program without side-effect's?

Functional Programming

... But what about the "real world"?

File system

Rest service

Database

My

program

Every expression is referentially transparent or it's a side effect

Description

Interpretation

Data type for console operations:

Example: Decimal to binary conversor

Functional Programming

sealed trait Console[A] { self =>

  def run: A

  def flatMap[B](f: A => Console[B]): Console[B] =
    new Console[B] { def run: B = f(self.run).run }

  def map[B](f: A => B): Console[B] =
    new Console[B] { def run: B = f(self.run) }

  // ...
}
abstract class ConsoleOps {
  def ReadLn: Console[Option[String]]
  def PrintLn(line: String): Console[Unit]
}

Program description:

Description

Interpretation

Functional Programming

class Program(ops: ConsoleOps) {

  def binaryConverter: Console[Unit] = {
    for {
      _    <- ops.PrintLn("Enter a positive integer:")
      line <- ops.ReadLn
      _    <- ops.PrintLn(response(line))
    } yield ()
  }

  private def response(line: Option[String]): String =
    line
      .flatMap(rawInput => stringToInt(rawInput))
      .map(decimalToBinary)
      .map(value => s"Binary representation: $value")
      .getOrElse(s"Wrong value introduced: $line")

  private def stringToInt(value: String): Option[Int] =
    Try(value.toInt).toOption

  private def decimalToBinary(decimal: Int): String =
    Integer.toBinaryString(decimal)
}

A useful interpreter:

Another interpreter:

Description

Interpretation

Functional Programming

object TerminalInterpreter extends ConsoleOps {
  def ReadLn: Console[Option[String]] =
    new Console[Option[String]] { def run = Option(StdIn.readLine()) }

  def PrintLn(line: String): Console[Unit] =
    new Console[Unit] { def run: Unit = println(line) }
}
case class TestingInterpreter(readValue: Option[String]) extends ConsoleOps {
  private val acc = mutable.ListBuffer.empty[String]

  def ReadLn: Console[Option[String]] =
    new Console[Option[String]] { def run: Option[String] = readValue }

  def PrintLn(line: String): Console[Unit] =
    new Console[Unit] { def run: Unit = acc += line }

  def extractValues: List[String] = acc.toList
}

Executing the program:

Description

Interpretation

Functional Programming

new Program(TerminalInterpreter).binaryConverter.run
object TerminalInterpreter extends ConsoleOps {
  def ReadLn: Console[Option[String]] =
    new Console[Option[String]] { def run = Option(StdIn.readLine()) }

  def PrintLn(line: String): Console[Unit] =
    new Console[Unit] { def run: Unit = println(line) }
}
class Program(ops: ConsoleOps) {

  def binaryConverter: Console[Unit] = {
    for {
      _    <- ops.PrintLn("Enter a positive integer:")
      line <- ops.ReadLn
      _    <- ops.PrintLn(response(line))
    } yield ()
  }

  // Implementation details...
}
sealed trait Console[A] { self =>

  def run: A

  // ...
}

Pros & Cons

  • Description => Interpretation

  • Pure functions

    • No side-effect

    • Easy to reason

    • Simple refactors and tests

    • Easy to compose and reuse

    • Lazy evaluation

    • Concurrency is more simple

  • Learning curve

  • Libs / frameworks maturity

Doobie

Doobie

"A pure functional JDBC layer for Scala. It is not an ORM, nor is it a relational algebra; it simply provides a principled way to construct programs (and higher-level libraries) that use JDBC."

https://github.com/tpolecat/doobie

 

 

libraryDependencies ++= Seq(
   "org.tpolecat" %% "doobie-core" % "0.5.3"
)

Programs as values: JDBC Programming with Doobie, by Rob Norris: https://www.youtube.com/watch?v=M5MF6M7FHPo

Transactor

import cats.effect.IO
import com.despegar.demo.conf.ConfigSupport
import com.despegar.demo.utils.ThreadUtils
import com.zaxxer.hikari.HikariDataSource
import doobie.util.transactor.Transactor

object DemoDS extends ConfigSupport {
  private[this] def dataSource: HikariDataSource = {
    val ds = new HikariDataSource
    ds.setPoolName("Demo-Hikari-Pool")
    ds.setMaximumPoolSize(config.datasource.maxPoolSize)
    ds.setDriverClassName(config.datasource.driverClassName)
    ds.setJdbcUrl(config.datasource.url)
    ds.addDataSourceProperty("user", config.datasource.username)
    ds.addDataSourceProperty("password", config.datasource.password)
    ds.addDataSourceProperty("connectTimeout", config.datasource.connectTimeout)
    ds.addDataSourceProperty("socketTimeout", config.datasource.socketTimeout)
    ds.setThreadFactory(ThreadUtils.namedThreadFactory("demo-hikari-pool"))
    ds
  }

  lazy val DemoTransactor: Transactor[IO] = {
    Transactor.fromDataSource[IO](dataSource)
  }
}

Query: definition & execution

import cats.effect.IO
import doobie.util.transactor.Transactor
import doobie.implicits._

class TxEmployeeStore() {

  def describeFindAllNames: ConnectionIO[List[String]] =
      sql"select name from Test.employee"   // Fragment
        .query[String]                      // Query0[String]
        .to[List]                           // ConnectionIO[List[String]]


  def executeFindAllNames(transactor: Transactor[IO]): List[String] =
      describeFindAllNames                  // ConnectionIO[List[String]]
        .transact(transactor)               // IO[List[String]]
        .unsafeRunSync()                    // List[String]

}

Magic mapping!

import com.despegar.demo.model._
import doobie._
import doobie.implicits._

class EmployeeStore() {

  def findAll: ConnectionIO[List[Employee]] =
    sql"select id, name, age, salary, start_date from Test.employee"
      .query[Employee]
      .to[List]
}
case class Employee(id: Option[Long],
                    name: String,
                    age: Option[Int],
                    salary: BigDecimal,
                    startDate: LocalDate)

Custom mapping...

import java.time.LocalDate
import com.despegar.demo.model._
import com.despegar.demo.store.DemoStore
import doobie._
import doobie.implicits._

class CompanyStore() extends DemoStore {

  def findCompanyWithStaff(companyId: Long): ConnectionIO[Option[Company]] = {
    val results = sql"""
         select c.id, c.name, e.id, e.name, e.age, e.salary, e.start_date
           from Test.company c
             inner join Test.employee e on e.company_id = c.id
           where c.id = $companyId
      """.query[(Long, String, Long, String, Option[Int], BigDecimal, LocalDate)].map {
      case (cId, cName, eId, eName, eAge, eSalary, eStartDate) =>

        val employee = Employee(Some(eId), eName, eAge, eSalary, eStartDate)
        val company = Company(companyId = Some(cId), name = cName)

        (company, employee)
    }.to[List]

    results.map(tuples => {
      val allEmployees: List[Employee] = tuples.map(_._2)
      tuples.headOption.map(tuple => tuple._1.copy(employees = allEmployees))
    })
  }
}

Fragments

def findByFilter(filter: EmployeeFilter, pageSize: Int = 100):ConnectionIO[List[Employee]] = {
  val idsCondition = NonEmptyList.fromList(filter.ids).map(ids => Fragments.in(fr"id", ids))
  val nameCondition: Option[Fragment] = filter.name.map(n => fr"name like $n")
  val ageCondition: Option[Fragment] = filter.minimumAge.map(age => fr"age >= $age")
  val fromCondition: Option[Fragment] = Some(fr"start_date >= ${filter.startDateFrom}")
  val toCondition: Option[Fragment] = Some(fr"start_date < ${filter.startDateTo}")

  val allConditions = Seq(idsCondition, nameCondition, ageCondition, fromCondition,toCondition)

  val q = fr"select id, name, age, salary, start_date from Test.employee" ++
        whereAndOpt(allConditions.toArray:_*) ++
        fr"limit $pageSize" ++
        filter.offset.map(off => fr"offset $off").getOrElse(Fragment.empty)

  q.query[Employee].to[List]
}
case class EmployeeFilter(name: Option[String],
                          minimumAge: Option[Int],
                          startDateFrom: LocalDate,
                          startDateTo: LocalDate,
                          offset: Option[Long],
                          ids: List[Long] = List())

Updates

Inserts

Batch operation

def happyBirthday(id: Long): ConnectionIO[Int] =
    fr"""update Test.employee set age = age + 1 where id = $id""".update.run
def save(name: String): ConnectionIO[Long] =
  fr"""insert into Test.company(name) values ($name)"""
    .update.withUniqueGeneratedKeys[Long]("id")
def saveAll(names: List[String]): ConnectionIO[Int] = {
  import cats.implicits._

  val sql = "insert into Test.company(name) values (?)"
  Update[String](sql).updateMany(names)
}

Transactions

def save(employee: Employee, companyId: Long): ConnectionIO[Long] =
    fr"""insert into Test.employee(name, age, salary, start_date, company_id)
         values (${employee.name},
                 ${employee.age},
                 ${employee.salary},
                 ${employee.startDate},
                 $companyId)"""
      .update.withUniqueGeneratedKeys[Long]("id")
def incrementStaff(companyId: Long): ConnectionIO[Int] =
    fr"""update Test.company set staff_count = staff_count + 1
           where id = $companyId""".update.run
import com.despegar.demo.model.{Company, Employee}
import com.despegar.demo.store.{CompanyStore, EmployeeStore}
import doobie.hi.ConnectionIO

class CompanyProgram(companyStore: CompanyStore, employeeStore: EmployeeStore) {

  def hire(companyId: Long, employee: Employee): ConnectionIO[Long] =
    for {
      employeeId <- employeeStore.save(employee, companyId)
      _ <- companyStore.incrementStaff(companyId)
    } yield employeeId
}

Logging

implicit val han: LogHandler = LogHandler {
    case Success(s, a, e1, e2) =>
      if (debugEnabled)
        log.debug(s"""...""")

    case ProcessingFailure(s, a, e1, e2, t) =>
      log.error(s"""...""")

    case ExecFailure(sql, args, elapsed, exc) =>
      log.error(s"""...""", exc)
  }

  • Transactions

  • Queries and times logging

  • Dynamic queries

  • Documentation and support

  • Streaming support

  • Hard mapping for some queries

  • It doesn't support timeout by query

Pros & Cons

Http4s

Http4s

 

"Typeful, functional, streaming HTTP for Scala"

http://http4s.org

 

 

libraryDependencies ++= Seq(
   "org.http4s" %% "http4s-blaze-server" % "0.18.11",

   "org.http4s" %% "http4s-blaze-client" % "0.18.11",

   "org.http4s" %% "http4s-circe"        % "0.18.11",

   "org.http4s" %% "http4s-dsl"          % "0.18.11"   
)

Server

object Server extends StreamApp[IO] with Stores with Programs with ClientFactory {

  val executor = Executors.newFixedThreadPool(30, namedThreadFactory("demo-server-pool"))
  implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(executor)

  override def stream(args: List[String], requestShutdown: IO[Unit]): Stream[IO, ExitCode] = {

    def router(): HttpService[IO] = Router[IO](
      "/demo/employee" -> EmployeeService(transactor).service(employeeProgram),
      "/demo/company" -> CompanyService(transactor).service(companyProgram),
      "/" -> HealthService().service()
    )

    for {
      exitCode <- BlazeBuilder[IO]
        .bindHttp(9290, "localhost")
        .mountService(router)
        .serve
    } yield exitCode

  }
}
trait Stores {
  val trans = DemoDS.DemoTransactor

  val txCompanyStore = new TxEmployeeStore(trans)
  // more stores...
}
trait Programs extends Stores with Clients {

  val companyProgram = new CompanyProgram(companyStore, employeeStore)
  val employeeProgram = new EmployeeProgram(employeeStore)

}

Service: Get

import cats.effect.IO
import com.despegar.demo.program.EmployeeProgram
import doobie._
import doobie.implicits._
import io.circe.syntax._
import org.http4s._
import org.http4s.dsl.io._

class EmployeeService(xa: Transactor[IO]) {
  import com.despegar.demo.model.Employee._
  import com.despegar.demo.utils.CirceUtils.circeCustomSyntax._

  def service(employeeProgram: EmployeeProgram): HttpService[IO] = {

    def route = HttpService[IO] {
      case GET -> Root / "list" => handleGetAll
    }

    def handleGetAll: IO[Response[IO]] =
      employeeProgram.findAll.transact(xa).attempt.flatMap {
        case Right(employeeList) => Ok(employeeList.asJson)
        case Left(cause) => InternalServerError("Error getting all employees")
      }

    route
  }
}

object EmployeeService {
  def apply(xa: Transactor[IO]): EmployeeService = new EmployeeService(xa)
}
class EmployeeStore() {

  def findAll: ConnectionIO[List[Employee]] =
    sql"""select id, name, age, salary, start_date 
        from Test.employee"""
      .query[Employee]
      .to[List]
}

Middleware

import cats.effect.IO
import com.despegar.demo.program.EmployeeProgram
import doobie._
import doobie.implicits._
import io.circe.syntax._
import org.http4s._
import org.http4s.dsl.io._
import org.http4s.server.middleware.GZip

class EmployeeService(xa: Transactor[IO]) {
  import com.despegar.demo.model.Employee._
  import com.despegar.demo.utils.CirceUtils.circeCustomSyntax._

  def service(employeeProgram: EmployeeProgram): HttpService[IO] = GZip {

    def route = HttpService[IO] {
      case GET -> Root / "list" => handleGetAll
    }

    def handleGetAll: IO[Response[IO]] =
      employeeProgram.findAll.transact(xa).attempt.flatMap {
        case Right(employeeList) => Ok(employeeList.asJson)
        case Left(cause) => InternalServerError("Error getting all employees")
      }
    route
  }
}

object EmployeeService {
  def apply(xa: Transactor[IO]): EmployeeService = new EmployeeService(xa)
}

Matching url & params

/demo/employee/6

class EmployeeService(xa: Transactor[IO]) extends LogSupport {
  
  import com.despegar.demo.api.EmployeeService._

  def service(employeeProgram: EmployeeProgram): HttpService[IO] = {
    
    def route = HttpService[IO] {
  
      case GET -> Root / LongVar(id) => handleFindById(id)
      // case GET -> ....

    }

    // def handleFindById(id: Long) = ...

    route
  }
}

object EmployeeService extends LogSupport {
  def apply(xa: Transactor[IO]): EmployeeService = new EmployeeService(xa)
}

Matching url & params

/demo/employee/filter/2015-01-01/2018-02-01?name=Pepe&minimum-age=25&offset=0

object EmployeeService extends LogSupport {
  def apply(xa: Transactor[IO]): EmployeeService = new EmployeeService(xa)

  object NameQueryParamMatcher extends OptionQueryParamDecoderMatcher[String]("name")
  object MinimumAgeQueryParamMatcher extends OptionQueryParamDecoderMatcher[Int]("minimum-age")
  object OffsetMatcher extends OptionQueryParamDecoderMatcher[Long]("offset")

  object LocalDateVar {
    def unapply(str: String): Option[LocalDate] = 
      if(!str.isEmpty) Try(LocalDate.parse(str)).toOption else None
  }
}
case GET -> Root / "filter" / LocalDateVar(startDateFrom) / LocalDateVar(startDateTo) :?
            NameQueryParamMatcher(name) +&
            MinimimAgeQueryParamMatcher(minimumAge) +&
            OffsetMatcher(offset ) =>
              handleFindByFilter(
                 EmployeeFilter(name, minimumAge, startDateFrom, startDateTo, offset)
              )

Service: Post

curl -v -X POST -H "Content-type: application/json" -d '{"name":"Juan","salary":12000}' 'http://localhost:9290/demo/company/hire/9'

class CompanyService(xa: Transactor[IO]) {

  import com.despegar.demo.model.Employee._

  def service(companyProgram: CompanyProgram): HttpService[IO] = {

    def route = HttpService[IO] {
      case req@POST -> Root / "hire" / LongVar(companyId) => handleHire(req, companyId)
    }

    def handleHire(request: Request[IO], companyId: Long) =
      request.decodeWith(jsonOf[IO, Employee], strict = true) { newEmployee => 
        companyProgram.hire(companyId, newEmployee). transact(xa).attempt.flatMap {
          case Right(employeeId) => Ok()
          case Left(cause) => InternalServerError("Error saving new employee")
        }
      }

    route
  }
}

object CompanyService extends LogSupport {
  def apply(xa: Transactor[IO]): CompanyService = new CompanyService(xa)
}

HttpClient

object Server extends StreamApp[IO] with Stores with Programs with ClientFactory {

  val executor = Executors.newFixedThreadPool(30, namedThreadFactory("demo-server-pool"))
  implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(executor)

  override def stream(args: List[String], requestShutdown: IO[Unit]): Stream[IO, ExitCode] = {
    def router(client: Client[IO]): HttpService[IO] = Router[IO]( ... )  

    for {
      client   <- httpClient
      exitCode <- BlazeBuilder[IO]
        .bindHttp(9290, "localhost")
        .mountService(router(client))
        .serve
    } yield exitCode
  }
}


trait ClientFactory extends ConfigSupport {

  private val clientConfig = BlazeClientConfig.defaultConfig.copy(
    maxTotalConnections = config.client.maxTotalConnections,
    idleTimeout = config.client.idleTimeout,
    requestTimeout = config.client.requestTimeout
  )

  def httpClient: Stream[IO, Client[IO]] = Http1Client.stream[IO](clientConfig)
}
trait Clients {
  def companyClient(client: Client[IO]): CompanyClient = 
    CompanyClient(client)
}

trait Programs extends Stores with Clients {

  def companyProgram(client: Client[IO]) = 
    new CompanyProgram(companyStore, 
                       employeeStore, 
                       companyClient(client)
                       )
}

Client: Get

case class CompanyClient(httpClient: Client[IO]) extends Http4sClientDsl[IO] {

  def getById(companyId: Long): IO[Option[Company]] = {
    httpClient.expect(s"http://localhost:9290/demo/company/$companyId")
                     (jsonOf[IO, Option[Company]])
  }

}

  • Lightweight

  • Middleware

  • Streaming support

  • All basics: Cookies, headers, circe integration, etc

  • Wrong urls

Pros & Cons

Resources

Thank you!

We are hiring!     

  http://www.despegar.com/sumate/#

  IT Talent Acq: maria.arean@despegar.com 

Functional Web Stack

By Cristian Spinetta

Source: María Alejandra Holman

Functional Web Stack

Functional programming in Scala with Http4s + Doobie. The goals of the talk are: A brief tour of the most important ideas and concepts of functional programming; Show the motivations to use functional programming in implementing business logic; Apply functional progrmaming in a real environment, using pure functional Scala libs: Http4s and Doobie.

  • 821

More from Cristian Spinetta