ducktape

holding Scala's productivity together

Aleksander Rainko

agenda

a quick rant about JSON APIs
what is ducktape?
IDLs & codegen
putting all of the above together

the story

Moving JSON from socket A to socket B pays the bills.

It's also really, really mundane.

create the route definition with a DSL of your choice
actually model the request/response model of the definition above
somehow transform the API model to your domain model (that usually means validation, parsing, sanitization etc.)
document the contract you just created for other teams to actually use (and remember not to break it!)
apply the business sauce

but what if we didn't actually have to?

document the contract you just created for other teams to actually use (and remember not to break it!)
apply the business sauce

what is ducktape?

a macro-based library for automatic and configurable transformations
heavily inspired by chimney
a dotty exclusive

what is it made of?

trait Transformer[Source, Dest] {
  def transform(value: Source): Dest
}

Transformer is the primitive that everything else is built out of
instances for the most common transformations available out of the box
automatic derivation for case classes and enums

import io.github.arainko.ducktape.*

case class Talk(name: String, pitch: String, presenter: Presenter)
case class Presenter(name: String, bio: String)

case class Talk2(pitch: String, presenter: Presenter2, name: String)
case class Presenter2(bio: String, name: String)

val talk: Talk = ???
val talk2: Talk2 = talk.to[Talk2]

val talk2: Talk2 = to[Talk](talk)[Talk2](
  inline$make$i1[Talk, Talk2](ForProduct)(
    (
      (source: Talk) =>
        new Talk2(
          pitch = source.pitch,
          presenter = new Presenter2(
            bio = source.presenter.bio,
            name = source.presenter.name
          ),
          name = source.name
        )
    ): Transformer[Talk, Talk2]
  ): ForProduct[Talk, Talk2]
)

enum EventType {
  case OnSite, Online
}

enum ConferenceType {
  case OnSite, Online, Hybrid
}

val confType = EventType.OnSite.to[ConferenceType]

val confType = to[EventType](EventType.OnSite)[ConferenceType] {
  val x$1$proxy1: Mirror.SumOf[EvenType] = ...
  val x$2$proxy1: Mirror.SumOf[ConferenceType] = ...

  inline$make$i2[EventType, ConferenceType](ForCoproduct)(
    (
      (source: EventType) =>
        if (source.isInstanceOf[OnSite.type]) OnSite
        else if (source.isInstanceOf[Online.type]) Online
        else throw new RuntimeException("Unhandled condition encountered during Coproduct Transformer derivation")
    ): Transformer[EventType, ConferenceType]
  ): ForCoproduct[EventType, ConferenceType]
}

but what if it doesn't perfectly match?

case class Person(firstName: String, lastName: String)
case class SeriousPerson(firstName: String, lastName: String, SSN: String)

val person: Person = ???
// compiletime error: No field named 'SSN' found in Person
val seriousPerson: SeriousPerson = person.to[SeriousPerson]

Yeah, it's not all lollipops and crisps - thankfully we've got this covered:

val seriousPerson: SeriousPerson = 
  person
    .into[SeriousPerson]
    .transform(
      // SSN is set to 'ConstSSN'
      Field.const(_.SSN, "ConstSSN"),
      // SSN is set to the concatenation of 'firstName' and "SSN"
      Field.computed(_.SSN, _.firstName + "SSN"),
      // SSN is taken from another field, 'lastName' in this case
      Field.renamed(_.SSN, _.lastName)
    )

okay, but what if the constructor is private?

case class Person(firstName: String, lastName: String)

case class SeriousPerson private (
  firstName: String, lastName: String, SSN: String
)
 
object SeriousPerson {
  def create(
    firstName: String, lastName: String, SSN: String
  ): Either[String, SeriousPerson] = ???
}

val person: Person = ???

//compiletime error: No field named 'SSN' found in Person
val seriousPerson: Either[String, SeriousPerson] = 
  person.via(SeriousPerson.create)

val seriousPerson: Either[String, SeriousPerson]  = 
  person
    .intoVia(SeriousPerson.create)
    .transform(
      // SSN is set to 'ConstSSN'
      Arg.const(_.SSN, "ConstSSN"),
      // SSN is set to the concatenation of 'firstName' and "SSN"
      Arg.computed(_.SSN, _.firstName + "SSN"),
      // SSN is taken from another field, 'lastName' in this case
      Arg.renamed(_.SSN, _.lastName)
    )

Believe it or not, that's also alright.

what about the case where a case class is made of fields that have private constructors that return Eithers or Options?

Once again, yes... Sort of - we just have to put in some additional work.

enter fallible transformers

trait FallibleTransformer[F[+x], Source, Dest] {
  def transform(value: Source): F[Dest]
}

sealed trait Mode[F[+x]] {
  def pure[A](value: A): F[A]
  def map[A, B](fa: F[A], f: A => B): F[B]
}

object Mode {
  trait Accumulating[F[+x]] extends Mode[F] {
    def product[A, B](fa: F[A], fb: F[B]): F[(A, B)]
  }
  
  trait FailFast[F[+x]] extends Mode[F] {
    def flatMap[A, B](fa: F[A], f: A => F[B]): F[B]
  }
}

a quick example

case class Person(name: String, age: Int)
case class SeriousPerson(name: String, age: Age, SSN: SSN)

case class Age private (value: Int)
object Age {
  export transformer.transform as make
  given transformer: Transformer.Fallible[Either[List[String], _], Int, Age] = 
    value => Either.cond(value > 0, Age(value), "age is not > 0" :: Nil)
}

case class SSN private (value: String)
object SSN {
  val default = SSN("0" * 12)
  def make(value: String): Either[List[String], SSN] =
    Either.cond(value.size >= 12, SSN(value), "SSN is not 12 chars long" :: Nil)
}

given Transformer.Mode.Accumulating[Either[List[String], _]] = 
  Transformer.Mode.Accumulating.either

val person = Person("Tristan Bongo", 21)

val seriousPerson: Either[List[String], SeriousPerson] =
  person
    .into[SeriousPerson]
    .fallible
    .transform( 
      // we can use all config options from total transformations
      Field.const(_.SSN, SSN.default),
      // a fallible variant of Field.const
      Field.fallibleConst(_.SSN, SSN.make("1" * 12)),
      // a fallible variant of Field.computed
      Field.fallibleComputed(_.SSN, SSN.make.compose(_.name))
    )

a quick example

val Fallible_this: Fallible[[A >: Nothing <: Any] =>> Either[List[String], A], Accumulating[
  [A >: Nothing <: Any] =>> Either[List[String], A]
], Person, SeriousPerson] =
  into[Person](person)[SeriousPerson].fallible[[A >: Nothing <: Any] =>> Either[List[String], A], Accumulating[
    [A >: Nothing <: Any] =>> Either[List[String], A]
  ]](given_Accumulating_Either)

{
  val $scrutinee1: inline$F.type = Fallible_this.inline$F
  val given_Accumulating_F: Accumulating[[A >: Nothing <: Any] =>> Either[List[String], A]] = $scrutinee1
  val source$proxy1: Person = Fallible_this.inline$source

  given_Accumulating_F.map[Tuple2[SSN, Age], SeriousPerson](
    given_Accumulating_F.product[SSN, Age](
      ((value: String) => SSN.make(value)).compose[Person]((_$2: Person) => _$2.name).apply(source$proxy1),
      transformer.transform(source$proxy1.age)
    ),
    (`value₂`: Tuple2[SSN, Age]) =>
      `value₂` match {
        case Tuple2(SSN, age) =>
          new SeriousPerson(name = source$proxy1.name, age = age, SSN = SSN)
        case x =>
          throw new MatchError(x)
      }
  ): Either[List[String], SeriousPerson]
}: Either[List[String], SeriousPerson]

and it gets expanded into this:

a quick example

val Fallible_this: Fallible[[A >: Nothing <: Any] =>> Either[List[String], A], FailFast[
  [A >: Nothing <: Any] =>> Either[List[String], A]
], Person, SeriousPerson] = into[Person](person)[SeriousPerson]
  .fallible[[A >: Nothing <: Any] =>> Either[List[String], A], FailFast[[A >: Nothing <: Any] =>> Either[List[String], A]]](
    given_FailFast_Either
  )

{
  val $scrutinee1: inline$F.type = Fallible_this.inline$F
  val given_FailFast_F: FailFast[[A >: Nothing <: Any] =>> Either[List[String], A]] = $scrutinee1
  val source$proxy1: Person = Fallible_this.inline$source

  given_FailFast_F.flatMap[SSN, SeriousPerson](
    ((value: String) => SSN.make(value)).compose[Person]((_$2: Person) => _$2.name).apply(source$proxy1),
    (SSN: SSN) =>
      given_FailFast_F.map[Age, SeriousPerson](
        transformer.transform(source$proxy1.age),
        (age: Age) => new SeriousPerson(name = source$proxy1.name, age = age, SSN = SSN)
      )
  ): Either[List[String], SeriousPerson]
}: Either[List[String], SeriousPerson]

it gets expanded into something slightly different:

given Transformer.Mode.FailFast[Either[List[String], _]] =
  Transformer.Mode.FailFast.either

and now if we change the transformation mode...

Interface Description Languages aka IDLs

Open API

openapi: "3.0.0"
info:
  version: 0.0.1
  title: Event Management
servers:
  - url: http://localhost:9001
paths:
  /conferences:
    post:
      summary: Create a new conference
      operationId: create-conference
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/CreateConference"
      responses:
        201:
          description: "Conference successfully created"
        400:
          description: "Invalid entity"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/ValidationErrors"
components:
  schemas:
    CreateConference:
      type: object
      properties:
        name:
          type: string
        dateSpan:
          $ref: "#/components/schemas/DateSpan"
        city:
          type: string
      required:
        - name
        - dateSpan
        - city
  
    DateSpan:
      type: object
      properties:
        start:
          type: string
          format: date
        end:
          type: string
          format: date
      required:
        - start
        - end
        
    ValidationErrors:
      type: object
      properties:
        errors:
          type: array
          items:
            type: string
      required:
        - errors

declarative and focused on a single thing
codegen support for every possible language under the sun
great Scala support in the form of guardrail
your source of truth for both implementers and consumers

what does this give us?

the API model

// file: CreateConference.scala
case class CreateConference(name: String, dateSpan: DateSpan, city: String)
object CreateConference {
  implicit val encodeCreateConference: Encoder.AsObject[CreateConference] = ...
  implicit val decodeCreateConference: Decoder[CreateConference] = ...
}

//file: DateSpan.scala
case class DateSpan(start: LocalDate, end: LocalDate)
object DateSpan {
  implicit val encodeDateSpan: Encoder.AsObject[DateSpan] = ...
  implicit val decodeDateSpan: Decoder[DateSpan] = ...
}

//file: ValidationErrors.scala
case class ValidationErrors(errors: Vector[String])
object ValidationErrors {
  implicit val encodeValidationErrors: Encoder.AsObject[ValidationErrors] = ...
  implicit val decodeValidationErrors: Decoder[ValidationErrors] = ...
}

//file: Routes.scala
class Resource[F[_]](implicit F: Async[F]) extends Http4sDsl[F] with CirceInstances {
  def routes(handler: Handler[F]): HttpRoutes[F] = 
    /* all the code you don't want to write is here */
}

object Resource {
  sealed abstract class CreateConferenceResponse
  object CreateConferenceResponse {
    case object Created extends CreateConferenceResponse
    case class BadRequest(value: ValidationErrors) extends CreateConferenceResponse
  }

the unfun part

//file: Routes.scala
trait Handler[F[_]] {
  def createConference(respond: Resource.CreateConferenceResponse.type)(
    body: CreateConference
  ): F[Resource.CreateConferenceResponse]
}

the actually useful part

about that transformation thingy...

Each validated field is a new type... NEWTYPES!

abstract class NewtypeValidated[A, Constraint](using
  Validate[A, Constraint], Name
) {
  opaque type Type = A Refined Constraint
  
  def make(value: A): Either[List[String], Type] =
    refineV[Constraint](value)
     .left.map(err => s"Invalid ${summon[Name].value} - $err" :: Nil)

  given transformer: Transformer.Fallible[Either[List[String], _], A, Type] =
    make(_)
}

each newtype will have an instance of Transformer.Fallible in implicit scope
all the user has to do is provide a refined type that will describe the validation

actually modellin'

case class Conference(
  name: Conference.Name, dateSpan: DateSpan, city: Conference.City
)

object Conference {
  type UnsurprisingString[Size <: Int] = Trimmed And NonEmpty And MaxSize[Size]

  object Name extends NewtypeValidated[String, UnsurprisingString[20]]
  export Name.Type as Name

  object City extends NewtypeValidated[String, UnsurprisingString[20]]
  export City.Type as City
}

case class DateSpan private (start: LocalDate, end: LocalDate)

object DateSpan {
  def create(start: LocalDate, end: LocalDate): Either[List[String], DateSpan] =
    Either
      .cond(
        start.isBefore(end),
        DateSpan(start, end),
        "Invalid DateSpan - start is not before the end"
       )
       .leftMap(_ :: Nil)
}

putting it all together

final class ConferenceRoutes(conferenceRepo: ConferenceRepository) extends
  Handler[IO] {

  private given Transformer.Mode.Accumulating[Either[List[String], _]] = 
    Transformer.Mode.Accumulating.either

  override def createConference(respond: CreateConferenceResponse.type)(
    body: API.CreateConference
  ): IO[CreateConferenceResponse] =
    body
      .into[Conference]
      .fallible
      .transform(
        Field.fallibleComputed(_.dateSpan, _.dateSpan.via(DateSpan.create))
       )
      .leftMap(errs => 
        respond.BadRequest(API.ValidationErrors(errs.toVector))
       )
      .toEitherT[IO]
      .semiflatMap(conf => conferenceRepo.create(conf).as(respond.Created))
      .merge
}

does it actually work though?

curl -X 'POST' \
  'http://localhost:9001/conferences' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "An illegal coference name with over 20 chars",
  "dateSpan": {
    "start": "2023-04-10",
    "end": "2023-04-10"
  },
  "city": "A really really long city name"
}'

400: Bad Request
{
  "errors": [
    "Invalid DateSpan - start is not before the end",
    "Invalid Conference.Name - Right predicate of ((An illegal coference name with over 20 chars is trimmed && !isEmpty(An illegal coference name with over 20 chars)) && (!(44 < 0) && !(44 > 20))) failed: Predicate taking size(An illegal coference name with over 20 chars) = 44 failed: Right predicate of (!(44 < 0) && !(44 > 20)) failed: Predicate (44 > 20) did not fail.",
    "Invalid Conference.City - Right predicate of ((A really really long city name is trimmed && !isEmpty(A really really long city name)) && (!(30 < 0) && !(30 > 20))) failed: Predicate taking size(A really really long city name) = 30 failed: Right predicate of (!(30 < 0) && !(30 > 20)) failed: Predicate (30 > 20) did not fail."
  ]
}

curl -X 'POST' \
  'http://localhost:9001/conferences' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "Loops and bloops",
  "dateSpan": {
    "start": "2023-04-01",
    "end": "2023-04-05"
  },
  "city": "Larks"
}'

201: Created

and a good-boy-request gives us this:

wrap up

https://github.com/arainko/ducktape

file your bugs, criticisms and ideas here:

code samples and alternative versions (smithy4s):

https://github.com/arainko/ducktape-talk

thanks!

Questions?

ducktape

By Aleksander Rainko