Crushing boilerplate with Scala macros

Marcin Rzeźnicki

mrzeznicki@iterato.rs

Conundrum

Our developers were losing time due to mindless, recurring boilerplate they were forced to write, over and over in each and every project.

 

What Scala, as a language, lacks that makes it susceptible to boilerplate code slipping in?

What did we do?

  • Went through various Scala code-bases we had been writing or maintaining
  • Identified those repetitive patterns
  • Pushed as much of these to a common library that we could reuse throughout existing and future projects

https://github.com/theiterators/kebs

Sources of boilerplate - Slick

object People {
  // BOILERPLATE ALERT!
  implicit val userIdColumnType: BaseColumnType[UserId] = 
    MappedColumnType.base(_.userId, UserId.apply)
  implicit val emailAddressColumnType: BaseColumnType[EmailAddress] = 
    MappedColumnType.base(_.emailAddress, EmailAddress.apply)
  implicit val fullNameColumnType: BaseColumnType[FullName] = 
    MappedColumnType.base(_.fullName, FullName.apply)
  //
}

class People(tag: Tag) extends Table[Person](tag, "people") {
  import People._

  def userId: Rep[UserId] = column[UserId]("user_id")
  def emailAddress: Rep[EmailAddress] = column[EmailAddress]("email_address")
  def fullName: Rep[FullName] = column[FullName]("full_name")
 
  override def * : ProvenShape[Person] =
        (userId, emailAddress, fullName) <> (Person.tupled, Person.unapply)
}

Sources of boilerplate - Slick

Possible solutions - generic wrapper

def valueColumnType[T <: Product : ClassTag, P : BaseColumnType](construct: P => T): 
    BaseColumnType[T] = 
  MappedColumnType.base[T, P](_.productElement(0).asInstanceOf[P], construct)
  • not really type-safe (cast is involved),
  • you cannot restrict types to cover only 1-element case-classes

Sources of boilerplate - Slick

Possible solutions - MappedTo

Some people rightly believe that this solution is bad, because it couples domain with storage too tightly.

Sources of boilerplate - Slick

Possible solutions - Shapeless

import shapeless._

implicit def hList1CoumnType[Head: BaseColumnType] =
  MappedColumnType
    .base[Head :: HNil, Head](l => l.head, head => head :: HNil)
implicit def valueColumnType[T, P](implicit gen: Generic.Aux[T, P],
                                   tag: ClassTag[T],
                                   baseColumnType: BaseColumnType[P]) =
  MappedColumnType
    .base[T, P](t => gen.to(t), p => gen.from(p))

type-safe, but... that’s a lot of code for essentially stating that 1-element tuple is isomorphic to its only element!

Sources of boilerplate - Slick

Possible solutions - Shapeless

case class Name(name: String) extends AnyVal

class OneElement(tag: Tag) extends Table[Name](tag, "MATRYOSHKA") {
  def name = column[String]("name")
  override def * : ProvenShape[Name] = name <> (Name.apply, Name.unapply)
}

type mismatch;
[error] found : slick.lifted.MappedProjection[Name,String]
[error] required: slick.lifted.ProvenShape[Name]
[error] override def * = name <> (Name.apply, Name.unapply)
[error]

Sources of boilerplate - Slick

Possible solutions - Shapeless

Turns out you just unwillingly provided two ambiguous mappings for Name. One — repColumnShape — materialized with help of our catch-all implicit, and the other — mappedProjectionShape — which you almost wrote yourself by providing table mapping *.

So a truly perfect solution could disable automatic derivation if some other derivation already existed.

Is it Scala's fault?

All this redundant code wouldn’t have been necessary if you were able to:

  • treat case-classes as instances of respective ProductN
  • access case-class companion’s apply method in a generic way

kebs-slick

Scala has one powerful escape hatch that lets programmers do things that language doesn’t let them do and it even can circumvent the language rules. You probably heard of it before.

`Tis macros!!

kebs-slick

Macros, among other useful things, can be used to materialize implicits. There is even a special flavor of macros called white-box macros that, when chosen by implicit resolver to provide an implicit, can decide if they materialize it, or bail out and not provide it all. This ability makes them useful to meet our extra condition — disable automatic derivation based on context!

kebs-slick

https://github.com/theiterators/kebs/blob/master/slick/src/main/scala/pl/iterators/kebs/Kebs.scala

kebs - slick

object People {
  implicit val userIdColumnType: BaseColumnType[UserId] = 
        MappedColumnType.base(_.userId, UserId.apply)
  implicit val emailAddressColumnType: BaseColumnType[EmailAddress] = 
        MappedColumnType.base(_.emailAddress, EmailAddress.apply)
  implicit val fullNameColumnType: BaseColumnType[FullName] = 
        MappedColumnType.base(_.fullName, FullName.apply)
  implicit val addressLineColumnType: BaseColumnType[AddressLine] = 
        MappedColumnType.base(_.line, AddressLine.apply)
  implicit val postalCodeColumnType: BaseColumnType[PostalCode] = 
        MappedColumnType.base(_.postalCode, PostalCode.apply)
  implicit val cityColumnType: BaseColumnType[City] = 
        MappedColumnType.base(_.city, City.apply)
  implicit val areaColumnType: BaseColumnType[Area] = 
        MappedColumnType.base(_.area, Area.apply)
  implicit val countryColumnType: BaseColumnType[Country] = 
        MappedColumnType.base(_.country, Country.apply)
  implicit val taxIdColumnType: BaseColumnType[TaxId] = 
        MappedColumnType.base(_.taxId, TaxId.apply)
  implicit val bankNameColumnType: BaseColumnType[BankName] = 
        MappedColumnType.base(_.name, BankName.apply)
  implicit val recipientNameColumnType: BaseColumnType[RecipientName] = 
        MappedColumnType.base(_.name, RecipientName.apply)
  implicit val additionalInfoColumnType: BaseColumnType[AdditionalInfo] = 
        MappedColumnType.base(_.content, AdditionalInfo.apply)
  implicit val bankAccountNumberColumnType: BaseColumnType[BankAccountNumber] =
        MappedColumnType.base(_.number, BankAccountNumber.apply)
}

kebs - slick

import pl.iterators.kebs._

kebs-slick

support for

  • slick_pg,
  • enumeratum,
  • using trait instead of import
  • one-column tables

Sources of boilerplate - spray-json

object ThingProtocol extends JsonProtocol {
  def jsonFlatFormat[P, T <: Product](construct: P => T)
                                     (implicit jw: JsonWriter[P], jr: JsonReader[P]): 
        JsonFormat[T] =
    new JsonFormat[T] {
      override def read(json: JsValue): T = construct(jr.read(json))
      override def write(obj: T): JsValue = jw.write(obj.productElement(0).asInstanceOf[P])
    }

  // BOILERPLATE ALERT!!
  implicit val errorJsonFormat = jsonFormat1(Error.apply)
  implicit val thingIdJsonFormat = jsonFlatFormat(ThingId.apply)
  implicit val tagIdJsonFormat = jsonFlatFormat(TagId.apply)
  implicit val thingNameJsonFormat = jsonFlatFormat(ThingName.apply)
  implicit val thingDescriptionJsonFormat = jsonFlatFormat(ThingDescription.apply)
  implicit val locationJsonFormat = jsonFormat2(Location.apply)
  implicit val createThingRequestJsonFormat = jsonFormat5(ThingCreateRequest.apply)
  implicit val thingJsonFormat = jsonFormat6(Thing.apply)
  // 
}

Sources of boilerplate - spray-json

Possible solutions

  • switch to play-json :-)
  • shapeless, but it can get a bit hairy. See this for some excellent code. But still you could not easily mix flat and non-flat formats in one scope

 

 

kebs-spray-json

  • automatically generate RootJsonFormat for any case-class or flat JsonFormat for 1-element case-classes
  • switch between formats depending on context

kebs-spray-json

Because white-box macros can decide whether they materialize an implicit or not, we are able to simultaneously bring two implicits into scope:

 

implicit def jsonFormatN[T <: Product]: RootJsonFormat[T] 
    = macro KebsSprayMacros.materializeRootFormat[T]
implicit def jsonFlatFormat[T <: Product]: JsonFormat[T] 
    = macro KebsSprayMacros.materializeFlatFormat[T]

kebs-spray-json

https://github.com/theiterators/kebs/blob/master/spray-json-macros/src/main/scala/pl/iterators/kebs/json/macros/KebsSprayMacros.scala

 

kebs-spray-json

object ThingProtocol extends JsonProtocol {
  def jsonFlatFormat[P, T <: Product](construct: P => T)
                                     (implicit jw: JsonWriter[P], jr: JsonReader[P]): 
    JsonFormat[T] =
    new JsonFormat[T] {
      override def read(json: JsValue): T = construct(jr.read(json))
      override def write(obj: T): JsValue = jw.write(obj.productElement(0).asInstanceOf[P])
    }

   implicit val errorJsonFormat              = jsonFormat1(Error.apply)
   implicit val thingIdJsonFormat            = jsonFlatFormat(ThingId.apply)
   implicit val tagIdJsonFormat              = jsonFlatFormat(TagId.apply)
   implicit val thingNameJsonFormat          = jsonFlatFormat(ThingName.apply)
   implicit val thingDescriptionJsonFormat   = jsonFlatFormat(ThingDescription.apply)
   implicit val locationJsonFormat           = jsonFormat2(Location.apply)
   implicit val createThingRequestJsonFormat = jsonFormat5(ThingCreateRequest.apply)
   implicit val thingJsonFormat              = jsonFormat6(Thing.apply)
}

kebs-spray-json

object ThingProtocol extends JsonProtocol with KebsSpray

Is it Scala's fault?

One might want better implicit prioritization. The current implementation (implicits defined in a type are preferred over the ones from a subtype) is a one-way street. You can only request the more specific implicit to be prioritized over the less specific one.

Is it Scala's fault?

class A
class B extends A

// THIS ONE WORKS CORRECTLY

trait LowPriorityImplicits {
  implicit val a: A = new A
}
object Implicits extends LowPriorityImplicits {
  implicit val b: A = new B
}

import Implicits._

implicitly[A]

Is it Scala's fault?

// THIS ONE FAILS TO COMPILE

class A
class B extends A

trait LowPriorityImplicits {
  implicit val b: B = new B
}
object Implicits extends LowPriorityImplicits {
  implicit val a: A = new A
}

import Implicits._

implicitly[A]

kebs-spray-json

Tons of other features:

  • snakified formats
  • recursive formats
  • enumeratum
  • @dbronecki added support for case-classes with > 22 fields

I’ll be delighted to hear of your sources of boilerplate

Bad stuff

  • API is not clearly documented
  • API allows you to generate wrong code. It’s not unlike managing memory in C — correct semantics are not enforced and you’re on your own discovering what you did wrong
  • Debugging is a pain

Bad stuff

  • Because macro programming is very fragile in its current state, you need to test it a lot to be sure it does not break a legitimate code.
  • Compilation time will suffer (we found ca. 10% growth in projects with number of serialized DTOs being about 40). That’s especially true if you do a lot of white-box macros.

Bad stuff

  • The nature of white box makes it materialize implicit at every expansion site. Figuratively speaking, it is like ‘pasting’ the generated code whenever it is used. This can cause more allocations and will certainly increase the size of compiled program (this will be mitigated by Scala 2.12 delambdafy allocation scheme)
  • You have to keep up with Scala’s reflect API changes and it can change considerably when dotty comes

Good stuff

  • You do not have to write it!
  • No slow run-time reflection, e.g., conversion to snake_case in JSON format
  • You can do amazing things that bend language rules, e.g., smart JSON formats

Thank you!

PS. kebs is a part of Scala Spree initiative. If you want to hack it please join us at:


Open Source Spree with Scala Center (Functional Tricity #8)

 

    Friday, July 7, 2017

    10:00 AM
    Olivia Business Centre, Olivia FOUR

    aleja Grunwaldzka 472a, Gdansk

Made with Slides.com