Metaprogramming

in Dotty

Nicolas Stucki

LAMP/EPFL

Scala 2.x Macros

Scala 2.x macros

```
Coupled with scalac internals
```
```
Portability problem
```

Dotty macros

```
Portable (TASTy)
```
```
Simpler and safer
```

Dotty language features

```
Achieve the same without macros
```

Inline: As a metaprogramming feature
Match Types: Computing new types
Macros
- Quotes & Splices (simple/high-level)
- TASTy Reflect (typed AST API)

Overview

Inline

As a metaprogramming feature

inline def log[T](msg: String)(thunk: => T): T = ...

Inline definitions

```
Guaranteed inline 
```
```
Potentially recursive
```

Potentially type specializing return type

```
Potentially macro entry point
```

object Logger {

  var indent = 0

  inline def log[T](msg: String)(thunk: => T): T =
    println(s"${"  " * indent}start $msg")
    indent += 1
    val result = thunk
    indent -= 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}

Inlining code

Logger.log("123L^5") {
  power(123L, 5)
}

val msg = "123L^5"
println(s"${"  " * indent}start $msg")
Logger.indent += 1
val result = power(123L, 5)
Logger.indent -= 1
println(s"${"  " * indent}$msg = $result")
result

```
By-value parameter is placed in a val
```
```
By-name parameters are inlined directly
```
```
Special handling for private references
```

expands to

object Logger {

  private var indent = 0



  inline def log[T](msg: String)(op: => T): T = {
    println(s"${"  " * indent}start $msg")
    indent += 1
    val result = op
    indent -= 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}

Inlining code with private references

object Logger {

  private var indent = 0
  def inline$indent: Int = indent
  def inline$indent_=(x$0: Int): Unit = indent = x$0

  inline def log[T](msg: String)(op: => T): T = {
    println(s"${"  " * indent}start $msg")
    inline$indent = inline$indent + 1
    val result = op
    inline$indent = inline$indent - 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}

Accessors will be generated for unacessible references

inline def power(x: Long, n: Int): Long = {
  if (n == 0) 
    1L
  else if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}

Recursive inline

We can call recursively an inline method

val x = expr
if (10 == 0) 
  // 1L
else if (10 % 2 == 1)
  // x * power(x, 10 - 1)
else {
  val y = x * x 
  power(y, 10 / 2) 
}

val x = expr
val y = x * x 
power(y, 5)

inline the code
while simplifying

Calling a recursive inline method

resulting in

val x = expr
val y = x * x 
y * {
  val y2 = y * y
  val y3 = y2 * y2
  y3 * 1L
}

at the end the code will be

power(x, 10)

...

recursively inline

def badPower(x: Long, n: Int): Int = {
  power(x, n)
}

What if n is not statically known?

def badPower(x: Long, n: Int): Int = {
  if (n == 0) 
      1L
  else if (n % 2 == 1)
    x * {
      if (n - 1 == 0) 
        1L
      else if ((n-1) % 2 == 1)
        x * power(x, (n-1) - 1)
      else {
        val y = x * x 
        power(y, (n-1) / 2)
      }
    }
  else {
    val y = x * x 
    if (n - 1 == 0) 
      1L
    else if ((n/1) % 2 == 1)
      y * power(y, (n/2) - 1)
    else {
      val y2 = y * y
      power(y2, (n/2) / 2)
    }
  }
}

Calling a recursive inline method... until when?

...

...

def badPower(x: Long, n: Int): Int = {
  power(x, n) // error: n is not a known constant
}

inline def power(x: Long, inline n: Int): Int = ...

The argument must be a known constant value

Primitive values: Boolean, Int, Double, String, ....

```
Some case classes: Option, ...
```

Inline parameter

Inline conditional

inline def power(x: Long, n: Int): Long = {
  inline if (n == 0) 
    1L
  else inline if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}

def badPower(x: Long, n: Int): Int = {
  power(x, n) 
   // error: cannot reduce inline if
   //        its condition n == 0 is not a constant value
   //        This location is in code that was inlined at ...
}

```
condition must be reduced
```
```
only one branch will remain
```

inline def toInt(n: Nat): Int = 
  inline n match {
    case Zero => 0
    case Succ(n1) => toInt(n1) + 1
  }

val natTwo = toInt(Succ(Succ(Zero)))

Inline match

trait Nat
case object Zero extends Nat
case class Succ[N <: Nat](n: N) extends Nat

```
one case must match the scrutinee
```
```
only one case will remain
```

Peano number as ADT

val natTwo: Int = 2

inlined as

Specializing inline

Return type is going to be specialized 
to a more precise type upon expansion

inline def toInt(n: Nat) <: Int = ...

val natTwo: 2 = 2

val natTwo = toInt(Succ(Succ(Zero)))

inlined as

val natTwo: Int = 2

instead of

class A

class B extends A {
  def meth(): Unit = ...
}

val a: A = choose(true)
val b: B = choose(false)

// error: meth() not defined on A
choose(true).meth()
// Ok
choose(false).meth()

inline def choose(b: Boolean) <: A =
  inline if (b) new A()
  else new B()

Specializing inline

The result type of choose is a subtype of A 
but not necessarily A

inline def setFor[T]: Set[T] = 
  implicit match {
    case ord: Ordering[T] => new TreeSet[T]
    case _                => new HashSet[T]
  }

Inline - Implicit match

setFor[String] // new TreeSet(scala.math.Ordering.String)
setFor[Object] // new HashSet

for each case try to find an implicit of that type

```
if found use that branch
```

import scala.collection.immutable._

Match Types

Computing New Types

type Elem[X] = X match {
  case String => Char
  case Array[t] => t
  case Iterable[t] => t
}

Elem[String]       =:=  Char
Elem[Array[Int]]   =:=  Int
Elem[List[Float]]  =:=  Float
Elem[Nil.type]     =:=  Nothing

Match Types

Elem[X] will have the type of
the first case that matches X

def addEventListener(tpe: String)(action: dom.Event => Any): Unit = js.native

Match type example

but the type is "click" the event will 
be of type dom.MouseEvent

In Scala.js we can define the interface

addEventListener("click") { (e: dom.Event) => ... }

which will be used as

type EventTypeOf[Tp <: String] <: dom.Event = Tp match {
  case "click" => dom.MouseEvent
  ...
  case _ => dom.Event
}

Match type example

def addEventListener[Tp <: String, Ev <: EventTypeOf[Tp]](tpe: Tp)(e: Ev => Any): Unit = js.native

if Tp is a know singleton string 
we can refine the type of the event

addEventListener("click") { (e: dom.MouseEvent) => ... }

Example: Tuples

T1 *: T2 *: ... *: Tn *: Unit

type Unit              <: Tuple

type *:[H, T <: Tuple] <: Tuple

A tuple type

x1 *: x2 *: ... *: xn *: ()

A tuple term

type Unit              <: Tuple

type *:[H, T <: Tuple] <: Tuple

trait Tuple {

  inline def *: [H](x: H): H *: this.type = ...

  inline def ++ [That <: Tuple](that: That): Concat[this.type, That] = ... 

  inline def size: Size[this.type] = ...

  ...
}

Some operation on Tuples

Example: Tuples

type Concat[X <: Tuple, +Y <: Tuple] <: Tuple = X match {
  case Unit => Y
  case head *: tail => head *: Concat[tail, Y]
}

Recursive Match Type

just like concat on a list

def concat(x: List[Int], y: List[Int]): List[Int] = x match {
  case Nil => y
  case head :: tail => head :: concat(tail, y)
}

import scala.compiletime.S

type Size[X <: Tuple] <: Int = X match {
  case Unit => 0
  case x *: xs => S[Size[xs]]
}

Successor type

type S[N <: Int] <: Int = N match {
  case 0 => 1
  case 1 => 2
  case 2 => 3
  ...
}

Successor of a literal Int

Intrinsified in the compiler

import scala.compiletime.constValueOpt

trait Tuple {
  inline def size: Size[this.type] = {
    inline constValueOpt[Size[this.type]] match {
      case Some(n) => n
      case _ => computeSizeAtRuntimeOn(this)
    }
  }
  ...
}

Constant values from types

inline def constValue[T]: T
inline def constValueOpt[T]: Option[T]

Value of the literal type

Macros

With Quotes & Splices

Quotes & Splices

Quoting expressions ...

val expr: Expr[T] = '{ e }

val t: Type[T] = '[ T ]

'{
  val e: T = ${ expr }
 }

Quoting types...

and splicing them inside quotes

'{
  val e2: ${ t } = e
 }

and splicing them inside quotes

import scala.quoted.{Expr, Type}

val x: String = ...
def f(x: String): String = ...

s"println($x)"

s"println(${ f(x) })"

val x: Expr[String] = ...
def f(x: Expr[String]): Expr[String] = ...

'{ println($x) }

'{ println(${ f(x) }) }

Syntax follows similar rules 
of string interpolators

import scala.quoted.{Expr, Type}

def f(x: Expr[String]): Expr[String] = ...

'{
  val x: String = ...
  println(${ f('x) }) 
}

Code inside quotes

can still be referred inside

Syntax

import scala.quoted._

Macro definition through inline

```
Code that will run while compiling
```
```
Code that the user will write
```

${...} outside a '{...} only as body of inline method

val x = 3L
val res = power(x, 10)

inline def power(x: Long, inline n: Int): Long = 
  ${ powerExpr('x, n) }

private def powerExpr(x: Expr[Long], n: Int): Expr[Long] = ...

inline def power(x: Long, inline n: Int): Long = {
  inline if (n == 0) 
    1L
  else inline if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}

def powerExpr(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L } 
  else if (n % 2 == 1)
    '{ $x * ${ powerExpr(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerExpr('y, n / 2) }
  }
}

import scala.quoted._

Conditions can be constant folded by user code

```
Arbitrary compiled code
```

Comparison with inline

Conditions must be constant foldable by the compiler

```
Low syntactic overhead
```

def powerCode(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L } 
  else if (n % 2 == 1)
    '{ $x * ${ powerCode(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerCode('y, n / 2) }
  }
}

    powerCode('{x}, 0) 
  // generates the code
   '{ 1L }

    powerCode('{x}, 2) 
  // generates the code
   '{ val y = x * x; y }

    powerCode('{x}, 4) 
  // generates the code
   '{ val y = x * x; y * y }

import scala.quoted._

Code generation

powerCode generates an Expr containing the generated code

Well-typed macros cannot go wrong

"For any free variable reference, the number of quoted scopes and the number of spliced scopes between the reference and its definition must be equal"

def powerExpr(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L }
  else if (n % 2 == 1)
    '{ $x * ${ powerExpr(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerExpr('y, n / 2) }
  }
}

import scala.quoted._

Passing known values into a quote

def (x: T) toExpr[T: Liftable]: Expr[T] = ...

val one: Expr[Int] = 1.toExpr

val hello: Expr[String] = "hello".toExpr

import scala.quoted._

```
Liftable type class
```
```
Extensible in libraries
```

implied for Liftable[Boolean] {
  def toExpr(bool: Boolean): Expr[Boolean] = 
    if (bool) '{true} else '{false}
}

implied [T: Liftable: Type] for Liftable[List[T]] {
  def toExpr(list: List[T]): Expr[List[T]] = list match {
    case x :: xs => '{ ${x.toExpr} :: ${toExpr(xs)} }
    case Nil => '{ List.empty[T] }
  }
}

Passing known values into a quote

Whitebox macro

inline def whiteboxMacro(...) <: X = ${ ... }

Simply combination

```
specialized return type
```
```
macro $
```

import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

Analyzing contents of Expr

Inspecting Quotes

inline def swap(tuple: =>(Int, Long)): (Long, Int) = 
  ${ swapExpr('tuple) }


val x = (1, 2L)
swap(x) // inlines: x.swap
swap((3, 4L)) // inlines: (4L, 3)

def swapExpr(tuple: Expr[(Int, Long)]) given Reflection: Expr[(Long, Int)] = {
  tuple match {
    case '{ ($x1, $x2) } => '{ ($x2, $x1) }
    case _ => '{ $tuple.swap }
  }
}

import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

// vanilla power implementation
def power(x: Long, n: Int): Int = ...

// power optimized for n
def powerExpr(x: Expr[Long], n: Int): Expr[Int] = ...

Analyzing contents of Expr

Inspecting Quotes

def powerExpr2(x: Expr[Long], n: Expr[Int]) given Reflection: Expr[Long] = {
  (x, n) match {
    case (Const(y), Const(k)) =>  power(y, k).toExpr
    case (       _, Const(k)) =>  powerExpr(x, k)
    case _                    => '{ power($x, $n) }
  }
}

inline def (self: => StringContext) S (args: => String*): String = 
  ${ sExpr('self, 'args) }

def sExpr(self: Expr[StringContext], args: Expr[Seq[String]]) given Reflection: Expr[String] = {
 self match {
   case '{ StringContext(${ConstSeq(parts)}: _*) } =>
     val upprerParts: List[String] = parts.toList.map(_.toUpperCase)
     val upprerPartsExpr: Expr[List[String]] = upprerParts.map(_.toExpr).toExprOfList
     '{ StringContext($upprerPartsExpr: _*).s($args: _*) }

   case _ => ...
  }
}

Example from a string interpolation macro

import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

Inspecting Quotes ... with Quotes

StringContext("hello ", "!").S(world)

Analyzing arguments to extract constants

Macros

With TASTy Reflection

(low-level tree API)

TASTy (Typed ASTs)

TASTy file format is the standard encoding for all Scala 3 typed trees
- Typed ASTs with source positions
- Scaladoc comments
- Extensible
TASTy Reflect provides an API over these typed trees
- Inspection and construction of trees
- Inspection of positions and Scaladoc comments
- Stability given by the stability of the file format

TASTy Reflect in a macro (tree API)

import scala.tasty.Reflection

object Const {
  def unapply[T](expr: Expr[T]) given (refl: Reflection): Option[T] = {
    import refl._
    def rec(tree: Term): Option[T] = tree match {
      case Literal(c) => Some(c.value.asInstanceOf[T])
      case Block(Nil, e) => rec(e)
      case _  => None
    }
    rec(expr.unseal)
  }
}

Look into the typed AST of the expression

```
Types not statically know
```

TASTy Reflect in a macro

A macro can pass an given instance of scala.tasty.Reflection

Reflection is used by the '{ ... } and Const(...) paterns

def powerExpr2(x: Expr[Long], n: Expr[Int]) given Reflection: Expr[Long] = {
  (x, n) match {
    case (Const(y), Const(k)) =>  power(y, k).toExpr
    case (       _, Const(k)) =>  powerExpr(x, k)
    case _                    => '{ power($x, $n) }
  }
}

import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

Shapeless 3

Features used: specializing inline def, inline matches, implicit match

scala.Tuple

```
Features used: Match types, inline
```

```
Scalatest
```
- ```
Features used: macros
```

f-interpolator and xml-interpolator

Features used: macros, specializing inline def

Prototypes

Thank you

https://slides.com/nicolasstucki/scala-days-2019

Questions?

Metaprogramming in Dotty

By Nicolas Stucki

Metaprogramming in Dotty

Talk at Scala Days 2019

1,733

Metaprogramming

in Dotty

Scala 2.x Macros

Overview

Inline

Inline definitions

Inlining code

Inlining code with private references

Recursive inline

Calling a recursive inline method

...

Calling a recursive inline method... until when?

Inline parameter

Inline conditional

Inline match

Specializing inline

Specializing inline

Inline - Implicit match

Match Types

Match Types

Match type example

Match type example

Example: Tuples

Example: Tuples

Recursive Match Type

Successor type

Constant values from types

Macros

Quotes & Splices

Syntax

Macro definition through inline

Comparison with inline

Code generation

Well-typed macros cannot go wrong

Passing known values into a quote

Passing known values into a quote

Whitebox macro

Inspecting Quotes

Inspecting Quotes

Inspecting Quotes ... with Quotes

Macros

TASTy (Typed ASTs)

TASTy Reflect in a macro (tree API)

TASTy Reflect in a macro

Prototypes

Thank you

Questions?

Metaprogramming in Dotty

More from Nicolas Stucki