Metaprogramming

in Dotty

Nicolas Stucki

LAMP/EPFL

Scala 2.x Macros

  • Scala 2.x macros
    • Coupled with scalac internals
    • Portability problem
  • Dotty macros
    • Portable (TASTy)
    • Simpler and safer
  • ​Dotty language features
    • Achieve the same without macros
  • Inline: As a metaprogramming feature
  • Match Types: Computing new types
  • Macros
    • Quotes & Splices (simple/high-level)
    • TASTy Reflect (typed AST API)

Overview

Inline

As a metaprogramming feature

inline def log[T](msg: String)(thunk: => T): T = ...

Inline definitions

  • Guaranteed inline 
  • Potentially recursive
  • Potentially type specializing return type
  • Potentially macro entry point
 
object Logger {

  var indent = 0

  inline def log[T](msg: String)(thunk: => T): T =
    println(s"${"  " * indent}start $msg")
    indent += 1
    val result = thunk
    indent -= 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}

Inlining code

Logger.log("123L^5") {
  power(123L, 5)
}
val msg = "123L^5"
println(s"${"  " * indent}start $msg")
Logger.indent += 1
val result = power(123L, 5)
Logger.indent -= 1
println(s"${"  " * indent}$msg = $result")
result
  • By-value parameter is placed in a val
  • By-name parameters are inlined directly
  • Special handling for private references
expands to
object Logger {

  private var indent = 0



  inline def log[T](msg: String)(op: => T): T = {
    println(s"${"  " * indent}start $msg")
    indent += 1
    val result = op
    indent -= 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}

Inlining code with private references

object Logger {

  private var indent = 0
  def inline$indent: Int = indent
  def inline$indent_=(x$0: Int): Unit = indent = x$0

  inline def log[T](msg: String)(op: => T): T = {
    println(s"${"  " * indent}start $msg")
    inline$indent = inline$indent + 1
    val result = op
    inline$indent = inline$indent - 1
    println(s"${"  " * indent}$msg = $result")
    result
  }
}
Accessors will be generated for unacessible references
inline def power(x: Long, n: Int): Long = {
  if (n == 0) 
    1L
  else if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}

Recursive inline

We can call recursively an inline method
val x = expr
if (10 == 0) 
  // 1L
else if (10 % 2 == 1)
  // x * power(x, 10 - 1)
else {
  val y = x * x 
  power(y, 10 / 2) 
}
val x = expr
val y = x * x 
power(y, 5)
inline the code
while simplifying

Calling a recursive inline method

resulting in
val x = expr
val y = x * x 
y * {
  val y2 = y * y
  val y3 = y2 * y2
  y3 * 1L
}
at the end the code will be
power(x, 10)

...

recursively inline
def badPower(x: Long, n: Int): Int = {
  power(x, n)
}
What if n is not statically known?
def badPower(x: Long, n: Int): Int = {
  if (n == 0) 
      1L
  else if (n % 2 == 1)
    x * {
      if (n - 1 == 0) 
        1L
      else if ((n-1) % 2 == 1)
        x * power(x, (n-1) - 1)
      else {
        val y = x * x 
        power(y, (n-1) / 2)
      }
    }
  else {
    val y = x * x 
    if (n - 1 == 0) 
      1L
    else if ((n/1) % 2 == 1)
      y * power(y, (n/2) - 1)
    else {
      val y2 = y * y
      power(y2, (n/2) / 2)
    }
  }
}

Calling a recursive inline method... until when?

...
...
def badPower(x: Long, n: Int): Int = {
  power(x, n) // error: n is not a known constant
}
inline def power(x: Long, inline n: Int): Int = ...
The argument must be a known constant value
  • Primitive values: Boolean, Int, Double, String, ....
  • Some case classes: Option, ...

Inline parameter

Inline conditional

inline def power(x: Long, n: Int): Long = {
  inline if (n == 0) 
    1L
  else inline if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}
def badPower(x: Long, n: Int): Int = {
  power(x, n) 
   // error: cannot reduce inline if
   //        its condition n == 0 is not a constant value
   //        This location is in code that was inlined at ...
}
  • condition must be reduced
    
    
  • ​only one branch will remain
inline def toInt(n: Nat): Int = 
  inline n match {
    case Zero => 0
    case Succ(n1) => toInt(n1) + 1
  }

val natTwo = toInt(Succ(Succ(Zero)))

Inline match

trait Nat
case object Zero extends Nat
case class Succ[N <: Nat](n: N) extends Nat
  • one case must match the scrutinee
    
  • ​only one case will remain
Peano number as ADT
val natTwo: Int = 2
inlined as

Specializing inline

Return type is going to be specialized 
to a more precise type upon expansion
inline def toInt(n: Nat) <: Int = ...
val natTwo: 2 = 2
val natTwo = toInt(Succ(Succ(Zero)))
inlined as
val natTwo: Int = 2
instead of
class A

class B extends A {
  def meth(): Unit = ...
}
val a: A = choose(true)
val b: B = choose(false)

// error: meth() not defined on A
choose(true).meth()
// Ok
choose(false).meth()
inline def choose(b: Boolean) <: A =
  inline if (b) new A()
  else new B()

Specializing inline

The result type of choose is a subtype of A 
but not necessarily A
inline def setFor[T]: Set[T] = 
  implicit match {
    case ord: Ordering[T] => new TreeSet[T]
    case _                => new HashSet[T]
  }

Inline - Implicit match

setFor[String] // new TreeSet(scala.math.Ordering.String)
setFor[Object] // new HashSet
  • for each case try to find an implicit of that type
  • if found use that branch
import scala.collection.immutable._

Match Types

Computing New Types

type Elem[X] = X match {
  case String => Char
  case Array[t] => t
  case Iterable[t] => t
}
Elem[String]       =:=  Char
Elem[Array[Int]]   =:=  Int
Elem[List[Float]]  =:=  Float
Elem[Nil.type]     =:=  Nothing

Match Types

Elem[X] will have the type of
the first case that matches X

def addEventListener(tpe: String)(action: dom.Event => Any): Unit = js.native

Match type example

but the type is "click" the event will 
be of type dom.MouseEvent

 
In Scala.js we can define the interface

 
addEventListener("click") { (e: dom.Event) => ... }
which will be used as

 
type EventTypeOf[Tp <: String] <: dom.Event = Tp match {
  case "click" => dom.MouseEvent
  ...
  case _ => dom.Event
}

Match type example

def addEventListener[Tp <: String, Ev <: EventTypeOf[Tp]](tpe: Tp)(e: Ev => Any): Unit = js.native
if Tp is a know singleton string 
we can refine the type of the event
addEventListener("click") { (e: dom.MouseEvent) => ... }

Example: Tuples

T1 *: T2 *: ... *: Tn *: Unit
type Unit              <: Tuple

type *:[H, T <: Tuple] <: Tuple
A tuple type
x1 *: x2 *: ... *: xn *: ()
A tuple term
type Unit              <: Tuple

type *:[H, T <: Tuple] <: Tuple

trait Tuple {

  inline def *: [H](x: H): H *: this.type = ...

  inline def ++ [That <: Tuple](that: That): Concat[this.type, That] = ... 

  inline def size: Size[this.type] = ...

  ...
}
Some operation on Tuples

Example: Tuples

type Concat[X <: Tuple, +Y <: Tuple] <: Tuple = X match {
  case Unit => Y
  case head *: tail => head *: Concat[tail, Y]
}

Recursive Match Type

just like concat on a list
 
def concat(x: List[Int], y: List[Int]): List[Int] = x match {
  case Nil => y
  case head :: tail => head :: concat(tail, y)
}
import scala.compiletime.S

type Size[X <: Tuple] <: Int = X match {
  case Unit => 0
  case x *: xs => S[Size[xs]]
}

Successor type

type S[N <: Int] <: Int = N match {
  case 0 => 1
  case 1 => 2
  case 2 => 3
  ...
}
Successor of a literal Int
Intrinsified in the compiler
import scala.compiletime.constValueOpt

trait Tuple {
  inline def size: Size[this.type] = {
    inline constValueOpt[Size[this.type]] match {
      case Some(n) => n
      case _ => computeSizeAtRuntimeOn(this)
    }
  }
  ...
}

Constant values from types

inline def constValue[T]: T
inline def constValueOpt[T]: Option[T]
Value of the literal type

Macros

With Quotes & Splices

Quotes & Splices

Quoting expressions ...
val expr: Expr[T] = '{ e }
val t: Type[T] = '[ T ]
'{
  val e: T = ${ expr }
 }
Quoting types...

and splicing them inside quotes 

'{
  val e2: ${ t } = e
 }

and splicing them inside quotes

import scala.quoted.{Expr, Type}
val x: String = ...
def f(x: String): String = ...

s"println($x)"

s"println(${ f(x) })"
val x: Expr[String] = ...
def f(x: Expr[String]): Expr[String] = ...

'{ println($x) }

'{ println(${ f(x) }) }
Syntax follows similar rules 
of string interpolators
import scala.quoted.{Expr, Type}
def f(x: Expr[String]): Expr[String] = ...

'{
  val x: String = ...
  println(${ f('x) }) 
}
Code inside quotes

can still be referred inside

Syntax

import scala.quoted._

Macro definition through inline

  • Code that will run while compiling
  •  
  •  
  •  
  • Code that the user will write
${...} outside a '{...} only as body of inline method
val x = 3L
val res = power(x, 10)
inline def power(x: Long, inline n: Int): Long = 
  ${ powerExpr('x, n) }

private def powerExpr(x: Expr[Long], n: Int): Expr[Long] = ...
inline def power(x: Long, inline n: Int): Long = {
  inline if (n == 0) 
    1L
  else inline if (n % 2 == 1)
    x * power(x, n - 1)
  else {
    val y: Long = x * x 
    power(y, n / 2) 
  }
}
def powerExpr(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L } 
  else if (n % 2 == 1)
    '{ $x * ${ powerExpr(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerExpr('y, n / 2) }
  }
}
import scala.quoted._
  • Conditions can be constant folded by user code
  • Arbitrary compiled code
    

Comparison with inline

  • Conditions must be constant foldable by the compiler
    
  • Low syntactic overhead
def powerCode(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L } 
  else if (n % 2 == 1)
    '{ $x * ${ powerCode(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerCode('y, n / 2) }
  }
}
    powerCode('{x}, 0) 
  // generates the code
   '{ 1L }

    powerCode('{x}, 2) 
  // generates the code
   '{ val y = x * x; y }

    powerCode('{x}, 4) 
  // generates the code
   '{ val y = x * x; y * y }
import scala.quoted._

Code generation

powerCode generates an Expr containing the generated code

Well-typed macros cannot go wrong

"For any free variable reference, the number of quoted scopes and the number of spliced scopes between the reference and its definition must be equal"
def powerExpr(x: Expr[Long], n: Int): Expr[Long] = {
  if (n == 0) 
    '{ 1L }
  else if (n % 2 == 1)
    '{ $x * ${ powerExpr(x, n - 1) } }
  else '{
    val y: Long = $x * $x
    ${ powerExpr('y, n / 2) }
  }
}
import scala.quoted._

Passing known values into a quote

def (x: T) toExpr[T: Liftable]: Expr[T] = ...
val one: Expr[Int] = 1.toExpr

val hello: Expr[String] = "hello".toExpr
import scala.quoted._
  • Liftable type class
  • Extensible in libraries
implied for Liftable[Boolean] {
  def toExpr(bool: Boolean): Expr[Boolean] = 
    if (bool) '{true} else '{false}
}
implied [T: Liftable: Type] for Liftable[List[T]] {
  def toExpr(list: List[T]): Expr[List[T]] = list match {
    case x :: xs => '{ ${x.toExpr} :: ${toExpr(xs)} }
    case Nil => '{ List.empty[T] }
  }
}

Passing known values into a quote

Whitebox macro

inline def whiteboxMacro(...) <: X = ${ ... }
Simply combination
  • specialized return type
  • macro $
import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

Analyzing contents of Expr

Inspecting Quotes

inline def swap(tuple: =>(Int, Long)): (Long, Int) = 
  ${ swapExpr('tuple) }


val x = (1, 2L)
swap(x) // inlines: x.swap
swap((3, 4L)) // inlines: (4L, 3)
def swapExpr(tuple: Expr[(Int, Long)]) given Reflection: Expr[(Long, Int)] = {
  tuple match {
    case '{ ($x1, $x2) } => '{ ($x2, $x1) }
    case _ => '{ $tuple.swap }
  }
}
import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection
// vanilla power implementation
def power(x: Long, n: Int): Int = ...

// power optimized for n
def powerExpr(x: Expr[Long], n: Int): Expr[Int] = ... 

Analyzing contents of Expr

Inspecting Quotes

def powerExpr2(x: Expr[Long], n: Expr[Int]) given Reflection: Expr[Long] = {
  (x, n) match {
    case (Const(y), Const(k)) =>  power(y, k).toExpr
    case (       _, Const(k)) =>  powerExpr(x, k)
    case _                    => '{ power($x, $n) }
  }
}
inline def (self: => StringContext) S (args: => String*): String = 
  ${ sExpr('self, 'args) }

def sExpr(self: Expr[StringContext], args: Expr[Seq[String]]) given Reflection: Expr[String] = {
 self match {
   case '{ StringContext(${ConstSeq(parts)}: _*) } =>
     val upprerParts: List[String] = parts.toList.map(_.toUpperCase)
     val upprerPartsExpr: Expr[List[String]] = upprerParts.map(_.toExpr).toExprOfList
     '{ StringContext($upprerPartsExpr: _*).s($args: _*) }

   case _ => ...
  }
}
Example from a string interpolation macro
import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection

Inspecting Quotes ... with Quotes

StringContext("hello ", "!").S(world)
Analyzing arguments to extract constants

Macros

With TASTy Reflection

(low-level tree API)

TASTy (Typed ASTs)

  • TASTy file format is the standard encoding for all Scala 3 typed trees
    • Typed ASTs with source positions
    • Scaladoc comments
    • Extensible
  • TASTy Reflect provides an API over these typed trees
    • Inspection and construction of trees
    • Inspection of positions and Scaladoc comments
    • Stability given by the stability of the file format

TASTy Reflect in a macro (tree API)

import scala.tasty.Reflection

object Const {
  def unapply[T](expr: Expr[T]) given (refl: Reflection): Option[T] = {
    import refl._
    def rec(tree: Term): Option[T] = tree match {
      case Literal(c) => Some(c.value.asInstanceOf[T])
      case Block(Nil, e) => rec(e)
      case _  => None
    }
    rec(expr.unseal)
  }
}
  • Look into the typed AST of the expression
  • Types not statically know

TASTy Reflect in a macro

  • A macro can pass an given instance of scala.tasty.Reflection
  • Reflection is used by the '{ ... } and Const(...) paterns
def powerExpr2(x: Expr[Long], n: Expr[Int]) given Reflection: Expr[Long] = {
  (x, n) match {
    case (Const(y), Const(k)) =>  power(y, k).toExpr
    case (       _, Const(k)) =>  powerExpr(x, k)
    case _                    => '{ power($x, $n) }
  }
}
import scala.quoted.Expr
import scala.quoted.matching.Const
import scala.tasty.Reflection
  • Shapeless 3 
    • Features used: specializing inline def, inline matches, implicit match
  • scala.Tuple
    • Features used: Match types, inline
  • Scalatest
    • Features used: macros
  • f-interpolator and xml-interpolator
    • Features used: macros, specializing inline def

Prototypes

Thank you

https://slides.com/nicolasstucki/scala-days-2019

Questions?

Metaprogramming in Dotty

By Nicolas Stucki

Metaprogramming in Dotty

Talk at Scala Days 2019

  • 226
Loading comments...