${all the world's a stage}
Staged programming in Scala 3
Chris Birchall
47 Degrees
Agenda
- Metaprogramming fundamentals
- Macros in Scala 3
- Runtime staging
- (Multi-stage programming)
# ALL THE WORLD'S A STAGE
Metaprogramming
# FUNDAMENTALS
"code about code"
we will be dealing with functions that:
- take representations of code as input
- return representations of code as output
Q: Why would we want to do that?
Quoting and splicing
# FUNDAMENTALS
val x = 3
val y = 2
x + y + 5
val expr: Expr[Int] = '{
val x = 3
val y = 2
x + y + 5
}
quote
plain old
Scala code
representation
of that code
Quoting and splicing
# FUNDAMENTALS
val x = 3
val y = 2
x + y + 5
quote
plain old
Scala code
'{
...
$expr
...
}
val expr: Expr[Int] = '{
val x = 3
val y = 2
x + y + 5
}
representation
of that code
splice
# FUNDAMENTALS
${'{e}} = e
'{${e}} = e
Quoting and splicing are duals.
For any expression e:
# FUNDAMENTALS
Stages (a.k.a. levels or phases)
val x = 1
Level
0
Plain old Scala code is level 0
# FUNDAMENTALS
Stages (a.k.a. levels or phases)
def foo(using Quotes): Expr[Int] = '{ 1 + 2 }
Level
0
1
Quoting increases level by 1
# FUNDAMENTALS
Stages (a.k.a. levels or phases)
def foo(x: Expr[Int])(using Quotes): Expr[Int] = '{ ${x} + 1 }
Level
0
1
Splicing reduces level by 1
# FUNDAMENTALS
Stages (a.k.a. levels or phases)
Variables must be bound and used on the same level
(x is bound on level 0, trying to use on level 1)
def foo(using Quotes): Expr[Int] =
val x = 1
'{ x + 2 }
(x is bound on level 1, trying to use on level 0)
def bar(using Quotes): Expr[Int => Int] =
'{ (x: Int) => $x + 2 }
# FUNDAMENTALS
Intuition
Quotes indicate that we are constructing a computation which will run in a future stage
A splice indicates that we must perform an immediate computation while building the quoted computation
def someComplicatedFunction(): Expr[Int] = ???
// returns '{ 4 + 5 }
'{ ${someComplicatedFunction()} * 2 }
# FUNDAMENTALS
Intuition: string interpolation
s"foo ${val y = "yeah"; s"hello $y wow"} baz"
val x = "bar"
s"foo $x baz"
# MACROS
World's simplest macro: unless
import scala.quoted.*
def unlessImpl(pred: Expr[Boolean], f: Expr[Unit])(using Quotes): Expr[Unit] =
'{ if (!$pred) $f }
# MACROS
World's simplest macro: unless
inline def unless(pred: Boolean)(f: => Unit): Unit = ${ unlessImpl('pred, 'f) }
import scala.quoted.*
def unlessImpl(pred: Expr[Boolean], f: Expr[Unit])(using Quotes): Expr[Unit] =
'{ if (!$pred) $f }
# MACROS
World's simplest macro: unless
unless(x >= 1){ println("x was less than 1") }
inline def unless(pred: Boolean)(f: => Unit): Unit = ${ unlessImpl('pred, 'f) }
import scala.quoted.*
def unlessImpl(pred: Expr[Boolean], f: Expr[Unit])(using Quotes): Expr[Unit] =
'{ if (!$pred) $f }
# MACROS
Macro expansion
unless(x >= 1){ println("x was less than 1")}
${ unlessImpl('{ x >= 1 }, '{ println("x was less than 1") }) }
inlining
if (!${'{ x >= 1 }}) ${'{ println("x was less than 1") }}
splice
if (!(x >= 1)) { println("x was less than 1") }
more splices
# MACROS
More interesting macro: factorial
Without a macro:
def factorial(n: Int): Int =
n match {
case 1 => 1
case x => x * factorial(x - 1)
}
Let's move that recursion from runtime to compile time
# MACROS
More interesting macro: factorial
Without a macro:
def factorial(n: Int): Int =
n match {
case 1 => 1
case x => x * factorial(x - 1)
}
Equivalent macro:
def factorialMacro(n: Expr[Int]): Expr[Int] =
n match {
case 1 => '{1}
case x => '{ x * ${factorialMacro(x - 1)} }
}
can't match on an Expr like that
# MACROS
More interesting macro: factorial
def factorialMacro(n: Expr[Int]): Expr[Int] =
n.valueOrError match {
case 1 => '{1}
case x => '{ x * ${factorialMacro(...)} }
}
can't reference value x like that
[error] 14 | case x => '{ x * ${factorialMacro(...)} }
[error] | ^
[error] | access to value x from wrong staging level:
[error] | - the definition is at level 0,
[error] | - but the access is at level 1.
# MACROS
More interesting macro: factorial
def factorialMacro(n: Expr[Int]): Expr[Int] =
n.valueOrError match {
case 1 => '{1}
case x => '{ $x * ${factorialMacro(...)} }
}
nope, that doesn't make sense
[error] 14 | case x => '{ $x * ${factorialMacro(...)} }
[error] | ^
[error] | Found: (x : Int)
[error] | Required: quoted.Expr[Any]
umm... splice it?
# MACROS
More interesting macro: factorial
def factorialMacro(n: Expr[Int]): Expr[Int] =
n.valueOrError match {
case 1 => '{1}
case x => '{ '{x} * ${factorialMacro(...)} }
}
no, that gives us weird nested quotes
[error] 14 | case x => '{ '{x} * ${factorialMacro(...)} }
[error] | ^^^^^^
[error] |value * is not a member of quoted.Expr[Int], ...
ok, quote it?
# MACROS
More interesting macro: factorial
def factorialMacro(n: Expr[Int]): Expr[Int] =
n.valueOrError match {
case 1 => '{1}
case x => '{ ${Expr(x)} * ${factorialMacro(Expr(x - 1))} }
}
we need to lift the static value into a representation
object Expr {
...
/** Creates an expression that will construct the value `x` */
def apply[T](x: T)(using ToExpr[T])(using Quotes): Expr[T] =
scala.Predef.summon[ToExpr[T]].apply(x)
}
# MACROS
More interesting macro: factorial
inline def factorial(n: Int): Int = ${ factorialMacro('n) }
finally we can call our macro!
println(factorial(5)) // prints 120
# MACROS
More interesting macro: factorial
println("Give me a number and I'll calculate its factorial for you")
println(factorial(scala.io.StdIn.readInt()))
can we build factorial-as-a-service?
[error] 8 | println(factorial(scala.io.StdIn.readInt()))
[error] | ^^^^^^^^^^^^^^^^^^^^^^^^
[error] |Expected a known value.
[error] |
[error] |The value of: n$proxy1
[error] |could not be extracted using scala.quoted.FromExpr$PrimitiveFromExpr@4755601c
nope, a macro can't match on a value that's not known until runtime
# STAGED PROGRAMMING
We need... runtime staged programming
- receive user input at runtime
- generate code at runtime, guided by that input
- run the generated code
# STAGED PROGRAMMING
Staged factorial
def factorialStaged(n: Int)(using Quotes): Expr[Int] =
n match {
case 1 => '{1}
case x => '{${Expr(x)} * ${factorialStaged(x - 1)}}
}
import scala.quoted.staging.*
def runFactorialStaged(n: Int): Int =
given Compiler = Compiler.make(getClass.getClassLoader)
run(factorialStaged(n))
# STAGED PROGRAMMING
Macros vs staged programming
Macros
Runtime staged programming
Use quotes 'n' splices...
Use quotes 'n' splices...
to construct a program fragment...
to construct a program fragment...
at compile time...
at runtime...
and inline it at compile time
and then interpret it at runtime
# STAGED PROGRAMMING
Staging example: list membership
def member[A](list: List[A])(a: A): Boolean =
list match {
case Nil => false
case x :: xs => (a == x) || member(xs)(a)
}
# STAGED PROGRAMMING
Staging example: list membership
def member[A](list: List[A])(a: A): Boolean =
list match {
case Nil => false
case x :: xs => (a == x) || member(xs)(a)
}
def memberStaged[A: Type: ToExpr](list: List[A])(a: Expr[A])(using Quotes): Expr[Boolean] =
list match {
case Nil => '{ false }
case x :: xs => '{ ($a == ${Expr(x)}) || ${memberStaged(xs)(a)} }
}
# STAGED PROGRAMMING
Staging example: list membership
def memberStaged[A: Type: ToExpr](list: List[A])(a: Expr[A])(using Quotes): Expr[Boolean] =
list match {
case Nil => '{ false }
case x :: xs => '{ ($a == ${Expr(x)}) || ${memberStaged(xs)(a)} }
}
def stage(list: List[String]): String => Boolean =
given Compiler = Compiler.make(getClass.getClassLoader)
run(
val code: Expr[String => Boolean] = '{ (x: String) => ${memberStaged(list)('x)} }
println("Staged code: " + code.show)
code
)
# STAGED PROGRAMMING
Staging example: list membership
def stage(list: List[String]): String => Boolean =
given Compiler = Compiler.make(getClass.getClassLoader)
run(
val code: Expr[String => Boolean] = '{ (x: String) => ${memberStaged(list)('x)} }
println("Staged code: " + code.show)
code
)
val contains: String => Boolean = stage(List("foo", "bar", "baz"))
// Staged code: ((x: String) => x.==("foo").||(x.==("bar").||(x.==("baz").||(false))))
contains("bar") // true
contains("wow") // false
# STAGED PROGRAMMING
Final example: staged DSL interpreter
Quantified Boolean Formula
# STAGED PROGRAMMING
Let's implement a DSL for QBF
Two main approaches
- Initial encoding (data structure)
- Final encoding (functions)
# STAGED PROGRAMMING
enum QBF:
case Var(name: String)
case And(a: QBF, b: QBF)
case Or(a: QBF, b: QBF)
case Not(a: QBF)
case Implies(ante: QBF, cons: QBF)
case Forall(name: String, a: QBF)
case Exists(name: String, a: QBF)
# STAGED PROGRAMMING
def eval(qbf: QBF, env: Map[String, Boolean]): Boolean =
qbf match {
case Var(name) => env(name)
case And(a, b) => eval(a, env) && eval(b, env)
case Or(a, b) => eval(a, env) || eval(b, env)
case Not(a) => !(eval(a, env))
case Implies(ante, cons) => eval(Or(cons, And(Not(ante), Not(cons))), env)
case Forall(name, a) =>
def check(value: Boolean) = eval(a, env + (name -> value))
check(true) && check(false)
case Exists(name, a) =>
def check(value: Boolean) = eval(a, env + (name -> value))
check(true) || check(false)
}
Single-stage interpreter
def evaluate(qbf: QBF): Boolean = eval(qbf, Map.empty)
# STAGED PROGRAMMING
def evalStaged(qbf: QBF, env: Map[String, Expr[Boolean]])(using Quotes): Expr[Boolean] =
qbf match {
case Var(name) => env(name)
case And(a, b) => '{ ${evalStaged(a, env)} && ${evalStaged(b, env)} }
case Or(a, b) => '{ ${evalStaged(a, env)} || ${evalStaged(b, env)} }
case Not(a) => '{ ! ${evalStaged(a, env)} }
case Implies(ante, cons) => evalStaged(Or(cons, And(Not(ante), Not(cons))), env)
case Forall(name, a) => '{
def check(value: Boolean) = ${evalStaged(a, env + (name -> 'value))}
check(true) && check(false)
}
case Exists(name, a) => '{
def check(value: Boolean) = ${evalStaged(a, env + (name -> 'value))}
check(true) || check(false)
}
}
Staged interpreter
def evaluateStaged(qbf: QBF): Boolean =
given Compiler = Compiler.make(getClass.getClassLoader)
run(evalStaged(qbf, Map.empty))
# STAGED PROGRAMMING
Staged interpreter - example
{
def check(value: scala.Boolean): scala.Boolean =
value.||(value.||(value.unary_!).||(value.unary_!.&&(value.||(value.unary_!).unary_!)))
check(true).&&(check(false))
}
# STAGED PROGRAMMING
Multi-stage programs
If we can construct
Expr[T]
there's nothing* to stop us from constructing
Expr[Expr[T]] Expr[Expr[Expr[T]]]
...
*apart from a desire to preserve our sanity, and a slightly clunky developer experience in Scala 3
# STAGED PROGRAMMING
Multi-stage programs
Examples
- Vector inner product in 3 stages
- PDF (section 2.7)
- Toy example of multi-stage program in Scala 3
# CONCLUSION
Summary
- Principled metaprogramming framework in Scala 3
- Two fundamental operations
- Quotes to construct a future computation
- Splices to perform an immediate computation
- Metaprogramming is useful for optimisation
- Staging allows code generation driven by runtime values
# CONCLUSION
Resources
-
Multi-Stage Programming: Its Theory and Applications
- PhD thesis, so quite long. Chapter 2 is a nice intro to staged programming in MetaML
- Staged list membership example came from here
-
DSL implementation in MetaOCaml, template Haskell, and C++
- QBF example is based on section 2 of this paper
-
Proof of Multi-Stage Programming with Generative and Analytical Macros
- Paper by Nicolas Stucki, implementor of Scala 3's metaprogramming features
- There's also a 12-minute summary video
- A Gentle Introduction to Multi-stage Programming
-
Macros (Scala 3 Language Reference)
- Try to read and understand everything on this page before you start writing macros!
# CONCLUSION
Resources (cont.)
-
Multi-stage Programming in the Large with Staged Classes
- Summary video (the paper is behind an ACM paywall)
- Explores staging at the level of classes/modules rather than individual functions
-
Dependently Typed Multi-Stage Programming, Revisited (video)
- First 15 minutes can help with building intuition about levels/stages
# CONCLUSION
${and let this world no longer be a stage}
Staged programming in Scala 3
By Chris Birchall
Staged programming in Scala 3
- 1,424