A deep dive into scalac

Chris Birchall

ScalaMatsuri

30th January 2016

• The journey from Foo.scala to Foo.class

• The scalac codebase

• Extending the compiler with plugins

• Hacking on the compiler itself

Agenda

• Foo.scala から Foo.class への旅

• scalac のコードベース

• プラグインでコンパイラを拡張してみた

• コンパイラ自体をハックしてみた

me me me

• ScalaCache
• Macro nerd
• @cbirchall
• github.com/cb372

• theguardian.com のバックエンド開発

• エムスリー卒

• ScalaMatsuri 委員会卒
• マクロおじさん

• Anthropological interest

• Debugging slow compilation

• Writing macros

• Contributing to Scala

Why is this useful?

なぜコンパイラを勉強する？

• 歴史のある大規模Scalaコード → 面白い勉強ネタ
• 遅いコンパイルの調査
• マクロを書く時に役立つ
• Scalaへ貢献：バグ修正、上手なバグ報告

The journey

from Foo.scala to Foo.class

Terminology

The compiler deals with 3 kinds of stuff

• Trees

• Types

• Symbols

コンパイラ用語

• 構文木

• 型

• シンボル

Tree

Abstract Syntax Tree (AST)

scala> import scala.reflect.runtime.universe._
import scala.reflect.runtime.universe._

scala> val tree = q"a + b"
tree: reflect.runtime.universe.Tree = a.$plus(b)

scala> showRaw(tree)
res0: String = 
  Apply(
    Select(
      Ident(TermName("a")), 
      TermName("$plus")
    ), 
    List(Ident(TermName("b")))
  )

Type

The, erm, type of a thing?

Scalaユーザなら「型」の意味は分かるはず

Symbol

A binding between a "thing" (e.g. a method, a class or a type) and the name for that thing

プログラム内のクラス、メソッド、型などを指す名前

Compilation (1/2)

構文木

変換

構文木

型付き構文木

シンボルテーブル

Compilation (2/2)

中間表現

$ scalac -Xshow-phases
    phase name  id  description
    ----------  --  -----------
        parser   1  parse source into ASTs, perform simple desugaring
         namer   2  resolve names, attach symbols to named trees
packageobjects   3  load package objects
         typer   4  the meat and potatoes: type the trees
        patmat   5  translate match expressions
superaccessors   6  add super accessors in traits and nested classes
    extmethods   7  add extension methods for inline classes
       pickler   8  serialize symbol tables
     refchecks   9  reference/override checking, translate nested objects
       uncurry  10  uncurry, translate function values to anonymous classes
     tailcalls  11  replace tail calls by jumps
    specialize  12  @specialized-driven class and method specialization
 explicitouter  13  this refs to outer pointers
       erasure  14  erase types, add interfaces for traits
   posterasure  15  clean up erased inline classes
      lazyvals  16  allocate bitmaps, translate lazy vals into lazified defs
    lambdalift  17  move nested functions to top level
  constructors  18  move field definitions into constructors
       flatten  19  eliminate inner classes
         mixin  20  mixin composition
       cleanup  21  platform-specific cleanups, generate reflective calls
    delambdafy  22  remove lambdas
         icode  23  generate portable intermediate code
           jvm  24  generate JVM bytecode
      terminal  25  the last phase during a compilation run

Why is my compile so slow?

$ scalac -verbose Foo.scala 2>&1 | grep "\[\w* in \d*ms\]"
[parser in 19ms]
[namer in 70ms]
[packageobjects in 0ms]
[typer in 570ms]
[patmat in 204ms]
[superaccessors in 12ms]
[extmethods in 3ms]
[pickler in 14ms]
[refchecks in 54ms]
...
[cleanup in 4ms]
[delambdafy in 0ms]
[jvm in 167ms]
[total in 2391ms]

コンパイルはどこが時間かかってる？

What is phase X doing?

object PatternMatch {
  def isGreeting(word: String) = word match {
    case "hello" => true
    case _ => false
  }
}

このフェーズは何してくれる？

Example: patmat

Before patmat phase

$ scalac -Xprint:typer PatternMatch.scala
[[syntax trees at end of                     typer]]
package <empty> {
  object PatternMatch extends scala.AnyRef {
    def <init>(): PatternMatch.type = {
      PatternMatch.super.<init>();
      ()
    };
    def isGreeting(word: String): Boolean = word match {
      case "hello" => true
      case _ => false
    }
  }
}

After patmat phase

$ scalac -Xprint:patmat PatternMatch.scala
// ...
    def isGreeting(word: String): Boolean = {
      case <synthetic> val x1: String = word;
      case5(){
        if ("hello".==(x1))
          matchEnd4(true)
        else
          case6()
      };
      case6(){
        matchEnd4(false)
      };
      matchEnd4(x: Boolean){
        x
      }
    }

The code

scalacのコードを見よう

History

• 2004 - Scala 1.0, compiler written in Java

• 2006 - Scala 2.0, nsc

• 2011-2012 - .Net backend

• 2013 - Scala.js

• (2.12) - new backend and optimizer

scalac の歴史

• 2004年：Scala 1.0（Java実装）
• 2006年：Scala 2.0（完全書き直し、Scala実装）
• 2011〜2012年：.Net 対応の実験
• 2013年：Scala.jsの登場
• 2.12：中間表現を使用しない新バックエンド

Size

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Scala                           321          12304          21416          63852
Bourne Again Shell                1             26             41            150
XML                               2              0              0             26
--------------------------------------------------------------------------------
SUM:                            324          12330          21457          64028
--------------------------------------------------------------------------------

(Scala 2.11.7)

mi casa es su casa

class Global {

  // ...

  lazy val analyzer = new {
    val global: Global.this.type = Global.this
  } with Analyzer

}

trait Analyzer extends ... {
  val global : Global
  import global._

  // ...
}

Walkthrough

simplified version of Global.Run.compile(files: List[String])

1. calculate order of phases
2. for each phase
   1. for each file
      1. run phase
   2. print debug stuff
   3. sanity check trees
3. print any errors

1. フェーズ順を計算

2. フェーズ毎に

   1. ファイル毎に

      1. フェーズを実行

   2. デバッグ情報を出力

   3. ツリーの整合性を確認

3. コンパイルエラーを出力

Phase

abstract class GlobalPhase(prev: Phase) extends Phase(prev) {

  // ...

  def apply(unit: CompilationUnit): Unit

}

CompilationUnit ＝１ソースファイルに該当し、ツリーや他のデータ構造を持つ

CompilationUnit に対してミュータブルな変更を行うため、戻り値は Unit

Transformer

class InvertingTransformer extends Transformer {

  def transform(tree: Tree): Tree = {
    val newTree = super.transform(tree)
    newTree match {
      case Literal(Constant(true)) => Literal(Constant(false))
      case Literal(Constant(false)) => Literal(Constant(true))
      case _ => newTree
    }
  }

}

構文木の変換に手伝うユーティリティクラスが色々ある

Plugins

プラグインの事例

• scoverage（テストカバレッジ測定ツール）
• Scala.js（JS用の中間表現を生成する）
• macro paradise（マクロ・アノテーションなど）
• sbtのインクリメンタルコンパイル（微妙にプラグインではないけど）

class MyPlugin(val global: Global) extends Plugin {

  val components = List[PluginComponent](Component)

  private object Component extends PluginComponent {
    val runsAfter = List("typer")
    def newPhase(prev: Phase) = new MyPhase(prev)

    class MyPhase(prev: Phase) extends StdPhase(prev) {

      def apply(unit: CompilationUnit): Unit = {
        // TODO: MAGIC GOES HERE
      }

    }
  }
}

BASIC!

Example

github.com/cb372/

scalac-plugin-basic

Let's hack!

$ git clone git@github.com:scala/scala.git && cd scala
# ... hack hack hack ...
$ echo "locker.skip=1" > build.properties
$ echo "docs.skip=1" >> build.properties
$ ant publish-core-local -Dmaven.version.suffix="-chris"
# ... go and make a cup of tea ...
# ...
$ cd ~/tmp
$ echo 'scalaVersion := "2.11.7-chris"' > build.sbt
$ echo 'resolvers += Resolver.mavenLocal' >> build.sbt
$ sbt console

Let's hack!

SLIP #28

My (trolling) contribution ...

私の（釣り）貢献 …

Let's hack!

Why stop at adding a json AST?

Everybody loves XML literals,

so why don't we have JSON literals?

scala> val foo = json [ { "a": "b", "c": 123, "d": [ true ]}, 1.23e+5 ]

標準ライブラリにJSON ASTはいいけど、

JSONリテラルもあれば素敵ではないですか？

だって、みんなXMLリテラル大好きでしょ？

1. Add json AST to stdlib

// src/library/scala/json/AST.scala
package scala.json

object AST {

  sealed trait JValue

  case object JNull extends JValue

  case class JString(s: String) extends JValue

  sealed trait JNumber
  case class JDecimal(num: BigDecimal) extends JValue with JNumber
  case class JInt(num: BigInt) extends JValue with JNumber

  case class JBool(value: Boolean) extends JValue
  
  // ...

}

2. Add keyword, token

// src/reflect/scala/reflect/internal/StdNames.scala

    // ...
    final val VIEWBOUNDkw: TermName = kw("<%")
    final val SUPERTYPEkw: TermName = kw(">:")
    final val HASHkw: TermName      = kw("#")
    final val ATkw: TermName        = kw("@")
    final val JSONkw: TermName      = kw("json")  // (*￣0￣)/

// src/compiler/scala/tools/nsc/ast/parser/Tokens.scala

  // ...
  final val VIEWBOUND = 136
  final val NEWLINE = 137
  final val NEWLINES = 138
  final val XMLSTART = 139
  final val JSONSTART = 140  // ヾ(＠⌒ー⌒＠)ノ

3. Add JSON parser

class JsonParser(parser: SourceFileParser) {

  def jLiteral: Tree = {
    val jvalue: JValue = parse()
    TreeBuilder.toTree(jvalue)
  }

  // ...

  private def parse(): JValue = {
    parser.in.nextToken()
    parser.in.token match {
      case LBRACE =>
        stack.push(JObject(Nil))
        parse()
      case RBRACE =>
        // ...
  }

}

4. Wire it together

// src/compiler/scala/tools/nsc/ast/parser/Parsers.scala

private[this] val jsonp = new JsonParser(this)

def jsonLiteral() : Tree = jsonp.jLiteral

def expr0(location: Location): Tree = in.token match {
  case IF => // ...
  case TRY => // ...
  case WHILE => // ...
  // ...
  case JSONSTART => jsonLiteral()
  // ...
}

DEMO

github.com/cb372/

scala/tree/json-literals

Further reading

• Symbols, trees and types

  • http://docs.scala-lang.org/overviews/reflection/symbols-trees-types.html

• ​Some very old videos by Martin Odersky

  • ​http://www.scala-lang.org/old/node/598.html​
• Writing Scala compiler plugins
  • http://www.scala-lang.org/old/node/140.html​
• Scalable Component Abstractions
  • http://lampwww.epfl.ch/~odersky/papers/ScalableComponent.pdf
• New backend and optimizer
  • ​​http://magarciaepfl.github.io/scala/

• ​Dotty: exploring the future of Scala

  • ​https://d-d.me/scalaworld2015/

Final remarks

Questions?

Bonus content

(time permitting)

Another example: tail-call optimization

import scala.annotation.tailrec

object Fib {
  def fib(n: Int) = {
    @tailrec
    def fibRec(n: Int, a:Int, b:Int): Int = n match {
      case 0 => a
      case _ => fibRec(n-1, b, a+b)
    }
    fibRec(n, 0, 1)
  }
}

Before tailcalls

$ scalac -Xprint:uncurry Fib.scala
// ...
def fib(n: Int): Int = {
  @scala.annotation.tailrec def fibRec(n: Int, a: Int, b: Int): Int = {
    case <synthetic> val x1: Int = n;
    x1 match {
      case 0 => a
      case _ => fibRec(n.-(1), b, a.+(b))
    }
  };
  fibRec(n, 0, 1)
}

After tailcalls

$ scalac -Xprint:tailcalls Fib.scala
// ...
def fib(n: Int): Int = {
  @scala.annotation.tailrec def fibRec(n: Int, a: Int, b: Int): Int = {
    <synthetic> val _$this: Fib.type = Fib.this;
    _fibRec(_$this: Fib.type, n: Int, a: Int, b: Int){
      {
        case <synthetic> val x1: Int = n;
        x1 match {
          case 0 => a
          case _ => _fibRec(Fib.this, n.-(1).asInstanceOf[Int](), b.asInstanceOf[Int](), a.+(b).asInstanceOf[Int]())
        }
      }.asInstanceOf[Int]()
    }
  };
  fibRec(n, 0, 1)
}

Let's look at the tree

[[syntax trees at end of tailcalls]]// Scala source: Fib.scala
PackageDef(
  "<empty>" // final package <empty>, tree.tpe=<empty>.type
  ClassDef( // class Fib extends Object
    <module>
    "Fib"
    []
    Template( // val <local Fib>: <notype> in object Fib, tree.tpe=Fib.type
      "java.lang.Object" // parents
      ValDef(
        private
        "_"
        <tpt>
        <empty>
      )
      // 2 statements
      DefDef( // def <init>(): Fib.type in object Fib
        <method>
        "<init>"
        []
        List(Nil)
        <tpt> // tree.tpe=Fib.type
        Block( // tree.tpe=Unit
          Apply( // def <init>(): Object in class Object, tree.tpe=Object
            Fib.super."<init>" // def <init>(): Object in class Object, tree.tpe=()Object
            Nil
          )
          ()
        )
      )
      DefDef( // def fib(n: Int): Int in object Fib
        <method>
        "fib"
        []
        // 1 parameter list
        ValDef( // n: Int
          <param> <triedcooking>
          "n"
          <tpt> // tree.tpe=Int
          <empty>
        )
        <tpt> // tree.tpe=Int
        Block( // tree.tpe=Int
          DefDef( // def fibRec(n: Int,a: Int,b: Int): Int
            <method> <triedcooking> @{ scala.annotation.tailrec }
            "fibRec"
            []
            // 1 parameter list
            ValDef( // n: Int
              <param> <triedcooking>
              "n"
              <tpt> // tree.tpe=Int
              <empty>
            )
            ValDef( // a: Int
              <param> <triedcooking>
              "a"
              <tpt> // tree.tpe=Int
              <empty>
            )
            ValDef( // b: Int
              <param> <triedcooking>
              "b"
              <tpt> // tree.tpe=Int
              <empty>
            )
            <tpt> // tree.tpe=Int
            Block( // tree.tpe=Int
              ValDef( // val _$this: Fib.type
                <synthetic>
                "_$this"
                <tpt> // tree.tpe=Fib.type
                This("Fib")class Fib extends Object, tree.tpe=Fib.type
              )
              LabelDef( // def _fibRec(x$1: Fib.type,n: Int,a: Int,b: Int): Int, tree.tpe=Int
                // 4 paramss
                "_$this" // val _$this: Fib.type, tree.tpe=Fib.type
                "n" // n: Int, tree.tpe=Int
                "a" // a: Int, tree.tpe=Int
                "b" // b: Int, tree.tpe=Int
                Apply( // final def asInstanceOf[T0](): T0 in class Any, tree.tpe=Int
                  TypeApply( // final def asInstanceOf[T0](): T0 in class Any, tree.tpe=()Int
                    {
  case <synthetic> val x1: Int = n;
  x1 match {
    case 0 => a
    case _ => _fibRec(Fib.this, n.-(1).asInstanceOf[Int](), b.asInstanceOf[Int](), a.+(b).asInstanceOf[Int]())
  }
}."asInstanceOf" // final def asInstanceOf[T0](): T0 in class Any, tree.tpe=[T0]()T0
                    <tpt> // tree.tpe=Int
                  )
                  Nil
                )
              )
            )
          )
          Apply( // def fibRec(n: Int,a: Int,b: Int): Int, tree.tpe=Int
            "fibRec" // def fibRec(n: Int,a: Int,b: Int): Int, tree.tpe=(n: Int, a: Int, b: Int)Int
            // 3 arguments
            "n" // n: Int, tree.tpe=Int
            0
            1
          )
        )
      )
    )
  )
)

$ scalac -Xprint:tailcalls -Yshow-trees Fib.scala

LabelDef( // def _fibRec(x$1: Fib.type,n: Int,a: Int,b: Int): Int, tree.tpe=Int
    // 4 paramss
    "_$this" // val _$this: Fib.type, tree.tpe=Fib.type
    "n" // n: Int, tree.tpe=Int
    "a" // a: Int, tree.tpe=Int
    "b" // b: Int, tree.tpe=Int

...and the bytecode

$ javap -private -c Fib$.class

  // ...

  private final int fibRec$1(int, int, int);
    Code:
       0: iload_1
       1: istore        5
       3: iload         5
       5: tableswitch   { // 0 to 0
                     0: 37
               default: 24
          }
      24: iload_1
      25: iconst_1
      26: isub
      27: iload_3
      28: iload_2
      29: iload_3
      30: iadd
      31: istore_3
      32: istore_2
      33: istore_1
      34: goto          0
      37: iload_2
      38: ireturn

In general

$ scalac -Xprint:<previous-phase> \
    [-Xprint-types] \
    [-Yshow-trees] \
    [-Yshow-syms] \
    [-Xshow-class Foo] \
    Foo.scala

$ scalac -Xprint:<phase-you-care-about> \
    [-Xprint-types] \
    [-Yshow-trees] \
    [-Yshow-syms] \
    [-Xshow-class Foo] \
    Foo.scala