Inside the IDE: Untangling the magic behind the scenes

What happens inside the IDE ?

case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@

Completion

/** Very good documentation */
case class Test(fieldA: Int, fieldB: Int)

val <<x>> = Test(1, 2)

Hover

case class Test(fieldA: Int, fieldB: Int)

val x = <<Test>>(1, 2)

Go-to Definition

How hard is it really to create an IDE ?

IDE

Editor + Language Support

Editor

Language Support

Editor

  • I/O operations
  • Performance
  • Interactive features
  • Debugger

Language Support

Editor

  • I/O operations
  • Performance
  • Interactive features
  • Debugger

Language Support

???

Language syntax and complexity

class Animal:
    def __init__(self, species):
        self.species = species

    def speak(self):
        print(f"I am a {self.species}")

cat = Animal("cat")
cat.speak()
given Conversion[Int, Long] = 
  (x: Int) => x.toLong
val str: Long = 42

Dynamic vs. Static Typing

def add(a, b):
    return a + b

def add(a: Number, b: Number) =
  a + b

Interoperability with Other Languages

new java.util.ArrayList[Int]()

class BetterJavaList extends java.util.List

Frequency of Language Changes

List(1, 2, 3).map { x => 
  x + x
}
List(1, 2, 3).map: x =>
  x + x
def test(implicit ctx: Context) = ???
def test(using ctx: Context) = ???

Approaches in different languages

Tool Parser Type Checker
Python Language Server built-in 'ast' module custom based on type hints or heuristics
Rust analyzer custom incremental parsing custom type inference engine
TS language server custom parser in the compiler compiler's built-in type checker
Gopls standard parser from Go stdlib go type checker from Go stdlib
Kotlin compiler parser compiler's type checker
IntelliJ Scala Plugin custom parser own type inference engine
Scala Metals compiler parser + scalameta parser compiler's type checker

Custom vs Built-in

  • easy IDE related optimizations
  • has to be updated by maintainers
  • results may diverge from the official language type checker
  • duplicated work
  • requires more resources
  • customization is harder as it has to be done directly in the compiler
  • updated by the compiler team
  • 1-1 alignment with the compiler
  • strictly bound to the compiler version

Custom vs Built-in

  • easy IDE related optimizations
  • has to be updated by maintainers
  • results may diverge from the official language type checker
  • duplicated work
  • requires more resources
  • customization is harder as it has to be done directly in the compiler
  • updated by the compiler team
  • 1-1 alignment with the compiler
  • strictly bound to the compiler version

Metals

EDITOR - LSP - BSP

       -         -

https://slides.com/tomekgodzik/metals-heaven

Lets go deeper

Presentation Compiler

What the F[_] is the Presentation Compiler ???

  • compiler designed to be used by IDE
  • provides high level API

The API

public abstract class PresentationCompiler {
    // type CF = CompletableFuture
    CF<List<Node>> semanticTokens(VirtualFileParams params);
    CF<CompletionList> complete(OffsetParams params);
    CF<CompletionItem> completionItemResolve(CompletionItem item, String symbol);
    CF<SignatureHelp> signatureHelp(OffsetParams params);
    CF<Optional<HoverSignature>> hover(OffsetParams params);
    CF<Optional<Range>> prepareRename(OffsetParams params);
    CF<List<TextEdit>> rename(OffsetParams params, String name);
    CF<DefinitionResult> definition(OffsetParams params);
    CF<DefinitionResult> typeDefinition(OffsetParams params);
    CF<List<DocumentHighlight>> documentHighlight(OffsetParams params);
    CF<String> getTasty(URI targetUri, boolean isHttpEnabled);
    CF<List<AutoImportsResult>> autoImports(String name, OffsetParams params, Boolean isExtension);
    CF<List<TextEdit>> implementAbstractMembers(OffsetParams params);
    CF<List<TextEdit>> insertInferredType(OffsetParams params);
    CF<List<TextEdit>> inlineValue(OffsetParams params)
    CF<List<TextEdit>> extractMethod(RangeParams range, OffsetParams extractionPos);
    CF<List<TextEdit>> convertToNamedArguments(OffsetParams params, List<Integer> argIndices);
    CF<List<Diagnostic>> didChange(VirtualFileParams params);
    void didClose(URI uri);
    CF<byte[]> semanticdbTextDocument(URI filename, String code);
    CF<List<SelectionRange>> selectionRange(List<OffsetParams> params);
}

The beginnings

Scala IDE

First Presentation Compiler

May. 2009

Feb. 2012

First Scala 3 commit

May 2017

Scala 3 Presentation Compiler with Dotty Language Server

May 2020

Scala 3 support in Metals

The beginnings

Scala IDE

May 2017

Scala 3 Presentation Compiler with Dotty Language Server

May 2020

Scala 3 support in Metals

ctrl + c, ctrl + v

First Presentation Compiler

May. 2009

Feb. 2012

First Scala 3 commit

API implementation

override def hover(params: TextDocumentPositionParams) = computeAsync { cancelToken =>
  val uri = new URI(params.getTextDocument.getUri)
  val driver = driverFor(uri)
  implicit def ctx: Context = driver.currentCtx

  val pos = sourcePosition(driver, uri, params.getPosition)
  val trees = driver.openedTrees(uri)
  val path = Interactive.pathTo(trees, pos)
  val tp = Interactive.enclosingType(trees, pos)
  val tpw = tp.widenTermRefExpr

  if (tp.isError || tpw == NoType) null 
  // null here indicates that no response should be sent
  else {
    Interactive.enclosingSourceSymbols(path, pos) match {
      case Nil => null
      case symbols =>
        val docComments = symbols.flatMap(ParsedComment.docOf)
        val content = hoverContent(Some(tpw.show), docComments)
        new Hover(content, null)
    }
  }

API implementation

  override def completion(params: CompletionParams) = computeAsync { cancelToken =>
    val uri = new URI(params.getTextDocument.getUri)
    val driver = driverFor(uri)
    implicit def ctx: Context = driver.currentCtx

    val pos = sourcePosition(driver, uri, params.getPosition)
    val items = driver.compilationUnits.get(uri) match {
      case Some(unit) =>
        val freshCtx = ctx.fresh.setCompilationUnit(unit)
        Completion.completions(pos)(using freshCtx)._2
      case None => Nil
    }

    JEither.forRight(new CompletionList(
      /*isIncomplete = */ false, items.map(completionItem).asJava))
  }

override def hover(params: TextDocumentPositionParams) = computeAsync { cancelToken =>
  val uri = new URI(params.getTextDocument.getUri)
  val driver = driverFor(uri)
  implicit def ctx: Context = driver.currentCtx

  val pos = sourcePosition(driver, uri, params.getPosition)
  val trees = driver.openedTrees(uri)
  val path = Interactive.pathTo(trees, pos)
  val tp = Interactive.enclosingType(trees, pos)
  val tpw = tp.widenTermRefExpr

  if (tp.isError || tpw == NoType) null 
  // null here indicates that no response should be sent
  else {
    Interactive.enclosingSourceSymbols(path, pos) match {
      case Nil => null
      case symbols =>
        val docComments = symbols.flatMap(ParsedComment.docOf)
        val content = hoverContent(Some(tpw.show), docComments)
        new Hover(content, null)
    }
  }

Lets go even deeper

Interactive Compiler

What the F[_] is the Interactive Compiler ???

  • asynchronous
  • interruptible
  • targeted
  • stop after type-checking

Parser

Typer

PostTyper

SetRootTree

Frontend

Pickler

Transform

Backend

InteractiveCompiler

class InteractiveCompiler extends Compiler {
  // TODO: Figure out what phases should be run in IDEs
  // More phases increase latency but allow us to report more errors.
  // This could be improved by reporting errors back to the IDE
  // after each phase group instead of waiting for the pipeline to finish.
  override def phases: List[List[Phase]] = List(
    List(new Parser),
    List(new TyperPhase),
    List(new transform.SetRootTree),
    List(new transform.CookComments)
  )
}

InteractiveDriver

/** A Driver subclass designed to be used from IDEs */
class InteractiveDriver(val settings: List[String]) extends Driver:
  import tpd._

  override def sourcesRequired: Boolean = false

  private val compiler: Compiler = new InteractiveCompiler
  def openedFiles: Map[URI, SourceFile] = myOpenedFiles
  def openedTrees: Map[URI, List[SourceTree]] = myOpenedTrees
  def compilationUnits: Map[URI, CompilationUnit] = myCompilationUnits
  def run(uri: URI, source: SourceFile): List[Diagnostic] = 
    compiler.run(uri, source) // simplification

Interactive

object Interactive {
  import ast.tpd._

  def isDefinition(tree: Tree): Boolean
  def enclosingType(trees: List[SourceTree], pos: SourcePosition)(using Context): Type
  def enclosingTree(trees: List[SourceTree], pos: SourcePosition)(using Context): Tree
  def enclosingTree(path: List[Tree])(using Context): Tree
  def enclosingSourceSymbols(path: List[Tree], pos: SourcePosition)(using Context): List[Symbol]
  def matchSymbol(tree: Tree, sym: Symbol, include: Include.Set)(using Context): Boolean
  def namedTrees(trees: List[SourceTree], include: Include.Set, sym: Symbol)(using Context): List[SourceTree]
  def pathTo(trees: List[SourceTree], pos: SourcePosition)(using Context): List[Tree]
  def contextOfStat(stats: List[Tree], stat: Tree, exprOwner: Symbol, ctx: Context): Context
  def contextOfPath(path: List[Tree])(using Context): Context
  def enclosingDefinitionInPath(path: List[Tree])(using Context): Tree
  def findDefinitions(path: List[Tree], pos: SourcePosition, driver: InteractiveDriver): List[SourceTree]
  def localize(symbol: Symbol, sourceDriver: InteractiveDriver, targetDriver: InteractiveDriver): Symbol
  def implementationFilter(sym: Symbol)(using Context): NameTree => Boolean
  def isRenamed(tree: NameTree)(using Context): Boolean
  def sameName(n0: Name, n1: Name): Boolean
}
case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@

Completion example

  override def completion(params: CompletionParams) = computeAsync { cancelToken =>
    val uri = new URI(params.getTextDocument.getUri)
    val driver = driverFor(uri)
    implicit def ctx: Context = driver.currentCtx

    val pos = sourcePosition(driver, uri, params.getPosition)
    val items = driver.compilationUnits.get(uri) match {
      case Some(unit) =>
        val freshCtx = ctx.fresh.setCompilationUnit(unit)
        Completion.completions(pos)(using freshCtx)._2
      case None => Nil
    }

    JEither.forRight(new CompletionList(
      /*isIncomplete = */ false, items.map(completionItem).asJava))
  }

  def completions(pos: SourcePosition)(using Context): (Int, List[Completion]) =
    val tpdPath = Interactive
      .pathTo(ctx.compilationUnit.tpdTree, pos.span)
    val completionContext = Interactive
      .contextOfPath(tpdPath).withPhase(Phases.typerPhase)
    computeCompletions(pos, mode, rawPrefix, tpdPath)(using completionContext)

Completions

case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@
case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@
package <empty> { [0..86]
  case class Test(fieldA: Int, fieldB: Int) extends Object(),  <17..60>
    _root_.scala.Product, _root_.scala.Serializable {
    val fieldA: Int
    val fieldB: Int
    def copy(fieldA: Int, fieldB: Int): Test.Test =
      new Test(fieldA, fieldB)
    def copy$default$1: Int @uncheckedVariance = Test.this.fieldA
    def copy$default$2: Int @uncheckedVariance = Test.this.fieldB
    def _1: Int = this.fieldA
    def _2: Int = this.fieldB
  }
  final lazy module val Test: Test = new Test() <60..68>
  final module class Test() extends AnyRef() { this: Test.type => <68..75>
    def apply(fieldA: Int, fieldB: Int): Test =
      new Test(fieldA, fieldB)
    def unapply(x$1: Test): Test = x$1
    override def toString: String = "Test"
  }
  val x: Test = Test.apply(1, 2) <75..81>
  x.fie  [81..83..86]
}

We want to find tree at `@@` (position 86)

case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@
List(
  x.fie,
  package <empty> { 
    case class Test(val fieldA: Int, val fieldB: Int) {}
    val x = Test(1, 2)
    x.fie
  }
)
List(
  Select(Ident(x),fie), 
  PackageDef(Ident(<empty>), { 
    case class def + ValDef(x, ...) + Select(Ident(x), fie) 
  }
)

Pretty Printed pathTo(query)

val query = SrcPos(86)

pathTo(query)

case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@
List(
  x.fie,
  package <empty> { 
    case class Test(val fieldA: Int, val fieldB: Int) {}
    val x = Test(1, 2)
    x.fie
  }
)
List(
  Select(Ident(x),fie), 
  PackageDef(Ident(<empty>), { 
    case class def + ValDef(x, ...) + Select(Ident(x), fie) 
  }
)

Pretty Printed pathTo(query)

val query = SrcPos(86)

pathTo(query)

case class Test(fieldA: Int, fieldB: Int)

val x = Test(1, 2)
x.fie@@
List(
  x.fie,
  package <empty> { 
    case class Test(val fieldA: Int, val fieldB: Int) {}
    val x = Test(1, 2)
    x.fie
  }
)
List(
  Select(Ident(x),fie), 
  PackageDef(Ident(<empty>), { 
    case class def + ValDef(x, ...) + Select(Ident(x), fie) 
  }
)

Pretty Printed pathTo(query)

val query = SrcPos(86)

pathTo(query)

  def completions(pos: SourcePosition)(using Context): (Int, List[Completion]) =
    val tpdPath = Interactive
      .pathTo(ctx.compilationUnit.tpdTree, pos.span)
    val completionContext = Interactive
      .contextOfPath(tpdPath).withPhase(Phases.typerPhase)
    computeCompletions(pos, tpdPath)(using completionContext)

Completions

private def computeCompletions(pos: SourcePosition, path: List[Tree])
    (using Context): (Int, List[Completion]) = {
  val path0 = pathBeforeDesugaring(path, pos) 
  // List(Select(Ident(x),fie), PackageDef(Ident(<empty>), ...), ...)
  val mode = completionMode(path0, pos) // Mode.Term
  val rawPrefix = completionPrefix(path0, pos) // "fie"

  val hasBackTick = rawPrefix.headOption.contains('`')
  val prefix = if hasBackTick then rawPrefix.drop(1) else rawPrefix

  val completer = new Completer(mode, prefix, pos)

  val completions = path0 match {
      case Select(qual @ This(_), _) :: _ if qual.span.isSynthetic  => 
        completer.scopeCompletions
                //qual.tpe = TypeDef("Test" ...)
      case Select(qual, _) :: _           if qual.tpe.hasSimpleKind => 
        completer.selectionCompletions(qual)
      case Select(qual, _) :: _                                     => 
        Map.empty
      case (tree: ImportOrExport) :: _                              => 
        completer.directMemberCompletions(tree.expr)
      case (_: untpd.ImportSelector) :: Import(expr, _) :: _        => 
        completer.directMemberCompletions(expr)
      case _                                                        => 
        completer.scopeCompletions
    }
  describeCompletions(completions)

Completions

def selectionCompletions(qual: Tree)(using Context): CompletionMap =
  val adjustedQual = widenQualifier(qual)

  implicitConversionMemberCompletions(adjustedQual) ++
    extensionCompletions(adjustedQual) ++
    directMemberCompletions(adjustedQual)

Completions

/** @param site The type to inspect.
 *  @return The members of `site` that are accessible and pass the include filter.
 */                           // type of x => Test
private def accessibleMembers(site: Type)(using Context): Seq[SingleDenotation] = {
  def appendMemberSyms(name: Name, buf: mutable.Buffer[SingleDenotation]): Unit =
    try
      val member = site.member(name)
      // case class Test(fieldA: Int, fieldB: Int)
      if member.symbol.is(ParamAccessor) && !member.symbol.isAccessibleFrom(site) then
        buf ++= site.nonPrivateMember(name).alternatives
      else
        buf ++= member.alternatives
    catch
      case ex: TypeError =>

  val members = site.memberDenots(completionsFilter, appendMemberSyms).collect {
    case mbr if include(mbr, mbr.name)
                && mbr.symbol.isAccessibleFrom(site) => mbr
  }
  val refinements = extractRefinements(site).filter(mbr => include(mbr, mbr.name))

  members ++ refinements
}

Completions

private def computeCompletions(pos: SourcePosition, path: List[Tree])
    (using Context): (Int, List[Completion]) = {
  { ... }

  val completions = path0 match {
      case Select(qual @ This(_), _) :: _ if qual.span.isSynthetic  => 
        completer.scopeCompletions
                //qual = TypeDef("Test" ...)
      case Select(qual, _) :: _           if qual.tpe.hasSimpleKind => 
        completer.selectionCompletions(qual)
      case Select(qual, _) :: _                                     => 
        Map.empty
      case (tree: ImportOrExport) :: _                              => 
        completer.directMemberCompletions(tree.expr)
      case (_: untpd.ImportSelector) :: Import(expr, _) :: _        => 
        completer.directMemberCompletions(expr)
      case _                                                        => 
        completer.scopeCompletions
    }
    
  describeCompletions(completions)

Completions

def describeCompletions(completions: CompletionMap)(using Context): List[Completion] =
  for
    (name, denots) <- completions.toList
    denot <- denots
  yield
    Completion(name.show, description(denot), List(denot.symbol))

Completions

Metals

Presentation Compiler

Interactive Compiler

Compiler

Compiler

Metals

Presentation Compiler

Interactive Compiler

Compiler

lampepfl/dotty

Compiler

scalameta/metals

Presentation Compiler Tradeoffs

Benefits Drawbacks
1 - 1 relation with the compiler strictly bound to the version
bugs can be fixed in the old Scala versions released from another repository
it can be broken by internal API changes
latest syntax and features may require additional changes thus may not be released immediately

Stable Presentation Compiler

Metals

Presentation Compiler

Interactive Compiler

Compiler

Compiler

lampepfl/dotty

scalameta/metals

Metals

Presentation Compiler

Interactive Compiler

Compiler

Compiler

lampepfl/dotty

scalameta/metals

Stable Presentation Compiler

May 2017

Scala 3 Presentation Compiler with Dotty Language Server

May 2020

Scala 3 support in Metals

ctrl + c, ctrl + v

First Presentation Compiler

May. 2009

Feb. 2012

First Scala 3 commit

Stable Presentation Compiler

May 2017

Scala 3 Presentation Compiler with Dotty Language Server

May 2020

Scala 3 support in Metals

ctrl + c, ctrl + v

June 2023

Scala 3 Presentation Compiler provided by Dotty

First Presentation Compiler

May. 2009

Feb. 2012

First Scala 3 commit

Stable Presentation Compiler

May 2017

Scala 3 Presentation Compiler with Dotty Language Server

May 2020

Scala 3 support in Metals

ctrl + c, ctrl + v

June 2023

Scala 3 Presentation Compiler provided by Dotty

ctrl + c, ctrl + v

First Presentation Compiler

May. 2009

Feb. 2012

First Scala 3 commit

What's the difference ?

Benefits Drawbacks
support for every new Scala version including nightly releases is now supported ! iterations are slower as everything happens in the compiler
1 - 1 relation with the compiler bugfixing / adding new features to already released version is very expensive
always supports latest syntax
internal compiler changes won't break the IDE
API is stabilized and tested with MiMA
easier maintanence

Thank you !

for your contributions

https://www.scala-lang.org/blog/2023/10/17/feedback-wanted-error-messages.html

Slides

https://slides.com/rochala

Do you have any questions ?

Copy of Inside the IDE: Untangling the magic behind the scenes

By rochala

Made with Slides.com

Copy of Inside the IDE: Untangling the magic behind the scenes

  • 93

More from rochala