Effect

Capture

Effect

Oleg Nizhnikov ^ {Evolution, Gaming}

Benefits of IO

reddit.com/r/scala/comments/8ygjcq/can_someone_explain_to_me_the_benefits_of_io/

Direct Style

Direct Style

Capture Checking

Capture Checking

AGENDA

  1. Imaginary Scala 1: Effects

  2. Half-Imaginary Scala 2: Uniqueness

  3. Real Scala: Capture Checking

Effects

Effects

  • Calculation
  • Unlimited recursion
  • Allocating on stack
  • Allocating on heap
  • Reading global variables
  • Aborting execution with an error
  • Operating on locally mutable state
  • Operating on shared mutable state
  • Waiting
  • Concurrent execution
  • Communicating between subprocesses
  • Reading from file system
  • Writing to file system
  • Network communication

Effects

  • Calculation
  • Unlimited recursion
  • Allocating on stack
  • Allocating on heap
  • Reading global variables
  • Aborting execution with an error
  • Operating on locally mutable state
  • Operating on shared mutable state
  • Waiting
  • Concurrent execution
  • Communicating between subprocesses
  • Reading from file system
  • Writing to file system
  • Network communication

Effects

  • Calculation
  • Unlimited recursion
  • Allocating on stack
  • Allocating on heap 
  • Reading global variables
  • Aborting execution with an error
  • Operating on locally mutable state
  • Operating on shared mutable state
  • Waiting
  • Concurrent execution
  • Communicating between subprocesses
  • Reading from file system
  • Writing to file system
  • Network communication

Non-Effects

Effects

Effects

  • Calculation
  • Unlimited recursion
  • Allocating on stack
  • Allocating on heap 
  • Reading global variables
  • Aborting execution with an error
  • Operating on locally mutable state
  • Operating on shared mutable state
  • Waiting
  • Concurrent execution
  • Communicating between subprocesses
  • Reading from file system
  • Writing to file system
  • Network communication

Non-Effects

Effects

Imaginary Scala 1:

Effects

Scala & Effects

def readInt(source: String): Int = 
	source.toIntOption.getOrElse(0)

def readInts(source: String): Vector[Int] = 
	source.split(",").toVector.map(readInt)
def readInt(source: String){Abort}: Int  = 
	source.toIntOption.toAbort

def readInts(source: String){Abort}: Vector[Int]  = 
	source.split(",").toVector.map(readInt)

def readIntsOrEmpty(source: String){}: Vector[Int] = 
	Abort.toDefaultValue(Vector.empty){ readInts(source) }

Scala & Effects

val readInt: String -> {Abort} Int  = 
	source => source.toIntOption.toAbort

val readInts: String -> {Abort} Vector[Int]  = 
	source => source.split(",").toVector.map(readInt)

val readIntsOrEmpty: String -> {} Vector[Int] = 
	source => Abort.toDefaultValue(Vector.empty){ readInts(source) }

Scala & Effects

effect Abort {
  def stop[A]{Abort}: A
}

object Abort {
  def toAbort[A](opt: Option[A]){Abort}: A  = 
    opt match {
      case None    => stop
      case Some(a) => a 
   }

  def toDefaultValue[A](a: A)(expr: ->{Abort} A): A = 
     try expr catch {
        case { Abort.stop[A] } => value
        case { result } => result
     }      
}

Scala & Effects

def parseInt(src: Text){Abort}: Int

def getLine(file: File){IO, Exception}: String 

def parseNextLine(file: File){IO, Exception, Abort}: Int = 
  parseInt(getLine(file))

def openFile(path: Path, mode: Mode){IO, Exception}: Handle

def parseFileInts(path: String){IO, Exception, Abort}: Vector[Int] = {
    val file = openFile(Paths.get(path), READ)
    val l1 = parseNextLine(file)
    val l2 = parseNextLine(file)
    val l3 = parseNextLine(file)
    Vector(l1, l2, l3)
 }

Scala & Effects

Capturing Effects

def sqrt(x: Float){Abort}: Float = 
  if n < 0.0 then Abort.stop else x.sqrt

val sqrtSum: Float -> Float -> {Abort} Float = 
  x => y => sqrt(x) + sqrt(y)
def sqrt(x: Float){Abort}: Float = 
  if n < 0.0 then Abort.stop else x.sqrt

val sqrtSum: Float -> Float -> Float = 
  x =>  {
    val sqrtX = sqrt(x)
  	y => sqrtX + sqrt(y) 
}

Capturing Effects

def sqrt(x: Float){Abort}: Float = 
  if n < 0.0 then Abort.stop else x.sqrt

val sqrtSum: Float -> Float -> Float = 
  x =>  {
    val sqrtX = sqrt(x)
  	y => sqrtX + sqrt(y) 
}
Float -> {} Float -> {Abort} Float
Float -> {Abort} Float -> {} Float
Float ->{Abort} Float ->{Abort} Float

✔

❌

❌

Capturing Effects

def parseInt(src: String){Abort}: Int
def openFile(path: Path, mode: Mode){IO, Exception}: Handle
def getLine(file: File){IO, Exception}: String 

  
val parseFileLines: Path -> Int -> Int = 
	path => {
      val file = openFile(Paths.get(path), READ)
      i => i + parseInt (getLine file)
    }
String -> {} Int -> {IO, Exception, Abort} Int
String -> {IO, Exception, Abort} Int -> {} Int
String ->{IO, Exception} Int ->{IO, Exception, Abort} Int

✔

❌

❌

Capturing Effects

Replacing Monads with effects

Option[A] 
Either[E, A]
Writer[List[W], A]
Reader[R, A]
State[S, A]
Try[A]
IO[A]

{Abort}
{Throw[E]}
{Stream[W]}
{Context[R]}
{Store[S]}
{Exception}
{IO}

Effect handlers

effect GenerateInt {
    def next: {GenerateInt} Int
}

object GenerateInt {
  def const[A, Effs](value: Int)(block: ->{GenerateInt, Effs}){Effs}: A
     try block catch {
        case { GenerateInt.next } -> resume(value)
        case { res } -> res
     }
}
resume : Int -> {GenerateInt, Effs} A

Effect handlers

effect GenerateInt {
    def next: {GenerateInt} Int
}

object GenerateInt {
	def from[A, Effs](start: Int)(expr: ->{GenerateInt, Effs} A){Effs}: A =
       try expr with {
          case { GenerateInt.next } => from(start + 1)(resume(start))
          case { res } -> res
       }
resume : Int -> {GenerateInt, Effs} A

Effect handlers

effect GenerateInt{
    def next: {GenerateInt} Int
}

object GenerateInt {
	def fallBack[A, Effs](value: A)(expr: ->{GenerateInt, Effs} A){Effs}: A =
       try expr with {
          case { GenerateInt.next } => value
          case { res } -> res
       }
resume : Int -> {GenerateInt, Effs} A

Effect handlers

effect GenerateInt{
    def next: {GenerateInt} Int
}

object GenerateInt {
	def withRandom[A, Effs](expr: ->{GenerateInt, Effs} A){Random, Effs}: A = 
       try expr with {
          case { GenerateInt.next } => resume(Random.next)
          case { res } -> res
       }
}
resume : Int -> {GenerateInt, Effs} A

Effect handlers

effect GenerateInt{
    def next: {GenerateInt} Int
}

object GenerateInt {
	def search[A, Effs](values: Seq[Int])(expr: ->{GenerateInt, Effs} Option[A]){Effs}: Option[A] = 
       try expr with {
          case { GenerateInt.next } => 
          	values.view.flatMap(resume).headOption
          case { res } -> res
       }
}

def pythagorean: {GenerateInt} Option[(Int, Int, Int)] = {
    val a = next
    val b = next
    val c = next
    if (a * a) + (b * b) == (c * c) then Some (a, b, c) else None
}

GenerateInt.search(1 to 8)(pythagorean) // Some((+3, +4, +5))

Effect handlers

  • Abstraction and Inversion of Control
  • Error Handling
  • Resource and Scope Control
  • Branching and Backtracking
  • (Async execution)
effect GenerateInt{
    def next: {GenerateInt} Int
}

Effect

  • Composition is hard
  • eDSL
  • No partial handling
  • Same but with aftertaste

vs

Monads

  • Composable
  • Require special runtime
  • Partial handlers
  • Covers a lot of language features

Half-Imaginary Scala 2:

Uniqueness Types

State

case class State[S, A](run: S => (S, A))


case class State[S, A](run: Eval[S => Eval[(S, A)]])


trait State[S, +A]{
   def run(s: S): (S, A)
}

State Monad

object State{
	def apply[S, A](a: A): State[S, A] = s => (s, a)
}

trait State[S, +A]{
   def run(s: S): (S, A)
	
   def map[B](f: A => B): State[S, B] = { s => 
      val (s1, a) = run(s)
      (s1, f(a))
   }
   
   def flatMap[B](f: A => State[S, B]) = { s =>
   	  val (s1, a) = run(s)
      f(a).run(s1) 
   }
   	
}

State Monad

object State{
	def apply[S, A](a: A): State[S, A] = s => (s, a)
    
    def get[S]: State[S, S] = s => (s, s)
    
    def set[S](s: S): State[S, Unit] = _ => (s, ())
    
    def update[S](f: S => S): State[S, Unit] = s => (f(s), ())
}

State Comprehension

def addClient(client: Client): State[WowEnterpriseData, Unit] = for {
    data   <- State.get
    clients = data.clients.updated(client.id, client)
    _      <- State.set(data.copy(clients = clients))
} yield ()

def addProduct(product: Product): State[WowEnterpriseData, Unit] = for {
    data    <- State.get
    products = data.products.updated(product.id, product)
    _       <- State.set(data.copy(products = products))
} yield ()

def addUsage(clientId: ClientId, productId: ProductId): State[WowEnterpriseData, Unit] = for {
    data <- State.get
    usage = data.usage + ((clientId, productId))
    _    <- State.set(data.copy(usage = usage))
} yield ()

def suchEnterpiseLogic(client: Client, product: Product): State[WowEnterpriseData, Unit] = for {
    _ <- addClient(client)
    _ <- addProduct(product)
    _ <- addUsage(client.id, product.id)
} yield ()
    
case class WowEnterpriseData(
    clients: Map[ClientId, Cliend],
    products: Map[ProductId, Product],
    usage: Set[(ClientId, ProductId)]
)

WARNING: THIS IS NOT REAL ENTERPRISE-LEVEL CODE

State Comprehension

clients = data.clients.updated(client.id, client)
data.copy(clients = clients)

products = data.products.updated(product.id, product)
data.copy(products = products)

usage = data.usage + ((clientId, productId))
data.copy(usage = usage)

case class WowEnterpriseData(
    clients: Map[ClientId, Cliend],
    products: Map[ProductId, Product],
    usage: Set[(ClientId, ProductId)]
)

Mutable State

clients = data.clients.updated(client.id, client)
data.clients = clients

products = data.products.updated(product.id, product)
data.products = products

usage = data.usage + ((clientId, productId))
data.usage = usage

class WowEnterpriseData(
    var clients: Map[ClientId, Cliend],
    var products: Map[ProductId, Product],
    var usage: Set[(ClientId, ProductId)]
)

Mutable Danger

def reportState(using MyState): String -> String = {
    name => s"$name current value is ${MyState.count}\n"
}

def compare(using MyState): String = {
    val oldReport = reportState
    for (i <- 0 to 9) {
        MyState.increment()
    }
    val newReport = reportState
    oldReport("old") ++ newReport("new")
}

@main def runMyState(): Unit = MyState.withState{
    println(compare)
}

Mutable Danger

def reportState(using MyState): String -> String = {
    name => s"$name current value is ${MyState.count}\n"
}

def compare(using MyState): String = {
    val oldReport = reportState
    for (i <- 0 to 9) {
        MyState.increment()
    }
    val newReport = reportState
    oldReport("old") ++ newReport("new")
}

@main def runMyState(): Unit = MyState.withState{
    println(compare)
}


// old current value is 10
// new current value is 10

Mutable Danger = Capture

def reportState(using MyState): String -> String = {
    val count = MyState.count
    name => s"$name current value is $count\n"
}

def compare(using MyState): String = {
    val oldReport = reportState
    for (i <- 0 to 9) {
        MyState.increment()
    }
    val newReport = reportState
    oldReport("old") ++ { newReport("new") }
}

@main def runMyState(): Unit = MyState.withState{
    println(compare)
}


// old current value is 0
// new current value is 10

Unique Mutable  State

def doSomething(data: Data): (Result, Data) = 
	val data1 = data.copy(field1 = value1)
    val data2 = data1.copy(field2 = value2)
    (result, data2)

Unique Mutable State

def doSomething(data: Data): (Result, Data) = 
	val data1 = data.copy(field1 = value1)
    val data2 = data1.copy(field2 = value2)
    (result, data2)

there is only one reference to data

def doSomething(data: Data): (Result, Data) = 
	data.field1 = value1
    data.field2 = value2
    (result, data)
  • Automatically rewrite updates to mutation if there is only one reference, a.k.a Functional But In Place (Perseus: Koka + Lean4)


     
  • Track uniqueness in type system, allow direct mutability (Clean)

Unique Mutable State

  • Automatically rewrite updates to mutation if there is only one reference, a.k.a Functional But In Place (Perseus: Koka + Lean4)

    Track uniqueness in type system, allow direct mutability (Clean)
(S) -> (S, A)
(S^) -> A

Unique Mutable State

Uniqueness Types

def reportState(using MyState^): String -> String = {
    name => s"$name current value is ${MyState.count}\n"
}

Uniqueness Types

def reportState(using MyState^): String -> String = {
    name => s"$name current value is ${MyState.count}\n"
}
def reportState(using MyState^): String -> String = {
    val count = MyState.count
    name => s"$name current value is $count\n"
}

Uniqueness as Effect

@capability class IO

object Console {
    def readLine(using IO^): String = ???
    def writeLine(line: String)(using IO^): Unit = ???
}

@capability class File

object File {
    def open[A](name: String, mode: Mode)(use: File^ -> IO^ ?-> A)(using IO^): A = ???
	def getLine(file: File^): String = ???
    def putLine(string: String)(file: File^): String = ???
}

Uniqueness as Effect

def sqrt(x: Float)(using Abort^): Float = 
  if n < 0.0 then Abort.stop else x.sqrt

val sqrtSum: Float -> Float -> Abort^ ?-> Float = 
  x => y => sqrt(x) + sqrt(y)

Uniqueness as Effect

def sqrt(x: Float)(using Abort^): Float = 
  if n < 0.0 then Abort.stop else x.sqrt

val sqrtSum: Float -> Abort^ ?-> Float -> Abort^ ?-> Float = 
  x =>  {
    val sqrtX = sqrt(x)
  	y => sqrtX + sqrt(y) 
}

Uniqueness as Effect

def parseInt(src: String)(using Abort^): Int
def open[A](name: String, mode: Mode)(use: File^ -> IO^ ?-> A)(using IO^): A = ???
def getLine(file: File^)(using IO^, Exceptions^): String 


  
val parseFileLines: Path -> Int -> Int = 
	path => {
      val file = openFile(Paths.get(path), READ)
      i => i + parseInt(getLine file)
    }

Uniqueness 

  • Usual types
  • no resume
  • refer by names or implicits
  • several vars of the same unique type
  • no inference
  • compose unique values in structures

vs

Effect 

  • Separate kind, special syntax
  • continuations in handlers
  • only refer to effect by its methods
  • refer to effect by type
  • effect inference
  • define complex handlers

Composing Unique Values

class CSVReader(
   file: File^,
   header: Vector[String],
   config: CsvConfig,
)

Partial Capture

class CSVReader(
   file: File^,
   header: Vector[String],
   config: CsvConfig,
)

def openCSV(file: File^)(using io: IO^): CSVReader // that captures file but not IO

Lifetimes?

class CSVReader(
   file: File^,
   header: Vector[String],
   config: CsvConfig,
)

def openCSV['a, 'b](file: File^'a)(using io: IO^'b): CSVReader^'a 
// that captures file but not IO

Lifetimes?

class CSVReader(
   file: File^,
   header: Vector[String],
   config: CsvConfig,
)

def openCSV(file: File^)(using io: IO^): CSVReader^{file}
// that captures file but not IO

Scala 3 Effects?

Capture Set

def foo(x: X^, y: Y): Z^{x} = ???

Capture Set

def foo(x: X^, y: Y): Z^{x} = ???

Capture Set

def foo(x: X^, y: Y): Z^{x} = ???
def foo(x: X, y: Y): A ->{x} B = ???
{}           // Nothing
{a, b, c}    // {a} | {b} | {c}
{cap}        // Any
// cap is the only thing that 
// allows capturing unnamed
X^{} <: X^{a, b} <: X^{a, b, c} <: X^{cap}

No Reader

@capability trait Get[+A](val value: A)

object Get:
    def get[A](using get: Get[A]^): A = get.value

No Copying

@capability class State[A](private var value: A)

object State:
    def get[A](using state: State[A]^): A = state.value
    def set[A](value: A)(using state: State[A]^): Unit = state.value = value
    def modify[A](f: A => A)(using state: State[A]^): Unit = 
        state.value = f(state.value)

No Copying

@capability class State[A](private var state: A):
    def get: A = state
    def set(value: A): Unit = state = value
    def modify(f: A => A): Unit = state = f(state)

object State:
    def of[A](using state: State[A]): State[A] = state

No Option, Neither Either

final class Aborted extends RuntimeException("", null, false, false)

type Abort = CanThrow[Aborted]

def getOrElse[T](default: T)(block: => T throws Aborted): T =
	try block
	catch case _: Aborted => default

No Comprehension

def lol(using Abort^, Get[String]^, State[Int]^): String = 
    if State.of.get > 10 then Abort.abort
    State.of.modify(_ + 1)
    s"Name is ${Get.get}"

No Comprehension

val lol: (Abort^, Get[String]^, State[Int]^) ?-> String = 
    if State.of.get > 10 then Abort.abort
    State.of.modify(_ + 1)
    s"Name is ${Get.get}"

Effects?

def open(path: Path, mode: AccessMode)(using IO^, IOExceptions^): Handle
def parseInt(string: String)(using Abort): Int
def getLine(handle: Handle)(using IO^, IOExceptions^): String

def parseFileLines(path: Path)(using IO^, IOExceptions^): Int -> (IO^, IOExceptions^, Abort^) ?-> Int = 
  val file = open(path, READ)
  i => i + parseInt(getLine(file))        
@capability trait Stream[A]:
    def emit(a: A): Unit

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    

def range(from: Int, until: Int)(using stream: Stream[Int]): Unit = 
    if from < until then 
        stream.emit(from)
        range(from + 1, until)

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    

def toList[A](stream: Stream[A]^ ?-> Unit): List[A] = 
    val builder = List.newBuilder[A]
    stream(using new Stream[A]:
        def emit(a: A) = builder += a)
    builder.result()

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    

def toList[A](stream: Stream[A]^ ?-> Unit): List[A] = 
    val builder = List.newBuilder[A]
    given Stream[A] with
        def emit(a: A) = builder += a
    stream
    builder.result()

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    
    
def map[A, B](es: Stream[A]^ ?-> Unit)(f: A -> B)(using bs: Stream[B]^): Unit = 
    es(using new Stream[A]:
        def emit(a: A) = bs.emit(f(a)))

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    
    
def map[A, B](es: Stream[A]^ ?-> Unit)(f: A -> B)(using bs: Stream[B]^): Unit = 
    es(using new Stream[A]:
        def emit(a: A) = bs.emit(f(a)))

def flatMap[A, B](es: Stream[A]^ ?-> Unit)(f: A -> Stream[B]^ ?-> Unit)(using bs: Stream[B]^): Unit =
    es(using new Stream[A]:
        def emit(a: A) = f(a)(using new Stream[B]:
            def emit(b: B) = bs.emit(b)))

Generators

@capability trait Stream[A]:
    def emit(a: A): Unit
    
    
def map[A, B](es: Stream[A]^ ?-> Unit)(f: A -> B)(using bs: Stream[B]^): Unit = 
    es(using new Stream[A]:
        def emit(a: A) = bs.emit(f(a)))

def flatMap[A, B](es: Stream[A]^ ?-> Unit)(f: A -> Stream[B]^ ?-> Unit)(using bs: Stream[B]^): Unit =
    es(using new Stream[A]:
        def emit(a: A) = f(a)(using new Stream[B]:
            def emit(b: B) = bs.emit(b)))
    
def filter[A](es: Stream[A]^ ?-> Unit)(p: A -> Boolean)(using bs: Stream[A]^): Unit = 
    es(using new Stream[A]:
        def emit(a: A) = if p(a) then bs.emit(a))

Generators

Stream.toList{
    Stream.range(1, 6)
    Stream.range(10, 16)
    Stream.range(20, 24)
    } // List(1, 2, 3, 4, 5, 10, 11, 12, 13, 14, 15, 20, 21, 22, 23)
    
Stream.toList{
    for(i <- 1 to 5) 
    	Stream.emit(i * i)
    } // List(1, 4, 9, 16, 25)

Generators

Capabilities

  • Reusing all the syntactical benefits of implicit (and problems)
    • Mark effects via using or  ?->
    • No inference, only manual annotations
    • Define and compose handlers as 1st class values
    • Use contextual lambda blocks
  • Embrace old-school imperative programming that's could be much more safe now
    • Replace for comprehensions with imperative sequencing
    • Use mutability in capability handlers
    • Use CanThrow for exceptions
    • Explore Loom's virtual threads for concurrency

Thank^{you}

odomontois@gmail.com

onizhnikov@evolution.com

Platform Developer - Stockholm

Game Server Developer - Stockholm

Game Server Developer – Gothenburg

Effects & Capture Stockholm

By Oleg Nizhnik

Effects & Capture Stockholm

  • 278

More from Oleg Nizhnik