Core.async Internals

What is this ?

  • A super quick overview of core.async internals
  • I was looking for different approaches for concurrency and parallelism in Clojure
  • This talk does not cover every details about core.async (not even close)

Topics

  • What is core.async ?
  • How does it works ? 
  • Channels
  • Go Blocks

What is Core.async ?

  • A Clojure/Clojurescript library for async programming.
  • An implementation of CSP in Clojure/Clojurescript.

How does it works ?

Building blocks

  • Channels
  • Channel operations
  • Buffers
  • `thread` macro
  • `go` macro 
  • Threadpools

Channels & Channel Ops

Buffers

`thread` macro

  • It's like Bullet Two
  • It's similar to `future`
  • Creates an actual thread
  • Returns a channel yielding the result
  • Use it for IO operations

`go` macro

  • It's a lightweight thread
  • Returns a channel yielding the result or the block
  • Do not use it for IO operations
  • Under the hood it's really complicated.

Threadpools

  • Execution threadpool
    • Executes `go` blocks
    • newFixedThreadPool
  • `thread` threadpool
    • newCachedThreadPool

Channels

ManyToManyChannel

  • Is the only implementation
  • impl/take!
  • impl/put!
  • impl/close!

What's inside ?

  • Readers Queue
  • Writers Queue
  • Buffer ?
  • Mutex

Rules of implementation

  • There should not be pending readers & writers in the same time.
  • There should not be any pending reads when buffer is not empty.
  • There should not be any bending writes while the buffer has some room.

close!

  • Reads will consume from the buffer and remaining writes.
  • Resolves pending takes with nil
  • Resolves subsequent puts with nil

Queue limits

  • No unbounded queue
  • (def ^:const ^{:tag 'int} MAX-QUEUE-SIZE 1024)
  • Will throw if exceeded

Go Blocks

Overview

  • Magical `go` macro.
  • `go` macro works like a compiler
  • `go` macro break down the clojure code into blocks
  • Core.async creates a state machine from those blocks.
  • The state machine can execute the blocks, pause the execution and move between them.

Compilation

  • Parses the Clojure code into SSA format
  • SSA format represents Clojure code in blocks of instructions
  • Each block has an ID and a state
  • Compile SSA format back to clojure code again.
  • Usually, The compiled clojure code is 50% slower.

Execution

  • Executes blocks in execution threadpool by passing the necessary state to them.
  • Core.asyc can start and stop blocks in any given time.
  • So it can park takes and puts instead of blocking the thread.

That's it

Core.async Internals

By Sameer Rahmani