Pneumatico

Il client IqBus scritto in Scala

Stato Di Cose

  • Clients:
    • "Pricing", Akka & Akka Streams
    • "TE", Akka & Netty
    • Analytics ?!
    • QA ?!
  • API improvements
    • Unused methods
    • Unclear methods
  • Typelevel stack
    • Cats, Circe
    • Cats Effect 1.0.0
    • Fs2 1.0.0

Ordine Del Giorno

  • Socket
  • Low Level
    • Connection
    • Publisher
    • Subscriber
  • High Level
    • Client
    • Publication
    • Subscription
    • Request/Response

Socket, take 1

trait Socket[F[_]] {
  // Reads exactly `numBytes` from the peer in a single chunk.
  def readN(numBytes: Int, timeout: Option[FiniteDuration]): F[Option[Chunk[Byte]]]

  // Writes `bytes` to the peer.
  def write(bytes: Chunk[Byte], timeout: Option[FiniteDuration]): F[Unit]
}

WTF is `F[_]`?!

F[_], explained

trait BlockSocket {
  def readN(numBytes: Int): Option[Chunk[Byte]]
  def write(bytes: Chunk[Byte]): Unit
}
trait CallbackSocket {
  def readN(numBytes: Int, cb: Option[Chunk[Byte]] => Unit): Unit
  def write(bytes: Chunk[Byte], cb: Unit => Unit): Unit
}
trait FutureSocket {
  def readN(numBytes: Int): Future[Option[Chunk[Byte]]]
  def write(bytes: Chunk[Byte]): Future[Unit]
}

F[_], explained

type Id[A] = A

trait BlockSocket {
  def readN(numBytes: Int): Id[Option[Chunk[Byte]]]
  def write(bytes: Chunk[Byte]): Id[Unit]
}
type Callback[A] = (A => Unit) => Unit

trait CallbackSocket {
  //  readN(numBytes: Int, cb: Option[Chunk[Byte]] => Unit): Unit
  //  readN(numBytes: Int)(Option[Chunk[Byte]] => Unit): Unit
  //  readN(numBytes: Int): ((Option[Chunk[Byte]] => Unit) => Unit)
  def readN(numBytes: Int): Callback[Option[Chunk[Byte]]]
  def write(bytes: Chunk[Byte]): Callback[Unit]
}
trait FutureSocket {
  def readN(numBytes: Int): Future[Option[Chunk[Byte]]]
  def write(bytes: Chunk[Byte]): Future[Unit]
}

F[_], explained

trait Socket[F[_]] {
  def readN(numBytes: Int): F[Option[Chunk[Byte]]]
  def write(bytes: Chunk[Byte]): F[Unit]
}

object Socket {
  // blocking synchronous socket
  def blocking: Socket[Id] = new Socket[Id] {
    def readN(numBytes: Int): Id[Option[Chunk[Byte]]] = ???
    def write(bytes: Chunk[Byte]): Id[Unit]           = ???
  }
 
  // asynchronous callback socket
  def async: Socket[Callback] = new Socket[Callback] {
    def readN(numBytes: Int): Callback[Option[Chunk[Byte]]] = ???
    def write(bytes: Chunk[Byte]): Callback[Unit]           = ???
  }

  // asynchronous future socket
  def future: Socket[Future] = new Socket[Future] { ... }
}

Socket, take 2

trait Socket[F[_]] {
  // Reads exactly `numBytes` from the peer in a single chunk.
  def readN(numBytes: Int, timeout: Option[FiniteDuration]): F[Option[Chunk[Byte]]]

  // Writes `bytes` to the peer.
  def write(bytes: Chunk[Byte], timeout: Option[FiniteDuration]): F[Unit]
}

Transmission Control Protocol

TCP provides reliable, ordered, and error-checked delivery of a stream of octets (bytes) between applications running on hosts communicating via an IP network.

TCP does not provide framing.

Socket, take 2

Framing

  • Control Characters: `[data] ++ [\cr\lf] ++ [data] ++ [\cr\lf] ++ ...`
  • Frame size: `[size] ++ [data] ++ [size] ++ [data] ++ ...`
  • Why not both? `[size] ++ [data] ++ [\cr\lf] ++ ...`

Codec

  • Protocol Buffers
  • IqBus Binary Format
  • Hence, `Speech`
PacketWrapper {
  int totalPacketLength;   ///< total length of packet
  int checksum;            ///< checksum of whole packet
  int correlationId;       ///< [optional] additional id
                           ///< to be used to identify response
  byte packetType;         ///< type of packet
  byte[] data;
}

Socket, readN

def read[F[_]: FlatMap](S: Socket[F[_]], 
                        timeout: Option[FiniteDuration]): F[Speech] = for {
  Some(bs) <- S.readN(4, timeout) if bs.size == 4
  size     <- readSize(bs) // Little Endian, Codec Dependent
  Some(bm) <- S.readN(size, timeout) if bm.size == size
  speech   <- readMessage(bm) 
} yield speech

Socket, write

def write[F[_]: FlatMap](S: Socket[F[_]],
                         speech: Speech,
                         timeout: Option[FiniteDuration]): F[Unit] = for {
  bm <- writeMessage(speech)
  bs <- writeSize(bm) // Little Endian, Codec Dependent
  _  <- S.write(bs)
  _  <- S.write(bm)
} yield ()

Low Level

  • Connection
  • Publisher
  • Subscriber

Connection, take 1

trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Speech]]
}

WTF is...

  • Stream[F, _]?!
  • Sink[F, _]?!
  • Stream[F, Stream[F, _]]????!

fs2 Streams

Stream[F[_], A]

  • discrete, potentially unbounded stream of `A`
  • may require evaluation in `F[_]`
  • pull-based
  • able to transform itself with:
type Pipe[F[_], A, B] = (Stream[F, A]) => Stream[F, B]

type Sink[F[_], A]    = Pipe[F, A, Unit]

Stream[F, Stream[F, _]]

How to recover?

  • Fail fast?
    Will need to recreate client from scratch
  • Silently?
    Still need some event stream, because IQBus do not handle reconnects and/or crashes
  • Hence, stream of streams -- will interrupt inner stream in case of failure and provide new one.

Connection, take 2

trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Correlation[F, Speech, Speech]]]
}

WTF is Correlation[F, _, _]?!

PacketWrapper {
  int totalPacketLength;   ///< total length of packet
  int checksum;            ///< checksum of whole packet
  int correlationId;       ///< [optional] additional id
                           ///< to be used to identify response
  byte packetType;         ///< type of packet
  byte[] data;
}
trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Speech]]
}

What is correlation?

  • Client send/receives data to/from IqBus in no particular order
  • Needs reply to `InitRequest`, `SubscribeRequest`, etc.
  • ... sometimes don't (`Message`, post 4.0)
  • Needs global counter
  • Not "type safe"
  • We need to somehow "remove" correlated data from `incoming` stream and deliver directly

Correlator

trait Correlation[F[_], +A, +B] {
  def data[A1 >: A]: A1
  def codata[B1 >: B]: Option[F[B1]]
}

final case class Absent[F[_], A](a: A)
  extends Correlation[F, A, Nothing] { ... }

final case class Affirmed[F[_], A, B](a: A, da: Deferred[F, B]) 
  extends Correlation[F, A, B] { ... }
abstract class Correlator[F[_]] {
  def index[A, B](ca: Correlation[F, A, B]): F[(A, Option[Int])]
  def split[A](ci: (A, Option[Int])): F[Option[A]]
}
  • read -> split -> collect Some(_) -> incoming
  • outgoing -> index -> write

Connection

trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Correlation[F, Speech, Speech]]]
}

Publisher

trait IQBusPublisher[F[_]] {
  def properties: IQBusPublisher.Properties

  def publish: Sink[F, Dispatch.Outbound[F]]
  def publish1(d: Dispatch.Outbound[F]): F[Option[MessageResponse]]
}

object IQBusPublisher {
  final case class Properties(kind: MessageKind,
                              name: String,
                              version: Version,
                              parameters: Parameters = Parameters())
}

WTF is `Dispatch.Outbound[F]`?!

Outbound data

abstract class Outbound[F[_]] extends Dispatch {
  def modify(f: Parameters => Parameters): Outbound[F]
  def message(kind: MessageKind,
              name: String,
              version: Version,
              defaults: Parameters = Parameters()): F[Messaging.Message]
}
trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Speech]]
}
def outbound[F[_], A, B](
    data: Option[A],
    meta: Option[B],
    parameters: Parameters = Parameters(),
    status: Status = 2000,
    uid: => UUID = UUID.randomUUID()
)(implicit F: Sync[F], A: Encoder[A, String], B: Encoder[B, String])
: F[Outbound[F]] = ???

Publisher

trait IQBusPublisher[F[_]] {
  def publish: Sink[F, Dispatch.Outbound[F]]
  def publish1(d: Dispatch.Outbound[F]): F[Option[MessageResponse]]
}

Connection

trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Correlation[F, Speech, Speech]]]
}

Subscriber

trait IQBusSubscriber[F[_]] {
  def properties: IQBusSubscriber.Properties
  def subscription: Signal[F, Option[Long]]

  def subscribe(buffer: Int): Stream[F, Dispatch.Inbound[F]]
  def subscribe1(f: Dispatch.Inbound[F] => F[Unit]): F[CancelToken[F]]
}

object IQBusSubscriber {
  final case class Properties(kind: MessageKind,
                              name: String,
                              version: Version,
                              service: Option[String],
                              parameters: Parameters = Parameters(),
                              matchers: Matchers = Matchers())
}

WTF is `Dispatch.Inbound[F]`?!

Inbound data

abstract class Inbound[F[_]] extends Dispatch {
  def message: Messaging.Message

  def data[A: Decoder[String, ?]]: F[Option[A]]
  def meta[B: Decoder[String, ?]]: F[Option[B]]
}

def inbound[F[_]: Concurrent](message: Messaging.Message): F[Inbound[F]] = ???

Publisher

trait IQBusPublisher[F[_]] {
  def publish: Sink[F, Dispatch.Outbound[F]]
  def publish1(d: Dispatch.Outbound[F]): F[Option[MessageResponse]]
}

Connection

trait IQBusConnection[F[_]] {
  def incoming: Stream[F, Stream[F, Speech]]
  def outgoing: Stream[F, Sink[F, Correlation[F, Speech, Speech]]]
}

Subscriber

trait IQBusSubscriber[F[_]] {
  def subscribe(buffer: Int): Stream[F, Dispatch.Inbound[F]]
  def subscribe1(f: Dispatch.Inbound[F] => F[Unit]): F[CancelToken[F]]
}

High Level

  • Client
  • Publication
  • Subscription
  • Request/Response

Client, take 1

trait IQBus[F[_]] {
  def address: Address.Full
  def config: cfg.Config

  def publisher(props: IQBusPublisher.Properties): Resource[F, IQBusPublisher[F]]
  def subscriber(props: IQBusSubscriber.Properties): Resource[F, IQBusSubscriber[F]]
}

WTF is `Resource[F, A]`?!

Resource[F, A]

sealed trait Resource[F[_], A] {
  def use[B](f: A => F[B])(implicit F: Bracket[F, Throwable]): F[B]
}

object Resource {
  def apply[F[_]: Functor, A](resource: F[(A, F[Unit])]): Resource[F, A] = ???
  def make[F[_]: Functor, A](acquire: F[A])(release: A => F[Unit]): Resource[F, A] = ???
}
  • [C++] Resource acquisition is initialization
  • [Haskell] Bracket: allocate, use, release

Resources

  • Socket
    Open connection, close connection
  • Connection
    Starts socket managing, closes open socket
  • Publisher|Subscriber
    • IQBus registration flow
      Register|Subscribe, Unregister|Unsubscribe
    • Shared resource handling
  • Client
    Starts connection/InitRequest, closes connection.

Client, take 2

trait IQBus[F[_]] {
  def address: Address.Full
  def config: cfg.Config

  def publisher(props: IQBusPublisher.Properties): Resource[F, IQBusPublisher[F]]
  def subscriber(props: IQBusSubscriber.Properties): Resource[F, IQBusSubscriber[F]]
}
object IQBus {
  def apply[F[_]](address: Address.Full, config: cfg.Config)(
    implicit F: ConcurrentEffect[F], M: Metrics[F], T: Timer[F]
  ): Resource[F, IQBus[F]] = ???
}

F[_] Requirements

  • F[_] is 'tagless' by default

  • Need something -- ask for it

  • Next in hierarchy is more powerful

Publication

trait IQPublication[A] { self =>
  def properties: IQBusPublisher.Properties
  def actor[F[_]](implicit B: IQBus[F]): Resource[F, IQBusPublisher[F]] = 
    B.publisher(self.properties)

  def publish[F[_]: Concurrent: IQBus]: Sink[F, A] = ???
  def publish1[F[_]: Sync: IQBus]: Resource[F, A => F[Option[MessageResponse]]] = ???
}
final class IQPublicationOps[F[_]](val B: IQBus[F]) extends AnyVal {
  def publish[A](buffer: Int)(
    implicit A: IQPublication[A], 
             F: Concurrent[F]
  ): Sink[F, A] = A.publish(B, F)
  
  def publish1[A](
    implicit A: IQPublication[A], 
             F: Sync[F]
  ): Resource[F, A => F[Option[MessageResponse]]] = A.publish1(B, F)
}

Publication, example

case class Quote(active: Int, value: Double, time: Long)

implicit val QP: IQPublication[Quote] = IQPublication.data.event(
  name = "quote",
  version = "1.0",
  service = "quote-provider"
)
import iqbus.syntax.publisher._

val iqbus: IQBus[F] = ???

iqbus.publish1[Quote].use { pub =>
  val qs = List(1.117, 1.118, 1.116).map { v =>
    Quote(1, v, System.currentTimeMillis / 1000)
  }
  qs.traverse(pub).map { responses =>
    responses.flatten.foreach(println)
  }
}

Subscription

trait IQSubscription[A] {
  def properties: IQBusSubscriber.Properties
  def actor[F[_]](implicit B: IQBus[F]): Resource[F, IQBusSubscriber[F]] = 
    B.subscriber(properties)

  def subscribe[F[_]: Sync: IQBus](buffer: Int): Stream[F, A] = ???
  def subscribe1[F[_]: Concurrent: IQBus](f: A => F[Unit]): F[CancelToken[F]] = ???
}
final class IQSubscriptionOps[F[_]](val B: IQBus[F]) extends AnyVal {
  def subscribe[A](buffer: Int)(
    implicit A: IQSubscription[A], 
             F: Sync[F]
  ): Stream[F, A] = A.subscribe(buffer)(B, F)
  def subscribe1[A](f: A => F[Unit])(
    implicit A: IQSubscription[A], 
             F: Concurrent[F]
  ): F[CancelToken[F]] = A.subscribe1(f)(B, F)
}

Subscription, example

case class Quote(active: Int, value: Double, time: Long)

implicit val QP: IQSubscription[Quote] = IQSubscription.data.event(
  name = "quote",
  version = "1.0",
  service = "quote-provider"
)
import iqbus.syntax.subscriber._

val iqbus: IQBus[F] = ???

iqbus
  .subscribe[Quote](100)
  .take(200)
  .map(println)
  .compile
  .drain

Request

trait IQRequest[A, B] {
  def publication: IQPublication[A]
  def subscription: IQSubscription[B]

  def request1[F[_]: Concurrent: IQBus]: Resource[F, A => F[B]] = ???

  def request[F[_]: Concurrent: IQBus](buffer: Long): Pipe[F, A, (A, B)] = ???
  def requestR[F[_]: Concurrent: IQBus, R](buffer: Long): Pipe[F, (R, A), (R, B)] = ???
}

Response

trait IQResponse[B, A] {
  def publication: IQPublication[B]
  def subscription: IQSubscription[A]

  def respond1[F[_]: Concurrent: IQBus](f: A => F[Option[B]]): F[CancelToken[F]] = ???

  def respond[F[_]: Concurrent: IQBus](buffer: Long): Pipe[F, (UUID, B), (UUID, A)] = ???
  def respondIf[F[_]: Concurrent: IQBus](buffer: Long)(
    f: A => F[Boolean]
  ): Pipe[F, (UUID, B), (UUID, A)] = ???
}

Capisce?

Made with Slides.com