What is a stream?

natural abstraction for sequenced data

(file, network, events, ...)

• has a beginning
• may be unbounded
• may be non-repeatable

What is the challenge?

mismatched producer and consumer

• fast producer / slow consumer
  • producer blocks (sync only)
  • consumer drops
• slow producer / fast consumer
  • consumer blocks e.g. iterator.next()

FAST (10/sec)

slooww (2/sec)

Reactive Streams Basics

• Publisher

• Subscriber

• Subscription

• Processor

Publisher

• how does data flow?

  • key concept: streams are push-based

• data is never directly pulled from a Publisher

  • no next() method

  • subscribe to Publisher, receive events (later) via Subscriber

Reactive Streams Basics

public interface Publisher<T> {
    void subscribe(Subscriber<? super T> s);
}

Subscriber

• asynchronous data events

  • publisher.subscribe(subscriber)
  • subscriber.onNext(element), ...
  • subscriber.onComplete()

• events pushed to subscriber

• what about errors?

  • subscriber.onError(error)

Reactive Streams Basics

public interface Subscriber<T> {
    void onSubscribe(Subscription s);
    void onNext(T t);
    void onError(Throwable t);
    void onComplete();
}

Subscription

• deconstruct Iterator.next()
  • signal demand (call implies demand)
  • deliver data (return value)
  • signal error (exception)
• data & error via subscriber.on(Next|Error)
• signal demand
• • subscribe() implies demand? no

Reactive Streams Basics

Subscription

• when does data flow?

  • key innovation: flow is demand-based
• explicitly signal demand

• • subscription.request(count)

• no data events flow until demand is signaled

• • publisher.subscribe(subscriber)

  • subscriber.onSubscribe(subscription)

  • subscription.request(count)

  • subscriber.onNext(element), ...

Reactive Streams Basics

public interface Subscription {
    void request(long n);
    void cancel();
}

Processor

• both Subscriber and Publisher

  • processing stage (e.g. data transformation)
  • non-terminal

Reactive Streams Basics

public interface Processor<T, R> 
    extends Subscriber<T>, Publisher<R> {
}

Reactive Streams Basics

Flow Visualization

the flow of demand and data

Summary

• key concept: streams are push-based
• key innovation: flow is demand-based
• Reactive Streams footprint
• • Publisher
  • Subscriber
  • Subscription
  • Processor
• async/non-blocking

Reactive Streams Basics

public interface Publisher<T> {
   void subscribe(Subscriber<? super T> s);
}
public interface Subscriber<T> {
   void onSubscribe(Subscription s);
   void onNext(T t);
   void onError(Throwable t);
   void onComplete();
}
public interface Subscription {
   void request(long n);
   void cancel();
}
public interface Processor<T, R> 
    extends Subscriber<T>, Publisher<R> {
}

Basic Building Blocks

• Source
• Sink
• Flow

Akka Streams Level 1 (Basics)

the streaming function

Shape: Inlets & Outlets

Akka Streams Level 1 (Basics)

Shape is to Graph as Signature is to Function

Function :

Graph :

Inputs & Outputs (Signature)

Inlets & Outlets (Shape)

Running a Graph

Akka Streams Level 1 (Basics)

Materializer is to Graph as

ExecutionContext is to Future

• Graph defines blueprint (akin to function def)
• Materializer runs a RunnableGraph (akin to function call)
  • materialization allocates runtime resources
  • Akka Streams uses Actors
  • other (Spark, for example) theoretically possible

Materialized Value

Akka Streams Level 1 (Basics)

Graph materialization result

• runtime resource produced by a Graph during materialization
• related to / used by processing, but not part of the stream itself

Concepts Checkpoint

Akka Streams Level 1 (Basics)

// Create flow materializer
implicit val system = ActorSystem("akka-streams")
implicit val materializer = ActorMaterializer()

// Create graph components
val nums = (1 to 10)
val source: Source[Int, Unit]      = Source(nums)
val sum:    Flow[Int, Int, Unit]   = Flow[Int].fold(0)(_ + _)
val triple: Flow[Int, Int, Unit]   = Flow[Int].map(_*3)
val head:   Sink[Int, Future[Int]] = Sink.head[Int]

// Assemble and run a couple of graphs
val future1a: Future[Int] = source.via(sum).to(head).run
val future2a: Future[Int] = source.via(triple).via(sum).to(head).run

// Perform some basic tests
whenReady(future1a)(_ shouldBe nums.sum)
whenReady(future2a)(_ shouldBe (nums.sum * 3))


// Equivalent to the above graphs, using shortcuts for brevity
val future1b = Source(nums).runFold(0)(_ + _)
val future2b = Source(nums).via(triple).runFold(0)(_ + _)

Transformations

• map, mapAsync
• fold, scan, filter, collect
• take, drop
• take/drop(While|Within)
• grouped, groupedWithin

Akka Streams Level 1 (Basics)

the usual suspects...

// Source[Out, Mat]
def mapAsync[T](parallelism: Int)(f: (Out) ⇒ Future[T]): Source[T, Mat]
def takeWhile(p: (Out) ⇒ Boolean): Source[Out, Mat]
def takeWithin(d: FiniteDuration): Source[Out, Mat]

Transformations

• concat, concatMat
• flatten
• prefixAndTail, split(After|When)
• groupBy
• conflate, expand

Akka Streams Level 1 (Basics)

...and a few more

// Source[Out, Mat]
def prefixAndTail[U >: Out](n: Int): Source[(Seq[Out], Source[U, Unit]), Mat]
def splitWhen[U >: Out](p: (Out) ⇒ Boolean): Source[Source[U, Unit], Mat]
def groupBy[K, U >: Out](f: (Out) ⇒ K): Source[(K, Source[U, Unit]), Mat]
def conflate[S](seed: (Out) ⇒ S)(aggregate: (S, Out) ⇒ S): Source[S, Mat]
def expand[S, U](seed: (Out) ⇒ S)(extrapolate: (S) ⇒ (U, S)): Source[U, Mat]

Sample Application

Akka Streams & HTTP Level 1 (Basics)

enhanced historical price service

• calculate simple moving average (SMA)
• parse historical price CSV stream
• enhance historical price CSV stream

Part I (Streams)

• expose enhanced price service endpoint
• request historical prices via HTTP
• stream enhanced prices

Part II (HTTP)

Enhanced Price Service

Akka Streams Level 1 (Basics)

sample data

Date,       Open,  High,  Low,   Close, Volume,  Adj Close
2014-12-31, 25.3,  25.3,  24.19, 24.84, 1438600, 24.84
2014-12-30, 26.28, 26.37, 25.29, 25.36, 766100,  25.36
2014-12-29, 26.64, 26.8,  26.13, 26.42, 619700,  26.42
2014-12-26, 27.25, 27.25, 26.42, 26.71, 360400,  26.71

Date,       Open,  High,  Low,   Close, Volume,  Adj Close, Adj Close SMA(3)
2014-12-31, 25.3,  25.3,  24.19, 24.84, 1438600, 24.84,     25.54
2014-12-30, 26.28, 26.37, 25.29, 25.36, 766100,  25.36,     26.16

input

output

• reverse time series (most recent price first)

http://localhost:8080/stock/price/daily/yhoo?calculated=sma(3)

http://real-chart.finance.yahoo.com/table.csv?s=yhoo& ...

Enhanced Price Service

Akka Streams Level 1 (Basics)

• calculate
• enhance
• expose

Akka Streams Level 1 (Basics)

// Calculate simple moving average (SMA) using scan()
// to maintain a running sum and a sliding window
def sma(N: Int) = Flow[Double]
  .scan((0.0, Seq.empty[Double])) {
    case ((sum, win), x) =>
      win.size match {
        case N => (sum + x - win.head, win.tail :+ x)
        case _ => (sum + x,            win      :+ x)
      }
  }
  .drop(N)    // Drop initial and incomplete windows
  .map { case (sum, _) => sum / N.toDouble }

Source(1 to 5)
  .map(n => (n*n).toDouble)
  .via(sma(3))
  .runForeach(sma => println(f"$sma%1.2f")

// Output:
// 4.67
// 9.67
// 16.67

Enhanced Price Stream

simple moving average (SMA)

Akka Streams Level 1 (Basics)

import akka.stream.io.Framing

  // A CSV file row, parsed into columns
  type Row = Array[String]

  // Parse incoming bytes into CSV record stream
  // Note: Each ByteString may contain more (or less) than one line
  def parse(maximumLineLength: Int = 256): Flow[ByteString, Row, Unit] =
    Framing.delimiter(ByteString("\n"), maximumLineLength, allowTruncation = true)
      .map(_.utf8String.split("\\s*,\\s*"))


  // Select a specific column (including header) by name
  def select(name: String): Flow[Row, String, Unit] = Flow[Row]
    .prefixAndTail(1).map { case (header, rows) =>
      header.head.indexOf(name) match {
        case -1    => Source.empty[String]    // Named column not found
        case index => Source.single(name)
          .concatMat(rows.map(_(index)))(Keep.right)
      }
    }.flatten(FlattenStrategy.concat)


  // Convert row into CSV formatted ByteString
  val format = Flow[Row].map(row => ByteString(row.mkString("", ",", "\n")))

Enhanced Price Stream

CSV handling

Akka Streams Level 1 (Basics)

Enhanced Price Stream

SMA column

// Calculate and format SMA for a column, renaming the column
def smaCol(name: String, n: Int, format: String = "%1.2f") = Flow[String]
  .prefixAndTail(1)
  .map { case (header, data) =>
    Source.single(name).concatMat(
      data.map(_.toDouble)
        .via(calculate.sma((n)))
        .map(_.formatted(format))
    )(Keep.right)
  }
  .flatten(FlattenStrategy.concat)

Adj Close
  24.84
  25.36
  26.42
  26.71

Adj Close SMA(3)
    25.54
    26.16