• AGH I@IET students
• Almost no previous Scala & Akka experience
  
• Only theoretical knowledge about 
  distributed systems, DDD, NoSQL, scalability
• Ready to dive into the topic

• PaaSage EU project
• Lufthansa Systems proposed the thesis
• Diversity of initial goals:
• PaaSage - industrial scalability use case
• Lufthansa - evaluate Scala/Akka stack
• Ours - learn a lot, try DDD/CQRS
• Pivoting several times
• Final thesis goal:  CQRS/ES/DDD scalability evaluation

• Demonstrate CQRS/ES concepts
• Show that CQRS/ES scales up/out
  
• Discuss Distributed DDD (DDDD)
• Get an external feedback
• Share our thoughts and lessons learned
• Find people interested in using scalable CQRS/ES

• Different databases on read and write side
• Separation of different query use cases
• Database tailored to the use case
• Lower latency of queries

CQRS DISADVANTAGES

• Full event log for free
• Append-only is enough
• Enables to add more CQRS read models in the future
• No object-relational impedance mismatch
  
• Event storming model maps 1:1 with ES

ES DISADVANTAGES

• Axon framework
• akka-persistence
• EventStore by Greg Young
• d60/Circus - very recent
• Jdon framework - already dead 
  
• Lokad.CQRS - already dead
• ncqrs - nearly dead
• handmade
  

• A toolkit,  not a framework
• Distributed by design
• Scala and Java API
• Message passing style
• Actor concurrency model
• Supervision hierarchy (let it crash)
• Location transparency

• Active object flavour
• ActorRef , location transparency
• Message passing
• Mailbox - message queue
• Unit of concurrency
• Hundred thousands of instances

• Pools and groups
• Round robin routing
• Consistent hashing routing
• Broadcast routing
• Balancing routing

akka.actor.deployment {
  /parent/router3 {
    router = round-robin-group
    routees.paths = ["/user/workers/w1", "/user/workers/w2", "/user/workers/w3"]
  }
}

val router3: ActorRef = context.actorOf(FromConfig.props(), "router3")
router3 ! Work()

• Akka streams (Rx)
• Akka HTTP
• Akka clustering
• Distributed Publish Subscribe in Cluster
  
• Akka cluster sharding
• Akka persistence

• Persistence mechanism for actors
• Based on Command/Event Sourcing concept
• PersistentActor
• defines "persistenceId"
• persists messages/events bound to the id
• when created, all correlated messages are replayed
• Variety of journal plugins
• Views
• Snapshots
• No support for CQRS and upcasting

class ExamplePersistentActor extends PersistentActor {
  override def persistenceId = "sample-id-1"
 
  var state = ExampleState()
  def updateState(event: Evt): Unit = { state = state.updated(event) }
 
  def numEvents = state.size


              val receiveRecover: Receive = {
    case evt: Evt                                 => updateState(evt)
    case SnapshotOffer(_, snapshot: ExampleState) => state = snapshot
  }
        

• P2P (gossip) clustering protocol
• Membership service
• Automatic failure detection
• Cluster-aware routers
• JMX metrics

 class SimpleClusterListener extends Actor with ActorLogging {
  val cluster = Cluster(context.system)
 
  override def preStart(): Unit = cluster.subscribe(
      self, initialStateMode = InitialStateAsEvents, classOf[MemberEvent])
  override def postStop(): Unit = cluster.unsubscribe(self)
 
  def receive = {
    case MemberUp(member) => log.info("Member is Up: {}", member.address)
    case MemberRemoved(member, previousStatus) =>
      log.info("Member is Removed: {} after {}", member.address, previousStatus)
    case _: MemberEvent => // ignore
  }
}

• Shards of stateful actors
• ShardRegion and ShardCoordinator services
• Rebalancing shards using akka-persistence
• Passivation of actors

ClusterSharding(system).start(
  typeName = "Counter",
  entryProps = Some(Props[Counter]),
  idExtractor = idExtractor,
  shardResolver = shardResolver)

val idExtractor: ShardRegion.IdExtractor = {
  case EntryEnvelope(id, payload) ⇒ (id.toString, payload)
  case msg @ Get(id)              ⇒ (id.toString, msg)
}
 
val shardResolver: ShardRegion.ShardResolver = msg ⇒ msg match {
  case EntryEnvelope(id, _) ⇒ (id % 10).toString
  case Get(id)              ⇒ (id % 10).toString
}

• Airplane and Rotation aggregates
  
• (Persistent) Actor = Aggregate
• Scala case classes = Value Objects, Events, Commands
• Publishing domain events, e.g. RotationAdded
• Separated domain from infrastructural concerns
• Hexagonal architecture
• Rest API

• Graph-oriented database (Neo4j)
• Denormalization of events from event bus
• REST API

• Simple replication of read model instances
• Round robin load balancing
• Took advantage of the replayability

• Aggregate roots sharding and rebalancing
• Round robin routers as load balancers
• Scalable event store - Cassandra

• Ansible
  
• Kamon
• Zipkin
• Akka tracing
• Gatling
• Akka clustering JMX
• Visual VM
• SBT

• Messsage delivery is not an issue
  
• DistributedPubSub Akka extension
• ZeroMQ
• RabbitMQ
• We need to replay events from the past

• Views can replay only events for a single persistent actor
• Views are polling the event store
• This will change in 2015 Q3 (see Akka Roadmap) [21]

1. Replay past events from Cassandra
2. Subscribe to Kafka

• We leveraged Kafka durability
  
• Kafka retention time set to X (e.g. 24h)
• Subscribing from scratch in Kafka after Cassandra replay
• Filtering duplicate events
• If replay takes less than X we won't miss any event

• Events ordering between aggregates [58]
• Single Kafka topic/partition for now
• Subscription only to all events
• often we need to listen for a specific events only
  (e.g. from a single aggregate)
• database is better in data filtering
• unnecessary traffic
• Not optimal for simultaneous replays on different nodes
• ATOM interface
• Spark connectors  

• Pat Helland's entities [45] matches aggregate definition
• CQRS makes scalability and distribution easier
• Saga/Business process is hard to implement efficiently
• Application services need to be replicated
• A single aggregate may be still a bottleneck
• Solution: CRDTs [20, 46, 47]

• Actors in the domain code? NO! [27]
• Prefer TypedIdClasses over UUID
• What to do when command validation fails?
• Return error? 
• Throw exception? 
• Publish event?

• Avoid ask pattern if possible [22, 24] 
• timeout hell
• performance issues
• tell, don't ask
• It's not easy to manage dependencies
• We don't like cake pattern!
• How we should handle stateless business logic?
• Actors behind routers?, futures?, static classes?
  continuation monad? dataflow?  [23, 25, 26]
• Cluster sharding is not fully dynamic yet
• Cannot change number of shards
  without restarting the app

• Test Data Builder Pattern rocks! [59]
• Start with integration tests on application service level
• Given/When/Then perfectly fits to DDD: [30]
• given past events
• when command fired
• then expected event(s)
• Eventual consistency forces you to wait in tests :(
• Sometimes it is possible to avoid it by using hacks
  e.g.: waiting on expected number of entities
• Where to put e2e tests?
• Completely outside of the app?
• In the REST port?

• EJB and Akka are in fact similar! [19-20]
• Active Object
• #unpopularopinion 
• Eventual consistency 
• often is feasible and realistic
• introduces new problems
• Performance measurement is not straightforward
  
• App monitoring is challenging
• Automatized deployment is crucial

• Open source - on the way!
• More detailed performance evaluation
• Replayable event bus improvements
• ATOM, Spark Streaming, Redis
• Causal consistency [58]
• Effective sagas implementation
• Effective stateless logic implementation
• Upcasting and snapshoting [33]
• Geo-based sharding

• Maciek & Adam
  
• Integration with PaaSage platform
• Metrics exposure
• Second read model
• More detailed performance evaluation
• Mariusz & Michał [55]
• Akka debugging tool
• Akka-tracing enhancement [54]