on the bleeding edge



Highly scalable, distributed, 
cloud-based architecture using Scala, 
actor model and RESTful Web Services


Andrzej Dębski
Bartłomiej Szczepanik

Agenda

  1. Project origin and goals
  2. Scalability problem
  3. Solution building blocks
  4. Application architecture
  5. Application evaluation
  6. Achievements
  7. Future work

project timeline



  • Oct 2012 - PaaSage project started
  • Lufthansa Systems accepts to provide industrial use case
  • Nov 2013 - Master thesis proposal
  • Feb 2014 - Development started, 3 weeks on-site in Berlin
  • Mar 2014 till now - Continuation of the work remotely
  • Dec 2014 - PaaSage deadline for the industrial use case

project goals


  • Build highly scalable and distributed application
  • Prove that chosen building blocks
    live up to their expectations

our background


  • Almost no previous Scala & Akka experience
  • Only theoretical knowledge about distributed programming, Domain-Driven Design, NoSQL, computer systems evaluation
  • Ready to dive into the topic




scalability

 

we need horizontal scalability




Reactive programming

  • react to events

    events are natural  for humans  and  simplify
     the architecture

  • react to load

    focus  on  scalability by avoiding  contention  on shared  resources


  • react to failure

    build  resilient  systems  
    with the  ability to  
    recover at  all levels

  • react to users

    honor response  
    time guarantees  
    regardless  of load

Reactive programming



www.reactivemanifesto.org



building blocks


  • Blends functional and object-oriented world
  • Scales to programmer needs
  • Fast-paced evolution
  • Type safe with type inference
  • Java interoperability (runs on JVM)
  • Steep learning curve


  • Actor concurrency model
  • Distributed by design
  • Actor location transparency
  • Let it crash philosophy
  • Part of Typesafe stack
    for reactive applications

akka modules

  • Many useful add-ons to basic Akka functionality
  • Developed both by Typesafe and community
  • Interesting examples:
    • Akka Persistence
    • Akka Cluster
    • Akka Cluster Sharding
    • Distributed Publish Subscribe in Cluster
    • Akka Reactive Streams



  • The World's Leading Graph Database
  • Index free adjacency
  • ACID compliance

spray


  • REST/HTTP toolkit
  • Implemented on top of Akka in Scala
  • Does not use servlets
  • Encourages asynchronous processing
  • Will be part of Akka project soon

Domain-driven design


  • Set of strategic and tactical patterns 
    in software engineering
  • More and more popular
  • Fits well with reactive programming - commands & events
  • All business logic resides in the "domain model"

Command-query responsibility segregation (CQRS)



event sourcing


on the bleeding edge


Application business requirements


  • Flight scheduling service
    • Legs and airplane rotations
    • Schedule validity checks
    • Reports
  • Simple domain 
  • Focus on architecture

Application architecture


write model Scalability


read models scalability


implementation details

  • Akka-persistence
  • Distributed publish-subscribe event bus
  • CQRS+Event Sourcing mini-framework
  • Akka-clustering
  • Data sharding based on business rules
  • Batch event processing

EVALUATION - THESIS



Chosen building blocks enable creating highly scalable distributed applications with way less effort than before.

evaluation facts


  • Flexiant cloud provider
  • Simulating production environment
  • Both synthetic and real workloads
  • Lots of available specialized tools, e.g.  Gatling, Kamon
  • We have to consider many low level JVM details

Example test scenarios

  • Responsiveness
    • Time of big data sets import
    • Different request rates
    • Different query distribution
    • Rebalancing influence on response time
  • Resilience
    • Mean time to recover when a node crashes
    • System startup time

JVM warmup matters


data import time


current progress

  • CQRS architecture
  • Neo4j read model
  • Most of business logic implemented
  • Tackled lots of technical problems
  • First evaluations done, more to come
  • We are approaching the scale-out phase
  • Got in touch with Typesafe
  • Joined Krakow DDD and Scala Community

Ideas for future work


  • CQRS + Event Sourcing framework for Scala
  • Generic benchmark specification 
    for distributed applications

On the bleeding edge







Andrzej Dębski
andrzejdebski91 @ gmail.com

Bartłomiej Szczepanik
mequrel @ gmail.com

On the bleeding edge

By mequrel

On the bleeding edge

  • 1,237