Dream of papers

Ideas for you to implement in Elixir

Thomas Depierre

@DianaO

Diana Olympos

 

Twitter :

Github :

  • B2B
  • Intelligence, Events, Advice
  • Unlock the future
  • Big Company ™
    • FTSE 250
    • Founded 1947
  • New organisation : Ascential Makers
  • Elixir for everything in the backend

What is this talk

  • I read a lot of papers
  • Mainly about distributed systems and Resilience. System thinking too
  • Some of these could fit in Elixir
  • Here are my preferred one
  • I have limited time to implement, so please do it for me
  • k thx bye

Lineage Driven Fault Injection

Orchestrating Chaos Applying Database Research in the Wild - Peter Alvaro

https://www.youtube.com/watch?v=YplkQu6a80Q

  • The concept is really neat
  • Especially interesting if you begin to have a complex system with circuit breakers of defaults
  • It is meant for "service" architecture, but is based on tracing
  • We could probably do it between processes
  • Or build a more generic one.

Musketeers and data processing

  • Data Processing is a lot of our work
  • Lot of my work at least.
  • Data from A go to B after being transformed.
  • PDF, Database Records, Kafka, Kinesys, RabbitMQ,...
  • Abstract that away ?
  • In a way that can switch implementation under the hood ?
  • Flow could do that.
  • Need more implementations
  • Where are the Data Processing Frameworks in Elixir!
  • Partisan unlock easily sending of big files !
  • Could be a middleware for every data processing frontend and backend.

Virtual Actors and optimisations

  • Virtual Actors are distributed
  • Think Riak Core or Swarm
  • How to minimise message passing over network in a dynamic environment ?
  • We need a "Virtual Actor" system in Elixir. With better global registry. Or at least more options that works
  • This is one approach. See Erleans and Orleans
  • Also offer an approach to automatically scale GenStage stages for processing.... (2nd part)

Back Pressure and Optimisations

Join Idle Queue

  • Personal area of research
  • Seems there is nearly no existing research 😨
  • At least not for small tasks. There are schedulers, but they tend to work for long tasks. Or one off processors. But not with message passing ala BEAM.
  • But it looks a lot like load balancing. If load balancing had proper back pressure
  • Historically, Load Balancing decided that multi producers/multi consumers was far too hard.
  • My own paper ?

Join Idle Queue

Back Pressure and Optimisations

  • Look a lot like what i would like, once adapted
  • Nearly no implementation 😨
  • Seems there is nearly no existing research 😨
  • Original algorithm is hard to use for optimisation
  • But Michael Mitzenmacher to the rescue !
  • Sadly not directly applicable
  • What about Producer side queuing...
  • But progress. Hope to have something to show one day
  • Feel free to help

Thank You

Questions ?