Dream of papers
Ideas for you to implement in Elixir
Thomas Depierre
@DianaO
Diana Olympos
Twitter :
Github :
- B2B
- Intelligence, Events, Advice
- Unlock the future
- Big Company ™
- FTSE 250
- Founded 1947
- New organisation : Ascential Makers
- Elixir for everything in the backend
What is this talk
- I read a lot of papers
- Mainly about distributed systems and Resilience. System thinking too
- Some of these could fit in Elixir
- Here are my preferred one
- I have limited time to implement, so please do it for me
- k thx bye
Lineage Driven Fault Injection
Orchestrating Chaos Applying Database Research in the Wild - Peter Alvaro
- The concept is really neat
- Especially interesting if you begin to have a complex system with circuit breakers of defaults
- It is meant for "service" architecture, but is based on tracing
- We could probably do it between processes
- Or build a more generic one.
Musketeers and data processing
- Data Processing is a lot of our work
- Lot of my work at least.
- Data from A go to B after being transformed.
- PDF, Database Records, Kafka, Kinesys, RabbitMQ,...
- Abstract that away ?
- In a way that can switch implementation under the hood ?
- Flow could do that.
- Need more implementations
- Where are the Data Processing Frameworks in Elixir!
- Partisan unlock easily sending of big files !
- Could be a middleware for every data processing frontend and backend.
Virtual Actors and optimisations
- Virtual Actors are distributed
- Think Riak Core or Swarm
- How to minimise message passing over network in a dynamic environment ?
- We need a "Virtual Actor" system in Elixir. With better global registry. Or at least more options that works
- This is one approach. See Erleans and Orleans
- Also offer an approach to automatically scale GenStage stages for processing.... (2nd part)
Efficient Geo distribution of actors !
https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/Geo-Orleans-OOPSLA17.pdf
Back Pressure and Optimisations
Join Idle Queue
- Personal area of research
- Seems there is nearly no existing research 😨
- At least not for small tasks. There are schedulers, but they tend to work for long tasks. Or one off processors. But not with message passing ala BEAM.
- But it looks a lot like load balancing. If load balancing had proper back pressure
- Historically, Load Balancing decided that multi producers/multi consumers was far too hard.
- My own paper ?
Join Idle Queue
Back Pressure and Optimisations
- Look a lot like what i would like, once adapted
- Nearly no implementation 😨
- Seems there is nearly no existing research 😨
- Original algorithm is hard to use for optimisation
- But Michael Mitzenmacher to the rescue !
- Sadly not directly applicable
- What about Producer side queuing...
- But progress. Hope to have something to show one day
- Feel free to help