Awaiting Big Data

About me

• Co-Founded truestory.one
• Working for Comfy
• Previous startups and companies
• In love with Python since 2007
• Attracted to biotech, IoT, automation
• Hobbies like reading, painting, running

Agenda

1. Problems & solutions
2. Traditional APIs
3. Naive scaling
4. Async intro
5. When to choose it
6. Async web framework
7. Scale with queues
8. Cloud deployment
9. Alternatives
10. This is not the end

Problems & solutions

A few slides about various problems and potential solutions requiring distributed calculus. Approaches using multithreading, multiprocessing, parallel workers. Basic concepts like: concurrency, parallelism, asynchronous.

Traditional APIs

Basic approaches in web development using Python. WSGI-based frameworks and REST APIs. Basic services processing data streams, assuring pipelines and DB consistency.

Naive scaling

Distribute tasks with dramatiq having configurable brokers and backends. Requirements, flexibility and performance gains.

Async intro

What is async programming. Generic concepts. How it's done using Python, what's a co-routine, the event loop etc. Keywords and useful 3rd-party libraries.

When to choose it

Is this the right choice for me? What are the advantages? Pro-cons on different use-cases. How to convert sync code into async. Tricks!

Async web framework

ASGI paradigm. Playing around with Bocadillo (based on Starlette) and uvicorn. Convert the previous API example into async code. Observe performance gains, debugging advantages, de-cluttering and getting rid of locks.

Scale with queues

Async task queues, like arq which is based on Redis. Integration with other services, maybe some extra monitoring and retrying mechanisms.

Cloud deployment

Optional: moving from development into cloud-based production. Example for Google Cloud. Service refactoring.

Alternatives

What else can you use or transform with microservicing in mind. Examples of other approaches, pro/cons. Is this complex async world for me? How should I overcome challenges?

This is not the end

Feel free to explore the same concepts in other languages as well (some of them having a more native or seamless approach). What to read next?

Reach me

Facebook

Github

Twitter

Gmail