Loading

Polars

yvictor

This is a live streamed presentation. You will automatically follow the presenter and see the slide they're currently on.

POLARS

Revolutionize Your Data Workflow with Polars: The Ultimate Pandas Replacement

@YVictor

Index

Introduction

Core Features and Advantages

Real-World Applications

Q&A

Introduction

 

"10 Things I Hate About pandas" by Wes McKinney

  • No query planning
  • Limited multicore algorithms
  • RAM management
  • Warty missing data support
  • Complex group by operations awkward and slow

  • Columnar

  • Standardization

  • Cross Languages

  • Diverse Data Type

  • Nested Data Type

Apache Arrow

Column vs ROW

https://www.pola.rs/posts/i-wrote-one-of-the-fastest-dataframe-libraries/

Core Features and Advantages

QUERY PLAN

Support for lazy mode and query optimization.

PARALLEL

Utilizes all available cores on your machine without any additional configuration.

Copy On Write

Minimizes redundant data copying, saves memory.

Core Features and Advantages

Rust

Rust which gives it C/C++ performance and allows it to fully control performance critical parts in a query engine.

ARROW

Apache Arrow is a cross-language, columnar memory format for big data analytics, enabling efficient data sharing between systems.

Expression

Enables custom computations and transformations for composable operations and better performance.

IO and Lazy mode

QUERY PLAN

  • select
  • filter
  • join
  • with_columns
  • sort
  • groupby | agg

Manipulation/selection

  • collect
  • fetch
  • serialize
  • deserialize
  • profile
  • explain
  • over
  • str
  • dt
  • cat
  • bin
  • list

JOIN

https://h2oai.github.io/db-benchmark/#join

GROUP BY

https://h2oai.github.io/db-benchmark/#groupby

Expressions

MERGE

LazyFRAME

Real-world Applications

Tick2kline pandas

Tick2kline pandas

Tick2kline polars

Tick2kline polars

Tick2kline

Real-world Applications

Daily data preprocess

Daily data preprocess

Daily data preprocess

Futures Data Clean

Futures Data Clean

tick2kline with rust

tick2kline with rust

tick2kline with rust

tick2kline with rust

tick2kline with rust

Polars Extension with rust

https://github.com/Yvictor/polars_ta_extension

QA

yvictor3141@gmail.com

Polars Rust Python

Alternatives compare

QUERY PLAN
LARGE DATA
DIRTY DATA
STRICTER API
COLUMNAR
PARALLELIZE
DISTRIBUTED
PANDAS DASK MODIN VAEX CUDF SPARK DUCKDB POLARS
Made with Slides.com