POLARS

Revolutionize Your Data Workflow with Polars: The Ultimate Pandas Replacement

@YVictor

Index

Introduction

Core Features and Advantages

Real-World Applications

Q&A

Introduction

"10 Things I Hate About pandas" by Wes McKinney

No query planning
Limited multicore algorithms
RAM management
Warty missing data support
Complex group by operations awkward and slow

Columnar
Standardization
Cross Languages
Diverse Data Type
Nested Data Type

Apache Arrow

Column vs ROW

https://www.pola.rs/posts/i-wrote-one-of-the-fastest-dataframe-libraries/

Core Features and Advantages

QUERY PLAN

Support for lazy mode and query optimization.

PARALLEL

Utilizes all available cores on your machine without any additional configuration.

Copy On Write

Minimizes redundant data copying, saves memory.

Core Features and Advantages

Rust

Rust which gives it C/C++ performance and allows it to fully control performance critical parts in a query engine.

ARROW

Apache Arrow is a cross-language, columnar memory format for big data analytics, enabling efficient data sharing between systems.

Expression

Enables custom computations and transformations for composable operations and better performance.

IO and Lazy mode

QUERY PLAN

select
filter
join
with_columns
sort
groupby | agg

Manipulation/selection

collect
fetch
serialize
deserialize
profile
explain

over
str
dt
cat
bin
list

JOIN

https://h2oai.github.io/db-benchmark/#join

GROUP BY

https://h2oai.github.io/db-benchmark/#groupby

Expressions

MERGE

LazyFRAME

Real-world Applications

Tick2kline pandas

Tick2kline pandas

Tick2kline polars

Tick2kline

Real-world Applications

Daily data preprocess

Daily data preprocess

Daily data preprocess

Futures Data Clean

tick2kline with rust

Polars Extension with rust

https://github.com/Yvictor/polars_ta_extension

QA

yvictor3141@gmail.com

Polars Rust Python

Alternatives compare

QUERY PLAN

LARGE DATA

DIRTY DATA

STRICTER API

COLUMNAR

PARALLELIZE

DISTRIBUTED

PANDAS	DASK	MODIN	VAEX	CUDF	SPARK	DUCKDB	POLARS