Skip to content
This repository has been archived by the owner on Jul 25, 2022. It is now read-only.

Use custom allocator in Python build #27

Closed
Dandandan opened this issue Feb 22, 2022 · 3 comments · Fixed by #30
Closed

Use custom allocator in Python build #27

Dandandan opened this issue Feb 22, 2022 · 3 comments · Fixed by #30

Comments

@Dandandan
Copy link
Contributor

When compiling in native Rust, we have the option to include mimalloc or smalloc for improved performance.
In my experience mimalloc and smalloc behave quite similar.

Polars uses mimalloc in the Python binary:
https://github.com/pola-rs/polars/blob/master/py-polars/Cargo.toml#L22

@Dandandan Dandandan changed the title Use custom allocator in Python bindings Use custom allocator in Python build Feb 22, 2022
@matthewmturner
Copy link
Contributor

Do you think this is the only optimization we'll be able to add? If so, ill close my issue.

@Dandandan
Copy link
Contributor Author

Do you think this is the only optimization we'll be able to add? If so, ill close my issue.

I think this + upgrading the CPU feature list will be the main two. LTO is already enabled.

@matthewmturner
Copy link
Contributor

Cross post from slack.

Wow. Updating allocator to sn-malloc made big difference. This run also included full target feature list. However, when I tried just target features there was minimal impact (maybe i did it wrong?). Im not sure if some combinatation of feature plus allocator also helped. Theres still some juice to squeeze but the result compared to others are quite good across the board. Exciting!

q1: 0.05378841600000017
q2: 0.3460620409999997
q3: 1.1944592499999995
q4: 0.051161000000000456
q5: 1.214791333
q6: 1.3261748329999996
q7: 1.3834479999999996
q8: 3.088559250000001
q9: 0.6630860410000015
q10: 17.969405375

this is how i built the python wheels / the feature list in case anyone is interested:
export RUSTFLAGS='-C target-feature=+fxsr,+sse,+sse2,+sse3,+ssse3,+sse4.1+sse4.2,+popcnt,+aes,+avx,+avx2' && maturin build --release
i will probably start working on the automation script next so others can test and we can sure what ive done is correct / accurate.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants