Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(python): reuse table state in write engine #2453

Merged
merged 3 commits into from
Apr 25, 2024

Conversation

ion-elgreco
Copy link
Collaborator

Description

Instead of reusing the table state, it was being instantiated every time you call write with the rust engine.

@github-actions github-actions bot added the binding/python Issues for the Python package label Apr 25, 2024
@ion-elgreco ion-elgreco force-pushed the fix/reuse_table_state_in_write branch from d026d75 to 255806a Compare April 25, 2024 08:25
@adriangb
Copy link
Contributor

Now we're cooking with gas!

Time to write tmp commit entry: 62.949541ms
Time to write final commit entry: 154.489917ms
Time taken: 0.5385329723358154
Time to write tmp commit entry: 102.603ms
Time to write final commit entry: 149.216375ms
Time taken: 0.5450408458709717

This immediately makes writes go a lot, lot faster 🥇!

Can we get this released asap?

@ion-elgreco
Copy link
Collaborator Author

ion-elgreco commented Apr 25, 2024

@adriangb good to see :)

Let me see what I can do on the release.

Btw, one last question, did you see also these performance issues with the pyarrow engine? Because there we are reusing the state

@ion-elgreco ion-elgreco enabled auto-merge (squash) April 25, 2024 12:45
@adriangb
Copy link
Contributor

I did notice the pyarrow engine being faster in some cases, this seems like a likely culprit.

@ion-elgreco ion-elgreco merged commit 6a7c684 into delta-io:main Apr 25, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants