Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance test for DataFusion compactions #3375

Open
patchwork01 opened this issue Sep 27, 2024 · 0 comments
Open

Performance test for DataFusion compactions #3375

patchwork01 opened this issue Sep 27, 2024 · 0 comments
Labels
documentation Improvements or additions to documentation system-test-module
Milestone

Comments

@patchwork01
Copy link
Collaborator

patchwork01 commented Sep 27, 2024

Background

We recently implemented compaction on DataFusion with Rust.

Description

We'd like a performance test for the speed of compaction on DataFusion.

We'd like to document the performance of this in the performance figures tables.

Analysis

We can add this to CompactionPerformanceST.

We could make both performance tests use 10 columns rather than 3, to align with the deploy all system test.

We should probably run the DataFusion compactions on a container with 4 virtual CPUs.

We could also test a single compaction that runs for an hour or more, in case performance slows down over time.

@patchwork01 patchwork01 added documentation Improvements or additions to documentation system-test-module labels Sep 27, 2024
@patchwork01 patchwork01 added this to the 0.26.0 milestone Sep 27, 2024
@patchwork01 patchwork01 changed the title Performance tests for DataFusion compactions Performance test for DataFusion compactions Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation system-test-module
Projects
None yet
Development

No branches or pull requests

1 participant