tests: update partition limits #9257
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In 3e3975b the default topic_partitions_per_shard changed to 1000: the test was previously working because the configured limit was so much higher than the test's limit. The internal_partition_slack value was probably set before we changed the default number of consumer offsets partitions to 16.
This was exceeding the 1000 partitions per core limit. It was getting away with it because it is a relatively low traffic test compared with ManyPartitionsTest. Revise its partition count to fit within the supported max.
|
/cdt tests/rptest/scale_tests/many_partitions_test.py tests/rptest/scale_tests/partition_balancer_scale_test.py |
andrwng
reviewed
Mar 4, 2023
There was a problem hiding this comment.
The cdt runs seemed to have trouble kicking off -- I retried the regular build, and maybe we can retry the cdt run once that passes (assuming the issue is with ordering). Opened https://github.com/redpanda-data/vtools/issues/1561 for it
Comment on lines
+187
to
+188
| partitions_count = 1000 * self.redpanda.get_node_cpu_count() * len( | ||
| self.redpanda.nodes) - 32 |
There was a problem hiding this comment.
Shouldn't we be dividing by the replication factor?
There was a problem hiding this comment.
Sorry, itchy merge finger -- I think was looking at a github view that had updated test statuses but not shown comments.
You are right! Followup PR here #9274
This was referenced Mar 6, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change 3e3975b caused some scale tests to fail https://ci-artifacts.dev.vectorized.cloud/vtools/0186a681-02da-49ac-8a24-e33caa78970f/vbuild/ducktape/results/2023-03-03--001/report.html
These limits need updating so that the tests operate within the supported range. ManyPartitionsTest was only out by a few, partition balancer test was further out.
Backports Required
Release Notes