Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: make splits/largerange more aggressive #61043

Merged

Conversation

nvanbenschoten
Copy link
Member

This commit addresses a TODO by extending the splits/largerange roachtest to run
on a 6 node cluster. This extension means that the test will begin performing
rebalancing once splits begin, which should stress the behavior of streaming
snapshots. If we were to ever hold one of these snapshots in-memory in its
entirety, the test would have a bad time.

Release justification: testing only

@cockroach-teamcity
Copy link
Member

This change is Reviewable

@nvanbenschoten
Copy link
Member Author

Here are some pretty cool graphs of this roachtest in action around the point where we let the single 32GB range start splitting and rebalancing:

Screen Shot 2021-02-24 at 12 38 02 AM

Screen Shot 2021-02-24 at 12 38 08 AM

Copy link
Contributor

@irfansharif irfansharif left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That Avg. QPS graph looks surprising, do you know why that happens? Does all the sudden splits processing starve out CPU that aggressively?

@nvanbenschoten
Copy link
Member Author

I think the test just cuts all load once the dataset is loaded in and waits for splitting. The test doesn't currently wait for rebalancing to complete, though. I'm going to add that.

@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/largerangeSplits branch 5 times, most recently from 5c9a816 to 975377d Compare February 24, 2021 21:11
This commit addresses a TODO by extending the splits/largerange roachtest to run
on a 6 node cluster. This extension means that the test will begin performing
rebalancing once splits begin, which should stress the behavior of streaming
snapshots. If we were to ever hold one of these snapshots in-memory in its
entirety, the test would have a bad time.

Release justification: testing only
@nvanbenschoten nvanbenschoten force-pushed the nvanbenschoten/largerangeSplits branch from 975377d to 4725b5b Compare February 25, 2021 01:05
@nvanbenschoten
Copy link
Member Author

bors r+

@craig
Copy link
Contributor

craig bot commented Feb 25, 2021

Build failed (retrying...):

@craig
Copy link
Contributor

craig bot commented Feb 25, 2021

Build succeeded:

@craig craig bot merged commit 5648691 into cockroachdb:master Feb 25, 2021
@nvanbenschoten nvanbenschoten deleted the nvanbenschoten/largerangeSplits branch March 1, 2021 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants