-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Roachperf shows poor performance on some days for YCSB #34458
Comments
With AWS data added we see the same sort of pattern on one day. This is definitely something to dig in to. |
This PR is a temporary measure to aid in debugging the very peculiar cockroachdb#34458. The idea is that if a run fails to meet the expected throughput (which is above but near the bad runs), we'd like the opportunity to poke around. Release note: None
This PR is a temporary measure to aid in debugging the very peculiar #34458. The idea is that if a run fails to meet the expected throughput (which is above but near the bad runs), we'd like the opportunity to poke around. Release note: None
This PR is a temporary measure to aid in debugging the very peculiar cockroachdb#34458. The idea is that if a run fails to meet the expected throughput (which is above but near the bad runs), we'd like the opportunity to poke around. Release note: None
34808: roachtest: fail YCSB for debugging if below performance expectations r=ajwerner a=ajwerner This PR is a temporary measure to aid in debugging the very peculiar #34458. The idea is that if a run fails to meet the expected throughput (which is above but near the bad runs), we'd like the opportunity to poke around. Release note: None Co-authored-by: Andrew Werner <[email protected]>
The good news is that there seems to be a very reasonable explanation for all of this. The bad news is that it's going to be somewhat difficult to rectify. These performance drops seem to be due to the fact that we run the same nightly tests off of different branches and releases yet the test script is completely unaware of this. The reason we find these dips to be highly correlated across cloud providers is that the roachperf code processes the first completed result on a given day based on lexicographical order which will end up being based on build ID and then will skip other builds. This means that these dips generally correspond to previous releases. There are a couple of mitigations and forward looking changes I have in mind.
|
Heh, this is somewhat amusing. I think it is worth writing up a short note to eng@ about the dips and how they were correlated across cloud providers, and then providing a reveal as to the cause. Storing only master runs makes sense to me. At the very least, we should include the build branch in the storage directory. |
Closing with the tweak to the teamcity scripts and some cleanup. There's still more to be done to increase the robustness of the roachperf information collection and management but that's for a different issue. |
Roachperf displays large dips in performance on YCSB in January.
The text was updated successfully, but these errors were encountered: