cmd/roachprod: ensure wipe really wipes #41552
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Release note: None
Noticed while debugging the `disk-stall` roachtest. Forcing 0755 permissions on the `cockroach-temp*` directory is done for consistency with every other directory created by cockroach. Release note: None
Previously, `roachprod wipe` was being run by the default user (now `ubuntu`) which meant it couldn't remove files created by root. It was also accidentally swallowing errors. Now `roachprod wipe` runs as root when wiping remote nodes, ensuring it really cleans up any files. The failure to remove root owned files was causing the `disk-stall` tests to be flaky because files created through `charybdefs` are created as root. These root owned files could be removed, but cockroach also creates a root owned directory and the files in that root owned directory could not be removed. If two of the `disk-stall` tests ran on the same cluster, the first could end up leaving around files that the second encountered and complained about. Fixes cockroachdb#41530 Release note: None
|
This change is |
|
I ran |
tbg
approved these changes
Oct 13, 2019
There was a problem hiding this comment.
Reviewed 1 of 1 files at r1, 1 of 1 files at r2, 2 of 2 files at r3.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @ajwerner)
tbg
approved these changes
Oct 13, 2019
There was a problem hiding this comment.
Thanks!
Reviewable status:
|
TFTR! bors r=tbg |
craig bot
pushed a commit
that referenced
this pull request
Oct 13, 2019
41552: cmd/roachprod: ensure wipe really wipes r=tbg a=petermattis Previously, `roachprod wipe` was being run by the default user (now `ubuntu`) which meant it couldn't remove files created by root. It was also accidentally swallowing errors. Now `roachprod wipe` runs as root when wiping remote nodes, ensuring it really cleans up any files. The failure to remove root owned files was causing the `disk-stall` tests to be flaky because files created through `charybdefs` are created as root. These root owned files could be removed, but cockroach also creates a root owned directory and the files in that root owned directory could not be removed. If two of the `disk-stall` tests ran on the same cluster, the first could end up leaving around files that the second encountered and complained about. Fixes #41530 Release note: None Co-authored-by: Peter Mattis <[email protected]>
Build succeeded |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously,
roachprod wipewas being run by the default user (nowubuntu) which meant it couldn't remove files created by root. It was alsoaccidentally swallowing errors. Now
roachprod wiperuns as root whenwiping remote nodes, ensuring it really cleans up any files.
The failure to remove root owned files was causing the
disk-stallteststo be flaky because files created through
charybdefsare created as root.These root owned files could be removed, but cockroach also creates a root
owned directory and the files in that root owned directory could not be
removed. If two of the
disk-stalltests ran on the same cluster, thefirst could end up leaving around files that the second encountered and
complained about.
Fixes #41530
Release note: None