Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition between volume reconstructor and volume discovery #1076

Open
Andrzej-Zukowski opened this issue Feb 7, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@Andrzej-Zukowski
Copy link
Collaborator

Describe the bug
Race condition after node reboot. Kubelet triggers volume reconstruction procedure while csi-driver still performs discovery procedure.

Environment (please complete the following information):
Issue is Hardware and OS independent

To Reproduce

  1. Shutdown the node
  2. Update one or more DriverCRs to point to incorrect block storage. f.e. /dev/sda -> /dev/sdb. Alternatively change storage position in an enclosure
  3. Boot the node
  4. Wait till node is online
  5. As soon as csi-baremetal-kernel-node pod is in creating stage isolate container outbound traffic to apiserver. Use whatever means suits you ( f.e NetworkPolicies ). Wait for updateDrivesCRs to fail. Immediately restore pods network connectivity
  6. Wait till NodeStageVolume fails as it operates on either modified DriverCR or outdated DriverCR

Expected behavior
DriverCR shall reflect actual state of the world before volume reconstruction starts

Screenshots
Not applicable

Additional context
Not applicable

@Andrzej-Zukowski Andrzej-Zukowski added the bug Something isn't working label Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant