[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status

[CraneShiEMC]

Purpose

Resolves #1051

1. Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status
2. Proceed Kubelet's CSI call also on Failed Volume

PR checklist

• Add link to the issue
• Choose Project
• Choose PR label
• New unit tests added
• Modified code has meaningful comments
• All TODOs are linked with the issues
• All comments are resolved

Testing

I've similuated the scenario of volume's k8s global device mountpoint missing in solely kubelet-issued NodePublishVolume in my standalone test. This CSI defensive enhancement can work well as expected in the test.

custom-ci passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-ci/1562/

custom-acceptance passed:
Atlantic (rke2): https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-tar_b_ona/39/
Openshift: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-oil_bd/255/

[codecov]

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (1f86bc6) 72.76% compared to head (5a36eea) 72.81%.

Files	Patch %	Lines
pkg/node/node.go	82.69%	6 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1050      +/-   ##
==========================================
+ Coverage   72.76%   72.81%   +0.04%     
==========================================
  Files          63       63              
  Lines        8949     8994      +45     
==========================================
+ Hits         6512     6549      +37     
- Misses       2147     2153       +6     
- Partials      290      292       +2     

Flag	Coverage Δ
unittests	72.81% <82.69%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[libzhang]

I am a little concerned about this process, as some mount logic(like global path setup) is done in kubelet. there redo nodeStage may not help in such case. do we have a test can prove that this will help in the failure case?

[CraneShiEMC]

The kubelet issued wrong CSI call NodePublishVolume in this case assumed that the volume's device global path's mountpoint has been successfully setup in some successful previous CSI call NodeStageVolume. But acutally the volume's device global path has been unmounted in the possible forceful node-removal and kubelet cannot successfully sync the volume's real status because of its continually-failed "orphan" pod's volume cleanup by calling CSI NodeUnpublish when CSI pods have not been intialized yet.

In spite of this, the device global path still exists there if the volume still exists in the most cases. Even though the device global path has also been removed in the worst case, CSI current logic can still create the device global path itself.

I will do the functional test on this code change to verify it.

[CraneShiEMC]

I've similuated the scenario of volume's k8s global device mountpoint missing in my standalone test. This CSI defensive enhancement can work well as expected in the test.

[libzhang]

it seems the UT in PR validation failed. please fix it.

[libzhang]

it seems this is not needed, the default state is Published in this function.

[CraneShiEMC]

because we will proceed on the failed volume now, here the set of status to Published is for Failed volume successfully unpublished and used by multiple pods.