ROX-20872: fix fleetshard reconciler race condition #1629
Conversation
| r.lastCentralHash = centralHash | ||
| r.lastCentralHashTime = time.Now() | ||
| } else { | ||
| r.lastCentralHash = [16]byte{} |
There was a problem hiding this comment.
When the Central hash should not be updated, delete it? I would have assumed it just doesn't set the last central hash instead of removing it.
There was a problem hiding this comment.
That was the source of the bug, because when the deployment is not ready, the hash is not updated, even though the central CR has been updated.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ludydoo, SimonBaeumer The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
New changes are detected. LGTM label has been removed. |
Description
Fixing of the fleetshard cache bug.
How to reproduce:
The reason that the reconcile loop at step 5 is ignored is that the hash is not stored at step 4, even if the central CR was updated. At step 5, the reconciler will be skipped because the hash (v2) matches the last stored hash (step 2).
The effect is that v1 is deployed, but the reconciler thinks it reconciled v2.
The fix is to unset the hash if the reconciler exited early.
As an added precaution, trigger the reconcile loop if central was reconciled more than 15 minutes ago.