Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix controller.machineStatusUpdate to retry on conflict #838

Closed
elankath opened this issue Aug 1, 2023 · 0 comments · Fixed by #877
Closed

Fix controller.machineStatusUpdate to retry on conflict #838

elankath opened this issue Aug 1, 2023 · 0 comments · Fixed by #877
Assignees
@himanshu-kun
Labels
area/quality Output qualification (tests, checks, scans, automation in general, etc.) related exp/beginner Issue that requires only basic skills kind/bug Bug priority/2 Priority (lower number equals higher priority) status/closed Issue is closed (either delivered or triaged)

Comments

@elankath
Copy link
Contributor

elankath commented Aug 1, 2023 •
edited by himanshu-kun
Loading

How to categorize this issue?

/area quality
/kind bug
/priority 2

What happened:
Machines are stuck in CrashLoopBackoff and never transitions to Failed if VM startup takes a very long time and context deadline is exceeeded. This is because the Machine obj gets outdated and controller.machineStatusUpdate unfortunately doesn’t use retry on conflict so machine status update to the Failed phase is missed inside controller.machineCreateErrorHandler

What you expected to happen:
Machines are not stuck in CrashLoopBackoff. Machine Status is updated successfully without frequent errors saying "object has been modified.the object has been modified; please apply your changes to the latest version and try again"

How to reproduce it (as minimally and precisely as possible):
set a sleep exceeding the context deadline inside the MC Driver.CreateMachine

Anything else we need to know?:

related #767
Environment:

  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • Others:
@elankath elankath added the kind/bug Bug label Aug 1, 2023
@gardener-robot gardener-robot added area/quality Output qualification (tests, checks, scans, automation in general, etc.) related priority/2 Priority (lower number equals higher priority) labels Aug 1, 2023
@himanshu-kun himanshu-kun added the exp/beginner Issue that requires only basic skills label Sep 12, 2023
@himanshu-kun himanshu-kun self-assigned this Nov 30, 2023
@himanshu-kun himanshu-kun mentioned this issue Dec 1, 2023
Merged
2 tasks
@gardener-robot gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@himanshu-kun himanshu-kun

Labels
area/quality Output qualification (tests, checks, scans, automation in general, etc.) related exp/beginner Issue that requires only basic skills kind/bug Bug priority/2 Priority (lower number equals higher priority) status/closed Issue is closed (either delivered or triaged)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reduce noisy reconciles + enhance logs himanshu-kun/machine-controller-manager
3 participants
@elankath @gardener-robot @himanshu-kun