Restarting controller manager admits all workloads

[kchopra456]

What happened:

• Restarting Kueue controller (common occurence when leader election fails), admits all previously finished workloads.

What you expected to happen:

• Finished workloads, must not be queued and admitted to Q and CQ.

How to reproduce it (as minimally and precisely as possible):

1. Submit 2 jobs, with sleep 5; exit 0 and sleep 5; exit 1, one job suceeds and other fails.
2. Both the workloads are marked as Finished.
3. Rollout restart to the Kueue controller, and both the workloads will be re-admitted, though the workloads are marked Finished.
4. Q resources are never released, until the workloads are deleted.

Anything else we need to know?:
Log attached.
manager-001.log

Environment:

• Kubernetes version (use kubectl version): 1.27.0
• Kueue version (use git describe --tags --dirty --always): v0.5.0, v0.5.1, v0.5.2, v0.6.0-rc.1
• Cloud provider or hardware configuration: minikube
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[kchopra456]

Related comment - #1450 (comment)
Helped but does not solve the issue - #1572

[alculquicondor]

/assign @trasc