-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Should we restart all pods to resync cache after vapp was updated ? #11
Comments
@Eikykun Yeah, that would be a serious problem. Maybe a tricky solution:
The question is controller will be leader for 1~2s in this way. I'm not sure if it is acceptable or it there some better solutions? |
Only one pod is allowed to be leader for safety... |
I'm just thinking... How about trigger the controller container restart by specific liveness probe? When ctrlmesh webhook injects ctrlmesh-init and ctrlmesh-proxy containers into pods, it can also set the liveness probe for the original controller container. For example it checks a file in shared volume, then ctrlmesh-proxy can trigger the container to be restarted by deleting the file... |
What happened:
If vapp specHash changed, the proxy container will block leader election to restart main container.
But only leader pod restarted, other proxy container always wait for main container to restart.
Why it happened:
In controller-runtime
LeaderElector
, it has 2 loops to run the leader election:acquire()
lockrenew()
lockOnly 2 will panic after catch an error.
Leader pod in loop 2, but other pods in loop 1.
What you expected to happen:
My expectation is to restart all pods after vapp specHash changed...
The text was updated successfully, but these errors were encountered: