Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add info log how to fix: More than one pod running with labels #17445

Closed
wants to merge 2 commits into from

Conversation

wojsamjan
Copy link

Logs information how to fix issue with: "More than one pod running with labels" - using is_delete_operator_pod argument when creating new KubernetesPodOperator
This is just a simple update which adds an extra info log how to avoid the mentioned problem. It does not change a logic at all.
Flag ensures that every time a pod is being deleted. So we always end up with single and unique running pod.

related: #10544
Accidentally closed: #17285


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added provider:cncf-kubernetes Kubernetes provider related issues area:providers labels Aug 5, 2021
@jedcunningham
Copy link
Member

As I mentioned in the other PR, I think the right fix is to only attempt to reattach to running pods. If we were to offer advice in a log message, setting reattach_on_restart to false might be better advice?

@wojsamjan
Copy link
Author

Hi, sorry for my late response. I have just gone through the code, how it should work with this flag set to False. And it should work, maybe this could be better. On Monday morning I will make a test of it and I will be back with response. I am also wondering, maybe 2 logs would be actually helpful. If someone wants to keep their pods and create all the time a new one - then your advice might be helpful. If someone else would like to re-attach (maybe its faster?) and deleting (to do not mess with running multiple pods) then my advice could be helpful as well. And in case that someone would like to create & delete the pod each time - then both could have sense.
@jedcunningham What do you think about it?

@jedcunningham
Copy link
Member

#18070 will fix the underlying issue allowing is_delete_operator_pod=False and
reattach_on_restart=True to function together properly during retries.

@wojsamjan
Copy link
Author

@jedcunningham as I understand I should close this PR, right?

@jedcunningham
Copy link
Member

@wojsamjan, yeah, if folks continue to hit this it won't be due to is_delete_operator_pod or reattach_on_restart.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:cncf-kubernetes Kubernetes provider related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants