-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create a generic callbacks class for KubernetesPodOperator #35714
Conversation
Also doc paragraph about this new cool ability is good to have |
@eladkal could you check the Google provider check? is it's complicated, I can bump the min |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
❤️ it
There is a test failure though :( |
I just tested on a real cluster and found that the type I provide to the trigger is converted to a string. I even tried providing an object instance but had the same problem. That's because we don't serialize the trigger kwargs, but we keep sqlalchemy convert them to string ( I need to revert the async part and keep it only for the sync part, and later we can add support for async callbacks once we find a clean solution for the issue; I will take this into account while working on the kwargs encryption. |
This PR adds a set of callbacks for
KubernetesPodOperator
called when the pod starts, terminates, and is cleaned up, and as when the client is initiated.It provides some valuable params, like the pod spec, the Kubernetes client, and the current mode (sync or async); it could be helpful in different use cases like:
The current implementation is only used in sync mode, but I provide the mode param for the future support for async mode. This support is blocked by the way of serializing the trigger kwargs, where all the provided types and methods are serialized as a string, and it's a bit difficult to recreate the original object without a custom serializer or impacting the event loop (for example provide the class path and import it in the triggerer)