-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix faulty executor config serialization logic #26191
Fix faulty executor config serialization logic #26191
Conversation
airflow/utils/sqlalchemy.py
Outdated
if isinstance(val_copy, dict) and 'pod_override' in val_copy: | ||
val_copy['pod_override'] = BaseSerialization._serialize(val_copy['pod_override']) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be more reliable to check for __type
and __var
instead? (We could have that logic in serialized_objects
instead as a is_serialized
function)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you maybe commenting on the wrong line? here we're dealing with non-serialized object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I’m thinking line 170 should be
if not is_serialized(val_copy):
with the function implemented as
def is_serialized(v):
if not isinstance(v, dict):
return False
return Encoding.TYPE in v and Encoding.VAR in v
This can be used to unify the logic on line 187/188.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so the thing is, re 170, i see what you are suggesting. but the reason we are checking specifically for pod_override here, is because we're not trying to serialize any and all executor configs. in general, they are simply pickled. but because of idiosyncrasies of the design of the k8s objects, it's safer to essentially convert them to json before pickling , which in effect is what the serializer does. basically, we just want to do this for k8s executor configs. if we wanted to do for all configs, your suggestion makes complete sense.
b5feb44
to
dc307c3
Compare
The bind processor logic had the effect of applying serialization logic multiple times, which produced an incorrect serialization output. Resolves apache#26101.
dc307c3
to
62c7999
Compare
The bind processor logic had the effect of applying serialization logic multiple times, which produced an incorrect serialization output. Resolves apache#26101. (cherry picked from commit 87108d7)
The bind processor logic had the effect of applying serialization logic multiple times, which produced an incorrect serialization output.
Resolves #26101.
Related: #24117