Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix serialization of Params with set data type #19267

Merged
merged 15 commits into from
Nov 5, 2021

Conversation

dstandish
Copy link
Contributor

@dstandish dstandish commented Oct 27, 2021

This is a solution for #19096

Previously, the serialization of params did not run the param value through the _serialize function, resulting in non-json-serializable dictionaries. This manifested when a user, for example, tried to use params with a default value of type set.

Here we change the logic to run the param value through the serialization process. And I add a test for the set case.

Could def use extra set of eyes on schema definition. btw i added hook for formatting this -- i can do that in separate pr if that's helpful.

@dstandish
Copy link
Contributor Author

dstandish commented Oct 28, 2021

there's a bit of awkwardness with the Param object as relates to serialization because, while for other objects we can reuse __dict__ as kwargs, for params, default is a kwarg but not an attribute. it is tempting to add default as an attribute perhaps make value a property or just remove it, but not sure of downstream consequences.

@dstandish dstandish marked this pull request as draft October 28, 2021 16:06
@dstandish dstandish marked this pull request as ready for review October 28, 2021 17:08
@dstandish dstandish requested a review from XD-DENG as a code owner October 28, 2021 20:26
uranusjr
uranusjr previously approved these changes Oct 29, 2021
@github-actions github-actions bot added the full tests needed We need to run full set of tests for this PR to merge label Oct 29, 2021
@github-actions

This comment has been minimized.

@uranusjr uranusjr dismissed their stale review October 29, 2021 02:00

Hmm, I think the test failures are related to the change and need to be fixed.

@dstandish dstandish marked this pull request as draft October 29, 2021 07:28
@dstandish dstandish marked this pull request as draft October 29, 2021 07:28
@dstandish dstandish marked this pull request as ready for review October 29, 2021 15:16
@kaxil kaxil requested a review from msumit November 2, 2021 22:11
@msumit
Copy link
Contributor

msumit commented Nov 3, 2021

lgtm overall. Also nice work in handling the backward compatibility 👍

.pre-commit-config.yaml Outdated Show resolved Hide resolved
airflow/serialization/serialized_objects.py Show resolved Hide resolved
@kaxil kaxil added this to the Airflow 2.2.2 milestone Nov 3, 2021
@kaxil
Copy link
Member

kaxil commented Nov 4, 2021

@dstandish Can you take a look at the failing tests please

@dstandish
Copy link
Contributor Author

@dstandish Can you take a look at the failing tests please

yup will do

@dstandish
Copy link
Contributor Author

fyi i updated Param init to check whether the default is json serializable and a deprecation warning if it is not.

tests/serialization/test_dag_serialization.py Show resolved Hide resolved
tests/serialization/test_dag_serialization.py Outdated Show resolved Hide resolved
airflow/models/param.py Outdated Show resolved Hide resolved
@kaxil kaxil merged commit 8512e05 into apache:main Nov 5, 2021
jedcunningham pushed a commit that referenced this pull request Nov 5, 2021
This is a solution for #19096

Previously, the serialization of params did not run the param value through the `_serialize` function, resulting in non-json-serializable dictionaries.  This manifested when a user, for example, tried to use params with a default value of type `set`.

Here we change the logic to run the param value through the serialization process.  And I add a test for the `set` case.

closes #19096

(cherry picked from commit 8512e05)
@jedcunningham jedcunningham added the type:bug-fix Changelog: Bug Fixes label Apr 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:serialization full tests needed We need to run full set of tests for this PR to merge type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants