Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to MyPy 1.2.0 #30687

Merged
merged 1 commit into from
Apr 17, 2023
Merged

Upgrade to MyPy 1.2.0 #30687

merged 1 commit into from
Apr 17, 2023

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Apr 17, 2023

Upgrading to latest (released a week ago) MyPy in the hopes it will fix some more problem with attrs after upgrading new packages, but it seems that even the latest MyPy does not know about the new typing changes introduced in attrs (traditionally mypy has attrs plugin that injects appropriate typing but apparently it needs to catch up with those changes.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Upgrading to latest (released a week ago) MyPy in the hopes it
will fix some more problem with attrs after upgrading new packages,
but it seems that even the latest MyPy does not know about the
new typing changes introduced in attrs (traditionally mypy has
attrs plugin that injects appropriate typing but apparently it
needs to catch up with those changes.
@potiuk
Copy link
Member Author

potiuk commented Apr 17, 2023

Seems that mypy in our main after some more (unrelated) upgraded packages started failing. They seem unrelated, but hey, it's MyPy and it's heuristing on when to start complaining about things that seemed to be ok earlier after some unrelated changes is something that we all got used to I think,

¯\_(ツ)_/¯

Here is the failure:

https://github.com/apache/airflow/actions/runs/4724060012/jobs/8381284205

Those were the change deps:

< # This constraints file was automatically generated on 2023-04-17T00:25:15Z
---
> # This constraints file was automatically generated on 2023-04-17T17:55:16Z
124c124
< cfn-lint==0.77.0
---
> cfn-lint==0.77.1
292c292
< limits==3.3.1
---
> limits==3.4.0
314c314
< msal==1.21.0
---
> msal==1.22.0
483c483
< soupsieve==2.4
---
> soupsieve==2.4.1
578c578
< zstandard==0.20.0
---
> zstandard==0.21.0

And after upgrading those, mypy provider checks resulted in:

airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "str"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "Dict[Any, Any]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "Tuple[Any, ...]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....
                                                        ^
airflow/providers/openlineage/utils/utils.py:345: error: Argument 1 to "asdict"
has incompatible type "List[Any]"; expected "AttrsInstance"  [arg-type]
    ...            for dict_key, subval in attrs.asdict(item, recurse=False)....

I upgraded mypy toi 1.2.0 in the hopes it will understand attrs a bit better, but this resulted in few more errors related to attrs (and some other small things I fixed in Pydantic AIP-44 classes:

airflow/serialization/serde.py:160: note: Possible overload variants:
airflow/serialization/serde.py:160: note:     def asdict(obj: DataclassInstance) -> Dict[str, Any]
airflow/serialization/serde.py:160: note:     def [_T] asdict(obj: DataclassInstance, *, dict_factory: Callable[[List[Tuple[str, Any]]], _T]) -> _T
airflow/decorators/task_group.py:117: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, None]" has incompatible type
"_TaskGroupFactory[FParams, None]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, tg_kwargs={**self.tg_kwargs, **kwargs...
                               ^
airflow/decorators/task_group.py:117: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, DAGNode]" has incompatible type
"_TaskGroupFactory[FParams, DAGNode]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, tg_kwargs={**self.tg_kwargs, **kwargs...
                               ^
airflow/decorators/task_group.py:123: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, None]" has incompatible type
"_TaskGroupFactory[FParams, None]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, partial_kwargs=kwargs)
                               ^
airflow/decorators/task_group.py:123: error: Argument 1 to "evolve" of
"_TaskGroupFactory[FParams, DAGNode]" has incompatible type
"_TaskGroupFactory[FParams, DAGNode]"; expected
"_TaskGroupFactory[FParams, FReturn]"  [arg-type]
            return attr.evolve(self, partial_kwargs=kwargs)

Unless we have a better idea (@uranusjr ?) temporary ignoring those, seem to be the best way (And as a bonus we wil have latest-and-greatest MyPy.

@potiuk potiuk merged commit cbde23e into apache:main Apr 17, 2023
@potiuk potiuk deleted the upgrade-mypy-to-1.2.0 branch April 17, 2023 20:38
wookiist pushed a commit to wookiist/airflow that referenced this pull request Apr 19, 2023
Upgrading to latest (released a week ago) MyPy in the hopes it
will fix some more problem with attrs after upgrading new packages,
but it seems that even the latest MyPy does not know about the
new typing changes introduced in attrs (traditionally mypy has
attrs plugin that injects appropriate typing but apparently it
needs to catch up with those changes.
@ephraimbuddy ephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label May 8, 2023
@ephraimbuddy ephraimbuddy added this to the Airflow 2.7.0 milestone May 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:providers area:serialization type:misc/internal Changelog: Misc changes that should appear in change log
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants