Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: refactor client classes for safer type checking #552

Merged
merged 11 commits into from
Jan 19, 2022

Conversation

plamut
Copy link
Contributor

@plamut plamut commented Dec 22, 2021

Towards #536.

This PR is based on #551, it addresses the false positives mypy produces due to the fact that it cannot know that most of the generated methods are dynamically injected into the hand-written client wrapper classes.

#551 should likely be merged first, as this change is somewhat more risky, and we want to have it in its own commit to make the rollback easier and smaller, should that be necessary.

@plamut plamut requested review from a team as code owners December 22, 2021 10:25
@plamut plamut requested a review from parthea December 22, 2021 10:25
@product-auto-label product-auto-label bot added the api: pubsub Issues related to the googleapis/python-pubsub API. label Dec 22, 2021
samples/snippets/requirements.txt Outdated Show resolved Hide resolved
samples/snippets/requirements.txt Outdated Show resolved Hide resolved
google/cloud/pubsub_v1/publisher/client.py Outdated Show resolved Hide resolved
google/cloud/pubsub_v1/subscriber/client.py Outdated Show resolved Hide resolved
@plamut plamut added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
@plamut plamut force-pushed the iss-536-refactor-composition branch from c1e59dd to 353be0a Compare December 27, 2021 07:38
@plamut plamut added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
@plamut plamut force-pushed the iss-536-refactor-composition branch from 353be0a to a937a15 Compare December 27, 2021 11:40
@plamut plamut added the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Dec 27, 2021
Copy link
Contributor

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks a lot cleaner! A couple of questions.

google/cloud/pubsub_v1/publisher/client.py Outdated Show resolved Hide resolved
noxfile.py Outdated Show resolved Hide resolved
@tswast tswast changed the title refactor: directly subclass generated clients fix: refactor client classes for safer type checking Jan 12, 2022
@plamut plamut force-pushed the iss-536-refactor-composition branch from a937a15 to 4457779 Compare January 18, 2022 11:12
@plamut
Copy link
Contributor Author

plamut commented Jan 18, 2022

@tswast Applied the suggestions.

Sorry for the force push, rebased just the relevant commits onto the latest main after #551 was merged.

@plamut plamut requested review from tswast and removed request for a team January 18, 2022 11:14
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jan 18, 2022
@tswast
Copy link
Contributor

tswast commented Jan 18, 2022

Getting a few mypy errors on the samples

nox > Running session mypy_samples
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/mypy_samples
nox > python -m pip install -e '.[all]'
nox > python -m pip install pytest
nox > python -m pip install mypy==0.910
nox > python -m pip install types-mock types-protobuf types-setuptools
nox > mypy --config-file /tmpfs/src/github/python-pubsub/samples/snippets/mypy.ini --no-incremental samples/
samples/snippets/utilities/us_states_pb2.py:11: error: Call to untyped function "Default" in typed context
samples/snippets/subscriber.py:107: error: Module "google.cloud.pubsub_v1.types" has no attribute "DeadLetterPolicy"
samples/snippets/subscriber.py:172: error: Module has no attribute "PushConfig"
samples/snippets/subscriber.py:260: error: Module has no attribute "PushConfig"
samples/snippets/subscriber.py:262: error: Module has no attribute "Subscription"
samples/snippets/subscriber.py:290: error: Module "google.cloud.pubsub_v1.types" has no attribute "DeadLetterPolicy"
samples/snippets/subscriber.py:290: error: Module "google.cloud.pubsub_v1.types" has no attribute "FieldMask"
samples/snippets/subscriber.py:330: error: Module has no attribute "Subscription"
samples/snippets/subscriber.py:335: error: Redundant cast to "Subscription"
samples/snippets/subscriber.py:353: error: Module "google.cloud.pubsub_v1.types" has no attribute "FieldMask"
samples/snippets/subscriber.py:378: error: Module has no attribute "Subscription"
samples/snippets/subscriber.py:383: error: Redundant cast to "Subscription"
samples/snippets/subscriber.py:585: error: Call to untyped function "Retry" in typed context
samples/snippets/subscriber.py:631: error: Call to untyped function "Retry" in typed context
samples/snippets/publisher.py:281: error: Call to untyped function "Retry" in typed context
samples/snippets/publisher.py:286: error: Call to untyped function "if_exception_type" in typed context
samples/snippets/subscriber_test.py:179: error: Module "google.cloud.pubsub_v1.types" has no attribute "DeadLetterPolicy"
Found 17 errors in 4 files (checked 13 source files)
nox > Command mypy --config-file /tmpfs/src/github/python-pubsub/samples/snippets/mypy.ini --no-incremental samples/ failed with exit code 1
nox > Session mypy_samples failed.

Copy link
Contributor

@tswast tswast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than the failing samples checks. Thanks for adding the api property back.

owlbot.py Outdated
@@ -467,12 +412,12 @@ def mypy(session):
# ----------------------------------------------------------------------------
s.replace(
"noxfile.py",
r'"pytype",',
r' "mypy",',
'\g<0>\n # "mypy_samples", # TODO: uncomment when the checks pass',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out-of-date comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, will remove it so that owlbot will not add it back.

@plamut
Copy link
Contributor Author

plamut commented Jan 19, 2022

Getting a few mypy errors on the samples

This used to pass (IIRC), will see what changes are needed to bring it back to green state.

@parthea parthea added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 19, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 19, 2022
@parthea parthea added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Jan 19, 2022
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jan 19, 2022
@plamut
Copy link
Contributor Author

plamut commented Jan 19, 2022

@tswast I confused the type-check errors with BigQuery, they were only fixed there.

There are two classes of errors reported by Pub/Sub samples type check:

  • Generated proto types are injected into google.cloud.pubsub_v1.types dynamically at import time, and mypy cannot know about them.
    Module has no attribute "ProtoMessageType"
    
  • Untyped objects from python-api-core - not all of them have been annotated, for example api_core.retry.if_exception_type and api_core.retry.Retry from the retry module.
    Call to untyped function "if_exception_type" in typed context
    

While the latter can be solved by adding the missing annotations to api-core, the former would require quite some acrobatics to make it work.

We would have to avoid dynamically injected content, perhaps by some sort of a preprocessor that would copy/paste the generated types directly into google/cloud/pubsub_v1/types.py. In principle doable, but would require changes to the process and an update to tooling, which is too large a task to fit into this PR... 😕

cc: @pradn


Edit:
There's also a third option - let's keep the mypy_samples session commented out and at least merge the refactoring improvements. With that we would only have two classes of errors (described above) left to deal with.

I'd say this gives us a good base for future iterations, while preserving the improvements accumulated so far.

@parthea parthea added the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 19, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 19, 2022
@parthea parthea added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. owlbot:run Add this label to trigger the Owlbot post processor. labels Jan 19, 2022
@gcf-owl-bot gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Jan 19, 2022
@yoshi-kokoro yoshi-kokoro removed kokoro:run Add this label to force Kokoro to re-run the tests. kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Jan 19, 2022
@tswast
Copy link
Contributor

tswast commented Jan 19, 2022

While the latter can be solved by adding the missing annotations to api-core, the former would require quite some acrobatics to make it work.

I think we should definitely annotate the missing methods in api-core regardless of next steps.

We would have to avoid dynamically injected content, perhaps by some sort of a preprocessor that would copy/paste the generated types directly into google/cloud/pubsub_v1/types.py

That is probably the right long-term solution. That said, it appears there are less than a dozen instances of "Module has no attribute" in the mypy_samples run. What if we use google.pubsub_v1.types instead of google.cloud.pubsub_v1.types in the code samples?

https://github.com/googleapis/python-pubsub/blob/main/google/pubsub_v1/types/__init__.py

The latter does have a few manual classes, but for the proto message types seems to just be redirecting (way too many in my opinion, there's no point in defining aliases for the shared types for example) various protobuf modules into the same namespace.

@tswast
Copy link
Contributor

tswast commented Jan 19, 2022

Re: third option, since we haven't added a py.typed file to https://github.com/googleapis/python-pubsub/tree/main/google/cloud/pubsub_v1 yet, that'd be okay with me, as this is an improvement. We do need mypy_samples as a blocker for that, so let's make sure we cross-link #536 with the todo to uncomment mypy_samples.

@tswast tswast merged commit 7f705be into googleapis:main Jan 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsub Issues related to the googleapis/python-pubsub API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants