71 of 95 python APIs fail to build with error ModuleNotFoundError: No module named 'google.api_core' #3334
Comments
SurferJeffAtGoogle
added
the
priority: p1
|
It's running version 3.0.0 of bazel, so I don't think this is related to bazel 4.0.0 being released last night. |
|
Most probably the failure was caused by these changes in protobuf: The "Pythony" gist of the problem (regardless of bazel) is that protobuf 3.14.0 removed the support for legacy types of packages (in the PR listed above). At the same time, some of our other google-namespaced packages (like google-api-core) still use legacy style. As mentioned in the warning in python documentation here: https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package, using legacy and and non-legacy packages under the same namespace ("google" in this case) makes python really unhappy. In bazel-specific sense, this coexistence of legacy packages in bazel build and non-legacy package as either system-wide dependency or as a transitive dependency pulled by bazel itself (protobuf 3.14.0) made python unable to properly resolve the packages under "google" namespace, resulting in not being able to import any of them during the gapic-generaror-python execution. This change was released with protobuf 3.14.0, and failures started happening the same time the protobuf got the the machines which were experiencing the problem. The "getting protobuf 3.14.0" could happen in different ways on different machines, that is how it got so confusing. The reason why it breaks autosynth builds is most probably because autosynth was installing protobuf 3.14.0 regardless of bazel build (I guess synthtool needs it for other but bazel reasons). The reason why some docker images and/or workstation do not work and our kokoro build does - is most probably caused by the transitive dependency pulling done differently on these machines (not sure about it, could be other explanations). How to fix:
|
…kaged as native namespace package) More details about Python namespace packaging here: https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages The protobuf changes, which made this fix necessary are here: protocolbuffers/protobuf#7902 protocolbuffers/protobuf#7908 The tracking bug for this issue (probably not the only one) googleapis/gapic-generator#3334 This is only part of the fix, for the proper fix other google-namespaced python packages must be changed as well.
|
An update: Once the fix above is integrated to The detailed explanation of what is causing the problem is too long and requires too much deep Python knowledge, but in brief the problem is this: To fix the rest, the other google-namespaced packages must be migrated from the legacy format, similarly to how it was done by The affected packages:
Protobuf has already migrated to native packages in protocolbuffers/protobuf#7902 and protocolbuffers/protobuf#7908, but namespace_packages is still there (while it is an attribute of Why we need to do it
Technically speaking, if a package is already in one of the styles it should not be migrated to avoid breckages, as the documentation warns about:
But protobuf has already migrated to native style packages in 3.14.0, so the rest of the google packages (see the list above) remained in |
|
@busunkim96 @noahdietz FYI (see the previous comment) |
|
@software-dov Hey Dov! Can you please take a look at the comment above and tell me if that makes sense from the Python perspective (i.e. provide some Python expert opinion, because the complexity of this issue is way above my Python level). |
* Use native namespace for google (remove google/__init__.py) * require python>=3.6 * remove 2.7 unit tests (There is more 2.7 cleanup left, but I am scoping down the changes in the interest of getting this done quickly for googleapis/gapic-generator#3334)
We are not ready to require >=3.6 in the main branch, so I am opening this PR against a new `2.0.0dev` branch. For googleapis/gapic-generator#3334 (comment) Use native namespaces (delete `google/__init__.py` and require python>=3.6). There is more cleanup that needs to happen (example: App Engine credentials) in future PRs.
|
I have cut pre-releases of
|
…kaged as native namespace package) (#753) More details about Python namespace packaging here: https://packaging.python.org/guides/packaging-namespace-packages/#native-namespace-packages The protobuf changes, which made this fix necessary are here: protocolbuffers/protobuf#7902 protocolbuffers/protobuf#7908 The tracking bug for this issue (probably not the only one) googleapis/gapic-generator#3334 This is only part of the fix, for the proper fix other google-namespaced python packages must be migrated to [pkgutil-style-namespace-packages](https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages) and make sure they **do not** have `namespace_packages` in their `setup.py` file (an artifact from the legacy `pkg_resources-style`packages)
This fixes the googleapis/gapic-generator#3334 by excluding system-wide site-packages dir from python packages resolution path completely. This pretty much implements the long-standing featrue request for rules_python bazelbuild/bazel#4939, but only in scope of gapic-generator-python.
* fix: Fix namespace packages conflict issue This fixes the googleapis/gapic-generator#3334 by excluding system-wide site-packages dir from python packages resolution path completely. This pretty much implements the long-standing featrue request for rules_python bazelbuild/bazel#4939, but only in scope of gapic-generator-python. * Format with autopep8
First saw around 2021-01-21 05:00.
https://fusion.corp.google.com/runanalysis/test/prod:cloud-devrel%2Fclient-libraries%2Fautosynth%2Fpython/prod:cloud-devrel%2Fclient-libraries%2Fautosynth%2Fpython/KOKORO/bd126a99-ddcf-4060-b192-d54993a500bc/1611250467223/build%20%231040/Targets?target=logs%2Fpython-bigquery&drilldownTab=logs
The text was updated successfully, but these errors were encountered: