Strange Firestore Installation Issue in Compute Engine #5023

hiranya911 · 2018-03-12T23:52:21Z

I encountered the following strange behavior on a Google Compute Engine node (in a Python 3 virtualenv sandbox).

I installed the firebase-admin module which has a dependency on google-cloud-firestore:

$ pip install firebase-admin
Collecting firebase-admin
  Downloading firebase_admin-2.9.0-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 2.4MB/s 
Collecting google-cloud-storage>=1.2.0 (from firebase-admin)
  Downloading google_cloud_storage-1.8.0-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 4.3MB/s 
Requirement already satisfied: six>=1.6.1 in /home/hkj/py3/lib/python3.5/site-packages (from firebase-admin)
Collecting requests>=2.13.0 (from firebase-admin)
  Downloading requests-2.18.4-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 3.7MB/s 
Collecting google-auth>=1.3.0 (from firebase-admin)
  Downloading google_auth-1.4.1-py2.py3-none-any.whl (65kB)
    100% |████████████████████████████████| 71kB 4.6MB/s 
Collecting google-cloud-firestore>=0.27.0 (from firebase-admin)
  Downloading google_cloud_firestore-0.29.0-py2.py3-none-any.whl (105kB)
    100% |████████████████████████████████| 112kB 4.1MB/s 
Collecting google-resumable-media>=0.3.1 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_resumable_media-0.3.1-py2.py3-none-any.whl
Collecting google-cloud-core<0.29dev,>=0.28.0 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_cloud_core-0.28.1-py2.py3-none-any.whl
Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage>=1.2.0->firebase-admin)
  Downloading google_api_core-1.0.0-py2.py3-none-any.whl (51kB)
    100% |████████████████████████████████| 61kB 5.4MB/s 
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/hkj/py3/lib/python3.5/site-packages (from requests>=2.13.0->firebase-admin)
Collecting idna<2.7,>=2.5 (from requests>=2.13.0->firebase-admin)
  Downloading idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 5.1MB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.13.0->firebase-admin)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 4.0MB/s 
Collecting certifi>=2017.4.17 (from requests>=2.13.0->firebase-admin)
  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
    100% |████████████████████████████████| 153kB 4.4MB/s 
Collecting rsa>=3.1.4 (from google-auth>=1.3.0->firebase-admin)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 8.1MB/s 
Collecting cachetools>=2.0.0 (from google-auth>=1.3.0->firebase-admin)
  Downloading cachetools-2.0.1-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.2.1 (from google-auth>=1.3.0->firebase-admin)
  Downloading pyasn1_modules-0.2.1-py2.py3-none-any.whl (60kB)
    100% |████████████████████████████████| 61kB 6.7MB/s 
Collecting googleapis-common-protos<2.0dev,>=1.5.3 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading googleapis-common-protos-1.5.3.tar.gz
Collecting protobuf>=3.0.0 (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading protobuf-3.5.2-cp35-cp35m-manylinux1_x86_64.whl (6.4MB)
    100% |████████████████████████████████| 6.4MB 218kB/s 
Collecting pytz (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
  Downloading pytz-2018.3-py2.py3-none-any.whl (509kB)
    100% |████████████████████████████████| 512kB 2.6MB/s 
Requirement already satisfied: setuptools>=34.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core<2.0.0dev,>=0.1.1->google-cloud-storage>=1.2.0->firebase-admin)
Collecting pyasn1>=0.1.3 (from rsa>=3.1.4->google-auth>=1.3.0->firebase-admin)
  Downloading pyasn1-0.4.2-py2.py3-none-any.whl (71kB)
    100% |████████████████████████████████| 71kB 10.0MB/s 
Building wheels for collected packages: googleapis-common-protos
  Running setup.py bdist_wheel for googleapis-common-protos ... done
  Stored in directory: /home/hkj/.cache/pip/wheels/93/e0/cb/b06866f012310b96fba79c37f199aaf973a2e97a42ca7ef696
Successfully built googleapis-common-protos
Installing collected packages: google-resumable-media, protobuf, googleapis-common-protos, pytz, idna, chardet, certifi, requests, pyasn1, rsa, cachetools, pyasn1-modules, google-auth, google-api-core, google-cloud-core, google-cloud-storage, google-cloud-firestore, firebase-admin
Successfully installed cachetools-2.0.1 certifi-2018.1.18 chardet-3.0.4 firebase-admin-2.9.0 google-api-core-1.0.0 google-auth-1.4.1 google-cloud-core-0.28.1 google-cloud-firestore-0.29.0 google-cloud-storage-1.8.0 google-resumable-media-0.3.1 googleapis-common-protos-1.5.3 idna-2.6 protobuf-3.5.2 pyasn1-0.4.2 pyasn1-modules-0.2.1 pytz-2018.3 requests-2.18.4 rsa-3.4.2

As you can see at the end it has installed google-cloud-firestore-0.29.0 package. But then I got the following error, trying to use it:

>>> import firebase_admin
>>> firebase_admin.initialize_app()
<firebase_admin.App object at 0x7f2f77fb95f8>
>>> from firebase_admin import firestore
Traceback (most recent call last):
  File "/home/hkj/py3/lib/python3.5/site-packages/firebase_admin/firestore.py", line 22, in <module>
    from google.cloud import firestore # pylint: disable=import-error,no-name-in-module
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore.py", line 18, in <module>
    from google.cloud.firestore_v1beta1 import __version__
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore_v1beta1/__init__.py", line 21, in <module>
    from google.cloud.firestore_v1beta1._helpers import GeoPoint
  File "/home/hkj/py3/lib/python3.5/site-packages/google/cloud/firestore_v1beta1/_helpers.py", line 24, in <module>
    import grpc
ImportError: No module named 'grpc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hkj/py3/lib/python3.5/site-packages/firebase_admin/firestore.py", line 28, in <module>
    raise ImportError('Failed to import the Cloud Firestore library for Python. Make sure '
ImportError: Failed to import the Cloud Firestore library for Python. Make sure to install the "google-cloud-firestore" module.

Installing google-cloud-firestore separately resolved the issue:

$ pip install google-cloud-firestore
Requirement already satisfied: google-cloud-firestore in /home/hkj/py3/lib/python3.5/site-packages
Requirement already satisfied: google-cloud-core<0.29dev,>=0.28.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-cloud-firestore)
Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=0.1.1 in /home/hkj/py3/lib/python3.5/site-packages (from google-cloud-firestore)
Requirement already satisfied: six>=1.10.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.3 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: setuptools>=34.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pytz in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: protobuf>=3.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Collecting grpcio>=1.8.2; extra == "grpc" (from google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
  Downloading grpcio-1.10.0-cp35-cp35m-manylinux1_x86_64.whl (7.5MB)
    100% |████████████████████████████████| 7.5MB 186kB/s 
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: idna<2.7,>=2.5 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: certifi>=2017.4.17 in /home/hkj/py3/lib/python3.5/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: rsa>=3.1.4 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: cachetools>=2.0.0 in /home/hkj/py3/lib/python3.5/site-packages (from google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.1 in /home/hkj/py3/lib/python3.5/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0.0dev,>=0.4.0->google-api-core[grpc]<2.0.0dev,>=0.1.1->google-cloud-firestore)
Installing collected packages: grpcio
Successfully installed grpcio-1.10.0

Any idea why grpcio did not install in the first attempt?

The text was updated successfully, but these errors were encountered:

chemelnucfin · 2018-03-12T23:59:54Z

Hello, firebase_admin is a separate library, please direct the question here:
firebase_admin

Please let us know if you encounter any more problems.

hiranya911 · 2018-03-13T01:10:49Z

I am the maintainer of the firebase_admin library. I'd like to know why the google-cloud-firestore won't install when we have declared the dependency as follows in setup.py:

install_requires = [
    'google-auth>=1.3.0',
    'google-cloud-firestore>=0.27.0',
    'google-cloud-storage>=1.2.0',
    'requests>=2.13.0',
    'six>=1.6.1'
]

Please reopen.

theacodes · 2018-03-13T01:32:06Z

This is very likely a bug in pip.

I can reproduce this pretty readily. I believe it's because you depend on google-cloud-firestore and google-cloud-storage. It happens because they declare their dependency on google-api-core slightly differently.

google-cloud-firestore requires google-api-core[grpc]<2.0.0dev,>=0.1.1.
google-cloud-storage requires google-api-core<2.0.0dev,>=0.1.1.

I think what's happening is that pip picks up the second one and ignores the extra declared in the first. See this line in the log:

Collecting google-api-core<2.0.0dev,>=0.1.1 (from google-cloud-storage>=1.2.0->firebase-admin)

Summoning @ncoghlan and @dstufft to see if they have thoughts on how best to either resolve this or report upstream.

theacodes · 2018-03-13T01:37:38Z

I think this is the same as pypa/pip#4957

@pradyunsg I'm happy to write a failing test case for pip that demonstrates this.

chemelnucfin · 2018-03-13T01:42:31Z

@hiranya911 My apologies.

pradyunsg · 2018-03-13T05:18:41Z

@pradyunsg I'm happy to write a failing test case for pip that demonstrates this.

Thanks for the mention @jonparrott! Please do so and file an issue over at pradyunsg/zazo. :)

This is very likely a bug in pip.

It is basically pypa/pip#4957 which is a specific case of pypa/pip#988.

You'll have to work around this by actually pinning your dependencies in the top-level requirements that pip install gets or using something like pip-tools (or pipenv?) on top that adds proper dependency resolution.

hiranya911 · 2018-03-13T06:06:33Z

Thanks @jonparrott and @chemelnucfin for looking into this. Should firebase-admin take a direct dependency on google-api-core[grpc]<2.0.0dev,>=0.1.1 (as a temp workaround at least)?

pradyunsg · 2018-03-13T06:14:00Z

Should firebase-admin take a direct dependency on google-api-core[grpc]<2.0.0dev,>=0.1.1 (as a temp workaround at least)?

Yes -- you'll have to keep this in sync with whatever google-cloud-storage and google-cloud-firestore does though.

theacodes · 2018-03-13T16:02:35Z

@hiranya911, I agree with @pradyunsg: take a dependency on google-api-core[grpc]. Drop your direct dependency on requests and google-auth.

Thanks, @pradyunsg for your insight! I'll try to write a failing test case sometime this week, time permitting. :)

theacodes · 2018-03-22T19:44:04Z

Closing this for now, as we have no actionable items for this repository.

constantinos07 · 2018-07-07T14:19:35Z

@hiranya911 I am a beginner in appegine and firestore, facing the same problem, which i describe in firebase/firebase-admin-python#179

Can you elaborate how did you fix yours?

tseaver · 2018-07-09T15:40:05Z

@constantinos07 Until the pip bug is fixed, or firebase-admin-python adds the explicit pin needed to work around it, you can work around this by creating an explicit requirements.txt which pins the needed versions, E.g.:

grpcio==1.13.0rc3
google-cloud-firestore==0.29.0
google-cloud-storage==1.10.0

hiranya911 · 2018-08-02T22:18:27Z

We define it as >= 0.27.0. So it ends up installing 0.29.0. This issue also seems to occur randomly. For instance I don't see it happening in our CI builds: https://travis-ci.org/firebase/firebase-admin-python/jobs/410999811

chemelnucfin closed this as completed Mar 12, 2018

theacodes reopened this Mar 13, 2018

theacodes self-assigned this Mar 13, 2018

theacodes added the packaging label Mar 13, 2018

tseaver added the api: firestore Issues related to the Firestore API. label Mar 16, 2018

theacodes closed this as completed Mar 22, 2018

hiranya911 mentioned this issue Jul 9, 2018

Firebase- google-cloud-firestore error firebase/firebase-admin-python#179

Closed

FreakTheMighty mentioned this issue Jul 29, 2018

ModuleNotFoundError: No module named 'google.auth' firebase/firebase-admin-python#184

Closed

evanj mentioned this issue Aug 8, 2018

datastore: pip install bigquery, datastore fails to install grpc #5776

Closed

TheKevJames mentioned this issue Aug 9, 2018

Extras not getting installed when the package that includes the extras has already been installed pypa/pip#4957

Closed

hiranya911 mentioned this issue Dec 21, 2018

Adding google-api-core[grpc] dependency firebase/firebase-admin-python#241

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strange Firestore Installation Issue in Compute Engine #5023

Strange Firestore Installation Issue in Compute Engine #5023

hiranya911 commented Mar 12, 2018

chemelnucfin commented Mar 12, 2018

hiranya911 commented Mar 13, 2018

theacodes commented Mar 13, 2018

theacodes commented Mar 13, 2018

chemelnucfin commented Mar 13, 2018

pradyunsg commented Mar 13, 2018

hiranya911 commented Mar 13, 2018

pradyunsg commented Mar 13, 2018

theacodes commented Mar 13, 2018

theacodes commented Mar 22, 2018

constantinos07 commented Jul 7, 2018

tseaver commented Jul 9, 2018

hiranya911 commented Aug 2, 2018

Strange Firestore Installation Issue in Compute Engine #5023

Strange Firestore Installation Issue in Compute Engine #5023

Comments

hiranya911 commented Mar 12, 2018

chemelnucfin commented Mar 12, 2018

hiranya911 commented Mar 13, 2018

theacodes commented Mar 13, 2018

theacodes commented Mar 13, 2018

chemelnucfin commented Mar 13, 2018

pradyunsg commented Mar 13, 2018

hiranya911 commented Mar 13, 2018

pradyunsg commented Mar 13, 2018

theacodes commented Mar 13, 2018

theacodes commented Mar 22, 2018

constantinos07 commented Jul 7, 2018

tseaver commented Jul 9, 2018

hiranya911 commented Aug 2, 2018