blob.download_to_filename fails with google-cloud-storage==1.3.0 #3736
Comments
|
@sonlac It looks like you're using a custom transport. When constructing For reference, see the release notes: |
|
@dhermes Thank you for your quick reply. No, I didn't pass anything as custom argument. I am using the latest code to test, all arguments are default ones. |
|
Before making a request, can you print |
|
RE: #3705, notice the code in |
|
@sonlac Can you give us the output of If it is a semi-old one, can you also try updating |
lukesneeringer
added
status: acknowledged
status: investigating
|
Thank you @dhermes @lukesneeringer for the comments. So, the problem 1 is about "user-beware". It's was not a bug. For the problem 2, I could not reproduce it in a context "out-of-box". I've just tested a simple code using download_to_filename, it worked with latest ones: google-cloud 0.27.0 and goole-cloud-storage 1.3.0. In fact, I've run the code inside a ML engine, so it'd take time to get more logs... I'll back to you for more information. pip freeze Click to expandcachetools==2.0.0 certifi==2017.7.27.1 chardet==3.0.4 dill==0.2.7.1 enum34==1.1.6 future==0.16.0 futures==3.1.1 gapic-google-cloud-datastore-v1==0.15.3 gapic-google-cloud-error-reporting-v1beta1==0.15.3 gapic-google-cloud-logging-v2==0.91.3 gapic-google-cloud-pubsub-v1==0.15.4 gapic-google-cloud-spanner-admin-database-v1==0.15.3 gapic-google-cloud-spanner-admin-instance-v1==0.15.3 gapic-google-cloud-spanner-v1==0.15.3 google-auth==1.0.2 google-cloud==0.27.0 google-cloud-bigquery==0.26.0 google-cloud-bigtable==0.26.0 google-cloud-core==0.26.0 google-cloud-datastore==1.2.0 google-cloud-dns==0.26.0 google-cloud-error-reporting==0.26.0 google-cloud-language==0.27.0 google-cloud-logging==1.2.0 google-cloud-monitoring==0.26.0 google-cloud-pubsub==0.27.0 google-cloud-resource-manager==0.26.0 google-cloud-runtimeconfig==0.26.0 google-cloud-spanner==0.26.0 google-cloud-speech==0.28.0 google-cloud-storage==1.3.0 google-cloud-translate==1.1.0 google-cloud-videointelligence==0.25.0 google-cloud-vision==0.26.0 google-gax==0.15.13 google-resumable-media==0.2.2 googleapis-common-protos==1.5.2 grpc-google-iam-v1==0.11.1 grpcio==1.4.0 httplib2==0.10.3 idna==2.5 monotonic==1.3 oauth2client==3.0.0 ply==3.8 proto-google-cloud-datastore-v1==0.90.4 proto-google-cloud-error-reporting-v1beta1==0.15.3 proto-google-cloud-logging-v2==0.91.3 proto-google-cloud-pubsub-v1==0.15.4 proto-google-cloud-spanner-admin-database-v1==0.15.3 proto-google-cloud-spanner-admin-instance-v1==0.15.3 proto-google-cloud-spanner-v1==0.15.3 protobuf==3.3.0 pyasn1==0.3.2 pyasn1-modules==0.0.11 requests==2.18.3 rsa==3.4.2 six==1.10.0 tenacity==4.4.0 urllib3==1.22 |
|
@sonlac I am going to preemptively close this based on
We're happy to keep discussing and re-open if there does turn out to be some issue. |
|
Hey guys, faced same issue here. Code ran fine on my local mac machine and ubuntu VM in Compute Engine, but failed on Debian VM in Compute engine that was deployed through Salt.
|
|
@lukesneeringer @jonparrott We should probably do a release of |
|
Seems reasonable |
Did this to update the lower bound on the `requests` dependency (see googleapis#3736).
Did this to update the lower bound on the `requests` dependency (see googleapis#3736).
* Update storage to 1.3.2. Did this to update the lower bound on the `requests` dependency (see #3736). * Updating minimum bound on `requests` in storage.
|
Hello, I'm having the same problem here. It's really easy for me to reproduce, and I'm using a Google Cloud Compute VM running ubuntu 16.0.4. I don't think it's just a cloud-storage/requests versioning issue as I have the latest version of both installed, as well as the latest of google-cloud I can reproduce the bug really simply. from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket(<STORAGE_BUCKET>)
blob = bucket.get_blob(<PATH_TO_VIDEO>)
blob.download_to_filename(<LOCAL_PATH_TO_VIDEO>)will raise Finally, here is my pip freeze (pip3 freeze): |
|
@ashwathnrajan You have |
|
meh. one day i'll learn to compare numbers. thanks, i'm sure this will fix it |
|
I'm getting this error with the follow versions this is my entire pip freeze output Almost identical virtualenv on my mac won't raise this error. The error I'm getting is on debian Any advice would be appreciated. |
|
@naoko Run |
|
@dhermes , thank you for your quick response. It seems that version matches to what shows on pip freeze... :( |
|
Is the Location ( On the breaking machine, can you do the following: |
|
For some reason, running that in Jenkins was so hard (looks like it strips single quote...) But the goal is to show that version of |
|
I don't know if it will help but I was able to solve the second issue ( client = storage.Client(PROJECT_NAME) #without PROJECT_NAME it breaks
bucket = client.get_bucket(bucket_name)
blob = bucket.get_blob(filename)
return blob.download_as_string() |
|
Hey there, I'm experiencing this issue too. removing with statement in line 117 appears to monkey patch it |
|
@swirlingsand are you sure you have requests 2.18.4? Can you verify with |
|
My apologies I think this was something with my conda setup. |
|
No worries!
…On Thu, Feb 22, 2018, 5:37 PM Anthony Sarkis ***@***.***> wrote:
My apologies I think this was something with my conda setup.
pip show requests yielded 2.18,
however running that print statement showed 2.14
Thanks for help @jonparrott <https://github.com/jonparrott>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3736 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPUc6G-tqGhrSySz7e2USQPsEdhjltfks5tXhZugaJpZM4OvlIi>
.
|
|
I had to uninstall requests 2.18.4 (pip uninstall requests) and install 2.18.0 (pip install requests==2.18.0) to make it work |
|
I downgraded |
|
Hi all, any idea on how to make this work on Cloud Composer? I am migrating DAGs from our current environment to Cloud Composer however I get this error even when I install google-cloud-storage==1.6.0 and requests==2.18.0 as PyPI package. [2018-08-20 09:32:09,128] {base_task_runner.py:98} INFO - Subtask: [2018-08-20 09:32:09,127] {bash_operator.py:101} INFO - from google.cloud import bigquery |
|
Hi @jocieA, with your logs I am assuming that your problem is not in the module google-cloud-storage==1.6.0 but rather in the module google-cloud-bigquery. You can do a Concretely, I saw in your logs, the error Hope this helps. Thanks. |
|
For anyone else running into this issue with Google Cloud Composer, I ran into it after adding apache-beam[gcp]==2.6.0 to my Composer dependencies. The issue here was that apache-beam is installing google-cloud-bigquery==0.25.0, which causes a bunch of other packages to be downgraded. The fix for me was to explicitly state/install google-cloud-core>=0.28.0 google-cloud-bigquery>=1.5.0 AFTER apache-beam[gcp]==2.6.0 in requirements.txt file used to specify Composer dependencies: |
|
Hi @sjungwirth and @sonlac , Thanks for getting back on my post, these have been very useful. |
|
Is this issue resolved ? what is the solution |
Due to the notable implementation changes in google-cloud-storage 1.3.0, the blob.download_to_filename fails whatever google-cloud from 0.24.0 to 0.27.0 (latest)
Our prod server which using google-cloud==0.24.0 has been broken since last Saturday. I analyzed a bit the error. The problem is come from the upgraded package google-cloud-storage==1.3.0. We are using the google-cloud-storage==1.2.0.
I did several tests which figured out the following problems:
Problem 1: blob.download_to_filename failed when using google-cloud==0.27.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:
Problem 2: blob.download_to_filename failed when using google-cloud==0.24.0 (included google-cloud-storage==1.3.0 as default, defined in setup.py). Here is a part of stacktrace:
So, why it happened? I realized that in the REQUIREMENTS of the package google-cloud which is defined in the file setup.py. An example for google-cloud==0.27.0 https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67
We can see that if we only define google-cloud==[0.24.0 to 0.27.0], "pip install" always try to install google-cloud-storage 1.3.0 which requires the dependencies google-cloud-core ~= 0.26. That's why the bug happens.
Solution:
Suggestion: better manage the deps for google-cloud in setup.py.
Thanks.
The text was updated successfully, but these errors were encountered: