-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gcloud NIO permissions bug introduced in 0.90.0 #5
Comments
Putting together an easily sharable example failure is a bit tricky since I've only been able to repeat it on travis and you need to provide an account, but if it would be helpful I can probably set up a tiny github project which would exhibit the failure. |
@jean-philippe-martin If you have any time to look into this it would be useful. |
Is there anything I can do to help move this along? It's blocking us from upgrading to newer versions of the library that have important bug fixes. |
Yes, some small repro where I have enough permissions to get Travis to run and to look at its logs would help! |
I've created a branch in the gatk project that is a much more precise repro. I just deleted almost everything from the repo and hacked it down to just be a test while keeping all the authentication stuff the same. The branch is lb_nio_bug_test . You have permissions to push to the repo, feel free to either copy the branch or work with that one directly. The test which fails is
It fails with the following exception:
That branch sets up a matrix of travis builds of different versions of the library. You can see that it starts failing at version 0.90.0: https://travis-ci.com/broadinstitute/gatk/builds/123451910 I also noticed that in version 0.90.0+ you get an NPE due to a null project when callings I also got in a weird state where the same method failed with different permission issues saying that accessing that file requires you to be authenticated because it's set as requester pays. However, that bucket is not set as requester pays so it seems wrong...
I can try to track down the exact causes of those other error modes, but they aren't blockers for us at the moment. In any case, it seems like something changed in 0.90.0 which makes checking |
@jean-philippe-martin Also, I think you know this already, but you can get the test results after the travis build, it outputs the path it's going to upload the reports too. |
Looking at the commits in 0.90.0 it seems very likely to me that the bug was introduced here: |
Looking deeper, I think the bug is here: I suspect the project being picked corresponds to the app engine project that travis is running instead of the service account project that I've authenticated as. |
Yup... that seems to be it.
|
👍 awesome! |
* Updating google-cloud-nio 0.81.0-alpha:shaded -> 0.107.0-alpha:shaded * Update picard version 2.20.5 -> 2.20.7 * Fixes #5935 * Exclude transitive dependencies from shaded dependencies. The shaded poms seem to include copies of the dependencies that were shaded which means we get multiple confusing copies of the same class. * Added a workaround on travis https://github.com/googleapis/google-cloud-java/issues/5884, an issue introduced in gcloud 0.90.0-alpha. This required redundantly specifying the google project through an environment variable in addition to logging in.
Hi @lbergelson, Are you still seeing this issue? Apologies for the delay in response. If it is, could you do me a favor and provide request logs by following these instruction. Thank you. |
IIUC, setting environment variable I'm closing due to inactivity. If the issue still persists please don't hesitate to reopen. |
Environment details
Steps to reproduce
gs://
files on travis-ciExplanation and theory
We encountered a new 403 error when using nio to access files in gcs after attempting to update from 0.81.0 -> anything 0.90.0 or greater.
This error occurs only when we run our test suite in travis-ci. Notably, travis-ci runs it's worker nodes on google cloud.
Our tests authenticate using a service account:
Prior to v0.90.0 everything worked fine. After updating to 0.90.0 we started getting this error:
The service account listed is our authenticated account. However the project ID is NOT the project we are attempting to access and it's not a project we own. After a bit of digging we discovered that the project belongs to travis-ci.
So it seems like some change between 0.89.0 and 0.90.0 is causing it to incorrectly make a request against the project that the host machine belongs to, rather than the account that is logged in.
Stack trace
The Project
Example travis log
Example test report
Example stack trace:
The text was updated successfully, but these errors were encountered: