ADC can load a json of impersonated service account credentials generated by gcloud

[liuchaoren]

google-auth-library-java supported ADC impersonation and it is proposed in aip-dev/google.aip.dev#728.

[MadJlzz]

I have created a question around this issue on SO here.

From impersonated_credentials.py, this functionnality seems already implemented but when we try to actually use it in others libraries like google-cloud-storage, it fails with:

google.auth.exceptions.DefaultCredentialsError: The file C:\Users\klaer\AppData\Roaming\gcloud\application_default_credentials.json does not have a valid type. Type is impersonated_service_account, expected one of ('authorized_user', 'service_account', 'external_account').

Thas is thrown after I loged in with a service account I can impersonate:

gcloud auth application-default login --impersonate-service-account=<SA email>

and running a piece of code like this:

from google.cloud import storage

client = storage.Client()
print(next(client.list_buckets()))

Am I missing something? Can I do something to help you support this feature?

[liuchaoren]

I have created a question around this issue on SO here.

From impersonated_credentials.py, this functionnality seems already implemented but when we try to actually use it in others libraries like google-cloud-storage, it fails with:

google.auth.exceptions.DefaultCredentialsError: The file C:\Users\klaer\AppData\Roaming\gcloud\application_default_credentials.json does not have a valid type. Type is impersonated_service_account, expected one of ('authorized_user', 'service_account', 'external_account').

Thas is thrown after I loged in with a service account I can impersonate:

gcloud auth application-default login --impersonate-service-account=<SA email>

and running a piece of code like this:

from google.cloud import storage

client = storage.Client()
print(next(client.list_buckets()))

Am I missing something? Can I do something to help you support this feature?

[liuchaoren]

Sorry, closed by accident!

impersonation credentials were implemented in impersonated_credentials.py, what is missing is that ADC cannot read the file generated by the gcloud command and construct the impersonation credentials object. ADC is used by client libraries (for example storage) to construct credential objects.

I think what we need to do is extending the load_credentials_from_file to handle the impersonation credential json generated by gcloud and construct a impersonation credential object.

[MadJlzz]

Thanks for answering!

So yeah, that being said, the error message I am getting makes fully sense. I can try to do that if you think that it might help.

[MadJlzz]

@liuchaoren what do you think? Do you think I can help to implement this? (or even test it!!)

[liuchaoren]

@MadJlzz sorry for the late response! Yes, please see https://github.com/googleapis/google-auth-library-python/blob/main/CONTRIBUTING.rst for how to contribute to the repo.

[DBCerigo]

Is not supporting impersonated_service_account a regression? I'm quite sure that we were using service account impersonation with python clients back in August 2020 (ref: https://stackoverflow.com/questions/69412702/how-to-manage-google-cloud-credentials-for-local-development/69417606)

[MadJlzz]

@DBCerigo What I am sure of is that you can use it directly from the code (like in my SO thread) but not from the default ADC

[MadJlzz]

I made a PR with a first implementation that works for a simple example.

[sloev]

thanks for doing some work on this @MadJlzz
coming from AWS, (which has STS for temporary credentials), i must say google is quite inferior on this point.
Its a necessity to be able to run my code locally as a service account in order to develop the strict and narrow permissions my program needs once deployed.

the gcloud auth application-default login --impersonate-service-account=[SA EMAIL] is painful (too much friction when context switching between local projects) but it is utterly useless when pythons cloud sdk can't recognize the credentials produced.

crossing my fingers for this to be fixed soon :-)

[MadJlzz]

Hey @sloev, thanks for your message. I was out from some time and was stuck with the environment I was working (Windows)
Wasn't able to run all unit tests because spme are using os and not pathlib if I remind correctly. I will try to move forward on this issue to have a fix asap.

[liuchaoren]

Hi @MadJlzz we noticed some some issues which are blocked by this one. Do you think it is possible to fix this issue in one week? I am happy to take it over if it is not feasible. Thanks!

[MadJlzz]

Hi @liuchaoren. I don't think I have experience enough with the way your testing this project finish this in one week. Also the PR is waiting your review for the next milestone I have to achieve (which should be covering the new section of the code)
Happy to let you finish if you guys need to be quick on this one.

[liuchaoren]

@MadJlzz Thank you for your contribution! I am happy to pick it up from what you have. I will send a PR on the coming Monday.

[liuchaoren]

It seems that the scopes are not passed successfully to the impersonated credentials by the default() function. I think we should change the impersonated credentials to inherit the Scoped base class so that the default() can add scopes correctly to it. I will reopen it and will follow up with a PR to fix it.