Google Cloud Dataflow/Apache Beam and gcloud-java-core

[lukecwik]

We have two external pull requests which add logic to support the active gcloud profile, and also to get the project from the service account which are already supported by gcloud-java-core.

Dataflow/Apache Beam would like to depend on gcloud-java-core for the project detection logic found here: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-core/src/main/java/com/google/cloud/ServiceOptions.java?l=66

And also the auth credentials detection here: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/gcloud-java-core/src/main/java/com/google/cloud/ServiceOptions.java?l=237

Currently these are hidden away, is there a way for you to expose these concepts without needing to create a ServiceOptions subclass?

If not, how do you advise Dataflow/Apache Beam to integrate with gcloud-java-core?

Note, that Apache Beam would be interested in the python versions of that code, and in general this would be useful in the other language implementations as well.

[mziccard]

Hi @lukecwik!

Currently these are hidden away, is there a way for you to expose these concepts without needing to create a ServiceOptions subclass?

I can have a look and see if we can expose those functionalities, perhaps as static methods. Out of curiosity (I can thing of a couple of valid reasons) what blocks you from subclassing ServiceOptions?

[lukecwik]

The constructor for ServiceOptions takes in a bunch of content which seems like it would be brittle to extend with the least amount of code.

Implementing defaultServiceFactory and defaultRpcFactory seems to have the same issues.

[lukecwik]

Ping?

[mziccard]

@lukecwik Sorry I haven't had the time to work on this yet. I'll try to draft a PR soon.

Did you consider using depending on google-auth-library for the credentials detection part?

[lukecwik]

We already depend on the application default credentials, unfortunately the Apiary generated libraries use a credentials object which is different than the credentials object used by gRPC generated libraries in Java (they are very similar, but not the same package). It seems as though gcloud-java-core creates an abstraction that sits on top of both of these.

[mziccard]

gRPC generated libraries in Java

Can you provide more details on this? What class are you referring to? Maybe CallCredentials?

It seems as though gcloud-java-core creates an abstraction that sits on top of both of these.

We surely sit on top of google-auth-library but I wouldn't say we sit on top of any gRPC-related credentials class. Also, the long term plan is for us to directly expose classes in google-auth-library as soon as those support all the functionality we need.

[lukecwik]

Apiary generated clients use:
com.google.api.client.googleapis.auth.oauth2.GoogleCredential

gRPC generated clients use:
com.google.auth.oauth2.GoogleCredentials

I though glcoud-java-core sat on top of both, but it seems as though HttpCredentialsAdapter was created which converts a com.google.auth.oauth2.GoogleCredentials to be a HttpRequestInitializer that is compatible with Apiary generated clients and gRPC generated clients use com.google.auth.oauth2.GoogleCredentials directly.

AuthCredentials was what I was thinking of was your common point between Apiary and gRPC generated code and it was what I was referring to as the common abstraction point that Dataflow could depend on/expose instead of exposing com.google.api.client.googleapis.auth.oauth2.GoogleCredential or com.google.auth.oauth2.GoogleCredentials

What functionality are you waiting on from google-auth-library?

[mziccard]

What functionality are you waiting on from google-auth-library?

Main blockers at the moment are that those classes are not Serializable and do not expose a method to sign byte arrays. Are you going to need this?

[lukecwik]

Signing isn't important for us yet, Serializable could be useful for our end users but not useful for us since Dataflow uses the GCE VM credentials, but for other runners like Spark/Flink, this could be very useful.

How do you deal with short lived credentials, do you have hooks into expiry time?

[mziccard]

How do you deal with short lived credentials, do you have hooks into expiry time?

Credentials expose a refresh method. The HttpCredentialsAdapter automatically refreshes tokens if needed (see HttpCredentialsAdapter.handlerResponse).

You can also create AuthCredentials from an existing accessToken (see createFor(String accessToken, Date expirationTime)), these are one-time-only credentials. Same applies to google-auth-library credentials created from just an access token (see this constructor).