Make Key.toUrlSafe() consistent with the URL-safe key displayed in Google Cloud Datastore Console

[sai-pullabhotla]

The Key.toUrlSafe() returns a URL encoded (similar to URLEncoder.encode) version of the key like the one below:

partition_id+%7B%0A++project_id%3A+%22my-project%22%0A++namespace_id%3A+%22MyNamespace%22%0A%7D%0Apath+%7B%0A++kind%3A+%22MyEntity%22%0A++name%3A+%22MyKey%22%0A%7D%0A

The Google Cloud Datastore Console displays a Base64 encoded version of the key, which looks something like -

agVoZWxsb3IPCxIHQWNjb3VudBiZiwIM

The Console allows filtering on key by URL-safe key, which works fine if I use the URL-safe value displayed by the GCD Console, but does not work when using the value returned from Key.toUrlSafe().

The Console Documentation states that -

URL-safe key

This is a base64-encoded, serialized version of your entity key. The encoding method is available in any Cloud Datastore client library

Example: agVoZWxsb3IPCxIHQWNjb3VudBiZiwIM

So, the client libraries are expected to match with the Base64 version? Nonetheless, it would be nice to have the client libraries and Console in sync to help with crosschecking/debugging purposes.

[mziccard]

@sai-pullabhotla This is interesting. @aozarov any reason our toUrlSafe is different from the one used in the cloud console? Do you think it's possible this changed over time and we failed to keep up-to-date?

[aozarov]

I think that at the time that we provided the toUrlSafe it only worked for this library (there was no "standard"). I am glad to see that now there is and we should change the code to adopt it.

[mziccard]

Then @sai-pullabhotla thanks for bringing this up. I'll work on a fix soon.

[mziccard]

@eddavisson @pcostell can you guys help me understand how the base64 URL-safe key shown in the Cloud Console is computed?

I expected it to match the following code:

BaseEncoding.base64Url().omitPadding().encode(toPb().toByteArray());

But that is not the case.

[pcostell]

The base64 URL-safe key shown in the console only works with the App Engine client libraries. If you think it would be useful for it to provide a URL-safe key usable in the Google Cloud client libraries, please send feedback in the Cloud Console so it can get triaged by the right team.

[mziccard]

@pcostell Can't we generate this base64 URL-safe in Google Cloud? I think it would make sense for us to generate that value as it is used in AE libraries as well as in the Cloud console.

[JoshuaFox]

Parsing and generating URL-Safe keys in the format of the Google App Engine API is essential for us to be able to use the Google Cloud API.

[lesv]

@JoshuaFox Are you asking for compatibility within the library, or between GAE Standard API's / Cloud Console and this library?

Also, how would it affect you if we changed the format generated by this library to be compatible with GAE Standard / Cloud Console?

[JoshuaFox]

I'm not sure what "compatibility within the library" means, but of course I'd like all kinds of compatibility. Specifically, we have URL-Safe keys generated in the GAE API which need to be parsed into Key objects in Google Cloud API; and the Google Cloud API code has to be able to generate keys in the GAE URL-Safe string format.

The Cloud Console is less important.

In any case, we have since Jan 12 written code that digs into this format and does the above conversions, so for us it is less urgent; though still, this compatibility seems like an obvious requirement.

[lesv]

@JoshuaFox What I'm trying to ask is, if we fix (ie. change the current behavior of the google-cloud API) to match what the GAE API does, would it break your app?

[JoshuaFox]

I don't think it would break my app.

[lesv]

@garrettjonesgoogle I think we should fix it, and the sooner, the better. (ie. before GA)

[lesv]

@jabubake PTAL - you might wish to inquire / make sure that that thing your working on is compatible in this.

[lesv]

UrlSafe keys are allowed to be used in URL's (ie. bookmark-able). Developers who move / augment their GAE Standard apps with GAE Flex apps will want to easily move between them. Not having this will block the adoption of GAE Flex for many developers.

[lesv]

Note - we should rename the existing toSafeUrl() to toOldSafeUrl() and fromUrlSafe() to fromOldUrlSafe as well, so that we don't mess up too many users.

Alternatively, but equally bad, we should create toGAESafeUrl() and fromGAEUrlSafe().

[garrettjonesgoogle]

Maybe: create toBase64SafeUrl() and fromBase64SafeUrl() so that the reasons to choose one vs the other is clearer, and not so tied to a specific technology that is incidental.

[garrettjonesgoogle]

Since we will be adding methods instead of changing them, this isn't a breaking change any more.

[lesv]

There has been some offline discussion and it doesn't seem like this can be fixed for GA, and might not be able to be fixed at all.

[garrettjonesgoogle]

It turns out that replicating the base 64 format of the cloud console in open source isn't possible right now; it requires a proto that isn't open source, and an app id that isn't available in the client library context.

The solution is probably for cloud console to produce an alternative base 64 format that the client library can reproduce.

[JoshuaFox]

My requirement was for comparability between Datastore Cloud API and Datastore GAE API; the Cloud Console (the GUI) is less relevant.

[garrettjonesgoogle]

@JoshuaFox ok, but the issue is the same anyway, right?

[JoshuaFox]

I don't necessarily know the implementation details, but if the Cloud Console value is generated by the GAE API; in contrast to the Cloud API; then yes.

[koichirokamoto]

Is there any plan to add implementation comparable to GAE API?
FWIW, in cloud golang library, implementation is at https://github.com/GoogleCloudPlatform/google-cloud-go/blob/master/datastore/key.go#L198.
Here is GAE API at https://github.com/golang/appengine/blob/master/datastore/key.go#L211.
It seems they are comparable. I would hope that implementation is also added in Java library.

[danoscarmike]

@garrettjonesgoogle have you had a chance to check if Go can achieve this compatibility why Java cannot?

[garrettjonesgoogle]

Theoretically Java could, it would just require duplicating code instead of reusing it.

[garrettjonesgoogle]

This has been added to our feature backlog: https://github.com/GoogleCloudPlatform/google-cloud-java/wiki/Feature-backlog . This issue will be closed but is linked in the backlog and can continue to be used for comment and discussion.

[kwiesmueller]

How is the status on this?
This issue seems to persist for quite some time now and we are unable to use the url safe key spanning services built in various languages due to this.