Latest release of the SDK does not work with Google Datastore emulator

[rajwilkhu]

Environment details

• OS: MacOS
• .NET version: .Net Core 3.1
• Package name and version: Google.Cloud.Datastore.V1 3.0.0

Steps to reproduce

1. Start the emulator on the command line or via docker eg

 /google-cloud-sdk/platform/cloud-datastore-emulator/cloud_datastore_emulator start --host=0.0.0.0 --port=8081 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown /opt/data

2. Set up the code to connect to the Emulator

_datastoreDb = new DatastoreDbBuilder
            {
                EmulatorDetection = _identityStoreConfiguration.EmulatorDetection
            }.Build();

Fails with:

google-datastore_1  | [datastore] Sep 25, 2020 5:01:14 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
google-datastore_1  | [datastore] INFO: Adding handler(s) to newly registered Channel.
google-datastore_1  | [datastore] Sep 25, 2020 5:01:15 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
google-datastore_1  | [datastore] INFO: Detected non-HTTP/2 connection.
google-datastore_1  | [datastore] Sep 25, 2020 5:01:15 PM io.gapi.emulators.netty.NotFoundHandler handleRequest
google-datastore_1  | [datastore] INFO: Unknown request URI: /bad-request

Now, this is the odd bit. If I downgrade to the older version of the SDK i.e. Google.Cloud.Datastore.V1 2.2.0, it all works fine. I am using the latest GCloud SDK/emulator. Tried with an older version of the SDK, does not work. This is definitely broken in the latest version of the SDK.

Please help.

[jskeet]

Thanks for reporting this. I'll look into it on Monday.

[rajwilkhu]

Thanks @jskeet. Looked into this further. Seems like it's not using Http/2 when connecting to the emulator. The following workaround seems to work for me.

private DatastoreDb BuildDateDatastoreDbForProduction() =>
          new DatastoreDbBuilder
          {
            ProjectId = _identityStoreConfiguration.ProjectId,
            EmulatorDetection = _identityStoreConfiguration.EmulatorDetection
          }.Build();

        private DatastoreDb BuildDatastoreDbForLocalEmulator()
        {
          return DatastoreDb.Create(_identityStoreConfiguration.ProjectId, string.Empty, new DatastoreClientImpl(
            new Datastore.DatastoreClient(
              new Channel(endpoint[0], int.Parse(endpoint[1]), ChannelCredentials.Insecure)),
            new DatastoreSettings()));
        }

May provide a hint. Will have a look through the code in this repo as well. Sorry, just been lazy about it. Looks like it's just the builder that needs fixing.

[jskeet]

It should absolutely be using gRPC (and therefore HTTP 2) talking to the emulator. I'm hoping it'll be easy to sort out on Monday.

[jskeet]

Just starting to investigate this now. That's not a way I've used for starting the emulator before - I've always used the approach documented here. Please could you let me know what your DATASTORE_EMULATOR_HOST and DATASTORE_PROJECT_ID environment variables are set to?

[jskeet]

Okay, I've reproduced the error with the code below, with the following steps:

• In terminal 1:
  • Run gcloud beta emulators datastore start
• In terminal 2:
  • export DATASTORE_EMULATOR_HOST=localhost:8081
  • dotnet run

Code:

using Google.Api.Gax;
using Google.Cloud.Datastore.V1;
using System;

namespace Issue5365
{
    class Program
    {
        static void Main()
        {
            var db = new DatastoreDbBuilder
            {
                EmulatorDetection = EmulatorDetection.EmulatorOnly,
                ProjectId = "Test",
            }.Build();
            var key = db.CreateKeyFactory("test").CreateKey(1L);
            var result = db.Lookup(key);
            Console.WriteLine("Test complete");
        }
    }
}

Looking further now.

[jskeet]

Okay, I've got it now - we weren't setting the credentials. This is a pleasantly-trivial fix - I hope to be able to get a fix created very soon, and then a release shortly afterwards.

[jskeet]

This has been released as part of Google.Cloud.Datastore.V1 version 3.1.0 - please let me know if you still have any problems with it.