Datastore emulator enforcing indexes when docs say it doesn't

[selvaradov]

[REQUIRED] Environment info

firebase-tools: I'm using the gcloud cli emulator, not the full firebase emulator suite. when I run gcloud version the output is

Google Cloud SDK 483.0.0
beta 2024.06.28
bq 2.1.6
bundled-python3-unix 3.11.8
cloud-datastore-emulator 2.3.1
cloud-firestore-emulator 1.19.7
core 2024.06.28
gcloud-crc32c 1.0.0
gsutil 5.30

Platform: Ubuntu (WSL)

Test case

router.get('/reviews', async (req, res) => {
  try {
    const query = datastore
      .createQuery('Review')
      .filter('submitter', '=', 'Alice')
      .order('submittedAt', { descending: true });

    const [reviews] = await datastore.runQuery(query);

    res.json(reviews);
  } catch (error) {
    console.error('Error fetching user reviews:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

Steps to reproduce

1. Start connection to datastore in Node js app with review route as above
2. Try making request to /review endpoint

Expected behavior

• The docs say that

The emulator does not track composite indexes and instead executes any valid query. Make sure to test your app against a real Datastore mode instance to determine which indexes you require.

• (See discussion in Add Firestore indexes to local emulator #2027)
• If it does track composite indexes, then there should be instructions in the docs on how to set these up with the emulator

Actual behavior

Error fetching user reviews: Error: 9 FAILED_PRECONDITION: no matching index found. recommended index is:
- kind: Review
  properties:
  - name: submitter
  - name: submittedAt
    direction: desc

    at callErrorFromStatus (.../node_modules/@grpc/grpc-js/src/call.ts:82:17)
    at Object.onReceiveStatus (.../node_modules/@grpc/grpc-js/src/client.ts:360:55)
    at Object.onReceiveStatus (.../node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34)
    at Object.onReceiveStatus (.../node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
    at <anonymous> (.../node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (.../node_modules/@grpc/grpc-js/src/client.ts:325:42)
    at ServiceClientImpl.<anonymous> (.../node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
    at <anonymous> (.../node_modules/@google-cloud/datastore/src/v1/datastore_client.ts:305:25)
    at .../node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
    at repeat (.../node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
    at .../node_modules/google-gax/build/src/normalCalls/retries.js:119:13
    at OngoingCall.call (.../node_modules/google-gax/build/src/call.js:67:27)
    at NormalApiCaller.call (.../node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
    at .../node_modules/google-gax/build/src/createApiCall.js:112:30
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 9,
  details: 'no matching index found. recommended index is:\n' +
    '- kind: Review\n' +
    '  properties:\n' +
    '  - name: submitter\n' +
    '  - name: submittedAt\n' +
    '    direction: desc\n',
  metadata: Metadata {
    internalRepr: Map(1) { 'x-debug-tracking-id' => [Array] },
    options: {}
  },
  note: 'Exception occurred in retry method that was not classified as transient'
}

[aalej]

Hey @selvaradov, thanks for filing a detailed report and for sharing your observations. I’m trying to reproduce the behavior you mentioned, but so far no errors are being raised and I’m able to get a list of reviews when running the query.

I created this repo to try and reproduce the issue, let me know in case I missed anything or made a mistake in the setup. Alternatively, could you share an MCVE so that we can replicate the issue on our end?

Also, just to verify, were you able to run export DATASTORE_EMULATOR_HOST=127.0.0.1:7079? It’s possible that you may be connecting to production instead of the emulators.

[selvaradov]

Hi,

Thanks for looking into it! I have gone back and tried to reproduce but wasn't able to -- I think you're right that there was a problem with the env variables that meant it was trying to connect to production: looking back in the logs for the datastore being set up it had as options

    servicePath: 'datastore.googleapis.com',
    port: 443,
    projectId: undefined

I'll close this now; sorry to take up your time