Deadline Exceeded error!

[warlockdn]

[REQUIRED] Step 2: Describe your environment

• Operating System version: Ubuntu 16
• Firebase SDK version: v6
• Firebase Product: Firestore

[REQUIRED] Step 3: Describe the problem

I am doing a basic Add to Firestore through Node Admin SDK. I just add a object to a document and then wait for the document ID to return after that I send it to the front end to be subscribed for realtime updates. But sometimes I receive the error while adding a document and also updating as to Deadline Exceeded

Steps to reproduce:

The code that I have added to my backend.

let ref = db.collection("orders");

return new Promise((resolve, reject) => {
            ref.add(order).then((order) => {
                logger.info(order);
                resolve(order);
            }).catch((err) => {
                logger.info(err);
                reject(err);
            })
})

There you go I receive errors at catch.

Also checked my account I don't see any high usage since it is in development phase so hardly adding 10-20 docs on a normal day.

Can you please help here. Thanks

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[schmidt-sebastian]

Hi @warlockdn! Sorry to hear that you are having connection issues to Firestore. Note that the initial connection attempt can sometimes be slow, as there is a lot of overhead (Channel establishment, SSL handshakes, authentication). Once you have a connection to Cloud Firestore, the underlying channel will get re-used for further operations. You may be able to work around your issue if you start by issuing a DocumentReference.get() call. We retry those automatically if the connection establishment takes a bit longer than it should.

[warlockdn]

Hi @schmidt-sebastian sorry I don't quite understand. On application restart I already created a connection to firebase. But anyways can you guide me to a resource where I can see some demo or sample code as to why and how this can be used. ?

[schmidt-sebastian]

As said, this is not a solution, but rather a workaround:

const docRef = db.collection("orders").doc();
await docRef.get(); // This will retry on DEADLINE_EXCEEDED
// Now the channel is established.
docRef.create({...}):

[warlockdn]

Do u have a permanent solution to this ? I don't know if with this I can go into production ? If I have to wait to retry on things. Coz the point of being realtime will be of no use if on a load i receive these errors. Or may be i need to follow some other process ?

[schmidt-sebastian]

If you continue to run into these issues, then we should look at your network stack and how you connect to our data centers. If we do need to go down that route, we can let you know some environment variables that will help us debug.

[warlockdn]

Yea you can help me then lemme know what needs to be done so that I don't end up embarrassing my product haha. Thanks

[schmidt-sebastian]

Please set the following environment variables:

GRPC_TRACE=all
GRPC_VERBOSITY=DEBUG

[warlockdn]

so this is the error I finally received

I0916 13:25:14.586417074   10852 completion_queue.cc:951]    RETURN_EVENT[0x23f6660]: QUEUE_TIMEOUT
I0916 13:25:14.591104089   10852 completion_queue.cc:851]    grpc_completion_queue_next(cq=0x23f6660, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=(nil))
I0916 13:25:14.591189393   10852 completion_queue.cc:951]    RETURN_EVENT[0x23f6660]: QUEUE_TIMEOUT
I0916 13:25:14.591880859   10852 completion_queue.cc:851]    grpc_completion_queue_next(cq=0x23f6660, deadline=gpr_timespec { tv_sec: -9223372036854775808, tv_nsec: 0, clock_type: 0 }, reserved=(nil))

[warlockdn]

so do I need to initialise firebase connection on each function call or does it work once on initialising the server ?

[warlockdn]

??

[schmidt-sebastian]

Sorry, I am just catching up on my emails. You only need to initialize the client connection only once, and the network channel should stay open.

[warlockdn]

So what's the issue with the time out then ? My internet connectivity is pretty good

…

On Wed, Sep 19, 2018, 12:23 AM Sebastian Schmidt ***@***.***> wrote: Sorry, I am just catching up on my emails. You only need to initialize the client connection only once, and the network channel should stay open. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#349 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADR8XkgdpookRGwgcuh3JlynA0OnFEDmks5ucUEygaJpZM4WYWqy> .

[schmidt-sebastian]

Just to confirm, you are getting most of these errors on the initial document write and before you are using any other Firestore operations? I am suggesting that your initial operation should be a document read, which we will retry. The "time to first byte" from Firestore is not necessarily related to your network connection.

[CapitanRedBeard]

I get this as well. I have a collection with about 300k documents on it and simply calling .get() on that collection will result in the timeout. Its a firebase problem and not your local system.

I've seen this happen in hosted cloud functions, and the fix is to increase the timeout limit in the function details (max time is 540 seconds, default is 60 seconds). However in the node sdk I can't get figure out how to extends the default past the 60 second limit.

[BrodaNoel]

I'm dealing with the same issue.
I reported it here: steadyequipment/node-firestore-backup#48, due to it's actually this library who is throwing the error.

[farhankhwaja]

I am having the same issue, when try to read my collection which has 100k docs, it gives me DEADLINE ERROR message. How to solve this, even a simple read is so trouble some. Is there a way to export the data from the DB without hitting this Error

[CapitanRedBeard]

My workaround is to find a size that doesn't hit the DEADLINE ERROR, like 10k, then just loop by 10k increments and read for the database until you have read all your docs.

You can do something like this:

  const FOO_READ_LIMIT = 10000
  const foosRef = firestore.collection('foo');

  try {
    let isCalculating = true;
    let lastDoc = null;

    async.whilst(() => {
      return isCalculating;
    },
    async (next) => {
      const fooQuery = foosRef.limit(FOO_READ_LIMIT)
      const pagedFooQuery = 
        lastDoc 
          ? pagedFooQuery.startAfter(lastDoc)
          : pagedFooQuery

      const fooSnapshots = await pagedFooQuery.get();
      const size = fooSnapshots && fooSnapshots.size;

      if (size < FOO_READ_LIMIT) {
        isCalculating = false;
        next()
      }
      
      await doStuff(fooSnapshots);

      lastDoc = fooSnapshots.docs[FOO_READ_LIMIT - 1]
      next()
    },
    (error) => {
      console.log('All data is read');
    });
  } catch(e) {
	// Something wen wrong
  }

[farhankhwaja]

@CapitanRedBeard I ended doing something similar. But, thanks for the reply.

If the Firebase team can add an export option to the Firestore DB like RTDB has, it would help some folks here

[jonrandahl]

@CapitanRedBeard & @farhankhwaja while I don't know if this will help you, still I was running into the same Deadline exceeded errors with only less than 2k of records! I have implemented this into all of my cloud functions, hopefully, this will help you too:

/**
* Set timeout and memory allocation
* In some cases, your functions may have special requirements for a long timeout value or a large * allocation of memory.
*You can set these values either in the Google Cloud Console or in the function source code (Firebase only).
*/
const runtimeOpts = {
  timeoutSeconds: 540, // 9 minutes
  memory: '2GB'
}
/**
 * When a csv file is uploaded in the Storage bucket it is parsed automatically using
 * PapaParse.
 */
exports.covertToJSON = functions
  .runWith(runtimeOpts) // <== apply timeout and memory limit here!
  .storage
  .object()
  .onFinalize( object => {
[...]

Sadly I can't remember where I found that snippet but as I said, every little bit helps!

Jon

[tallalj]

So why is this happening? what is the limit that is being exceeded here? It is on blaze plan so this should not happen. Instead firebase should let us query as much as we want..
Kindly specify the technical issue that needs to be overcome to resolve this.

[schmidt-sebastian]

We currently configure a 60 second deadline for the RunQuery call (which both Query.get() and Query.stream() use). We believe that this should fit most use cases, and would like to refrain from increasing this deadline. Having a shorter deadline limits the amount of work that the backend does for clients that are no longer interested in a Querie's result, which generally results in fewer billed reads.

While somewhat convoluted, you can configure a different deadline for your client:

 const firestore = new Firestore({
      clientConfig: {
        interfaces: {
          'google.firestore.v1.Firestore': {
              methods: {
                RunQuery: {
                  timeout_millis: YOUR_NUMBER_HERE
                }
              }
            }
          }
        }
      });

[npomfret]

I'm getting these DEADLINE_EXCEEDED error when running tests using only very small sets of data (with "firebase-admin": "^7.0.0" / node v8.14.0). I'm also using the blaze plan so I don't think it's billing related. I'm not running inside a cloud function, it's just a regular node app.

It doesn't happen consistently either. I run small tests and it feels like 50% of the time my query times-out and runs into this error. The query itself returns a single record and should run in a fraction of a second.

Is it possible a contention issue? I have a number of parts to my app that may well be issuing queries and updates at the same time. It feels like something is deadlocked.

An example query that gets stuck in this way is:

        const collection = this._firestore.collection(Collections.OMS);
        const existingOpenOrders = await collection
            .where("accountId", "==", this._accountId)
            .where("exchangeId", "==", exchangeId)
            .where("type", "==", DocTypes.OPEN_ORDER)
            .get()

It returns between zero and maybe 4 records only.

I ran why-is-node-running at the point that it all locks up and got this, which doesn't mean much to me unfortunately.

[schmidt-sebastian]

Can you do two things for me:

• Do a blind write before this query to make the connection is established.
• Share you Firestore logs (via setLogFunction())?

Thanks!

[schmidt-sebastian]

I would like to clarify something that I personally wasn't aware of - increasing the request timeouts well past 60 seconds actually will not have the desired effect. 60 seconds is currently the maximum timeout supported by the backed, so the only other solution is to reduce your response size.

[ashking]

@pkwiatkowski0 , I made this error go away.

After a lot of testing and educated guesses on the code developed, here is what made the fix go away -

When the CF is triggered, we were calling a few async calls to perform some business operations. In one of the asynchronous call we had to fetch a few hundred records and that's the place where we were always getting DEADLINE_EXCEEDED errors. After having some console.time() around the documents fetch it was seen that block of code was taking about 100s of seconds to fetch a mere 200 records from the collection. Though the function execution time in total was <100ms. Strange. Strange. Strange...

We made the calls synchronous by having async/await and waited for the entire execution to complete before exiting the function. This greatly reduced the time it took to fetch 200 records from 100s of seconds to <1000ms. Post this we have not seen the DEADLINE_EXCEEDED anymore and code is stable. Hope this helps.

I guess the fire-and-forget nature of the code blocks in the CF puts the entire function on some throttle mode and degrades the performance.

[ghost]

I had the same error where I only was only reading and writing small amounts of records (not even a few hundred) per firestore get()/set(). I discovered for me the issue was I had too many operations happening in parallel as i was performing these operations nested within a for loop. Increasing timeout and memory did not help.

To resolve the error (similar to @ashking resolution) I updated my code to handle the firestore database operation callbacks using async/wait. I effectively removed any parallel firestore operations. This is working really well and no more errors, and things happen quickly which is good as I was concerned I would then hit the timeout limit if things happens one at a time. But seems faster if anything.

Hope this helps anyone stuck on a similar issue.

[yogskumar]

@pkwiatkowski0 , I made this error go away.

After a lot of testing and educated guesses on the code developed, here is what made the fix go away -

When the CF is triggered, we were calling a few async calls to perform some business operations. In one of the asynchronous call we had to fetch a few hundred records and that's the place where we were always getting DEADLINE_EXCEEDED errors. After having some console.time() around the documents fetch it was seen that block of code was taking about 100s of seconds to fetch a mere 200 records from the collection. Though the function execution time in total was <100ms. Strange. Strange. Strange...

We made the calls synchronous by having async/await and waited for the entire execution to complete before exiting the function. This greatly reduced the time it took to fetch 200 records from 100s of seconds to <1000ms. Post this we have not seen the DEADLINE_EXCEEDED anymore and code is stable. Hope this helps.

I guess the fire-and-forget nature of the code blocks in the CF puts the entire function on some throttle mode and degrades the performance.

It's not working. Even I try it multiple times

[ashking]

@Yogeshkumar9291 , Can you isolate and share a sample snippet of your cloud function?

[yogskumar]

.limit(50000);

Hi , here i mentioned the code as per your suggestion. Still facing the same error

let collectionReference = db.collection('userdatabog');
    collectionReference
        .where("date", ">", "20200315")
        .limit(500)
        .get()
        .then(snapshot => {
            dataList = snapshot.docs.map((documentSnapshot) => {
                console.log("Noti Data " + documentSnapshot.data())
                return documentSnapshot.data();
            });
            console.log("DataList  " + JSON.stringify(dataList))
        })

[ashking]

@Yogeshkumar9291 this is the exact issue I had. using .then I could not control the exit of the function before the promise completed succcessfully. I moved it to async/await structure to make sure that my functions don't exit before promises are resolved and that solved my issue. I suggest you give it a try.

let collectionReference = db.collection('userdatabog').where("date", ">", "20200315").limit(500)

const snapshot = await collectionReference.get()
        
const dataList = snapshot.docs.map((documentSnapshot) => {
    console.log("Noti Data " + documentSnapshot.data())
    return documentSnapshot.data();
});
console.log("DataList  " + JSON.stringify(dataList))
        

[weilies]

i have extended my timeout to 540sec, but yet i hit the deadline exceed issue about a min of the program run

here my CF snippet

const runtimeOpts = {
  timeoutSeconds: 540,
//   memory: '1GB'  --> deployment error if defined
}
...
export const GCPSpeech2TranscriptLRR = functions.runWith(runtimeOpts).https.onRequest((req, res) => {

    cors(req, res, async () => {

        const gcsUri = `${storageServer}/${req.body.data.fullPath}`

        const encoding = 'FLAC';
        const languageCode = 'ms-MY'

        const config = {
            encoding: encoding,
            languageCode: languageCode,
            audioChannelCount: 2, // hit 'Invalid audio channel count' if not specify
            enableAutomaticPunctuation: true, // https://cloud.google.com/speech-to-text/docs/automatic-punctuation
        };
        const audio = {
            uri: gcsUri,
        };
        const request = {
            audio: audio,
            config: config,
        };

        // Detects speech in the audio file
        const [operation] = await client.longRunningRecognize(request);
        const [response] = await operation.promise();

        // resolve src/index.ts:175:18 - error TS7006: Parameter 'result' implicitly has an 'any' type.
        const transcription = response.results
            .map((result:any) => result.alternatives[0].transcript)
            .join('\n');
        res.status(200).send({ data: { status: true, message: transcription } })
    })
})

error msg in Chrome Console

Subscriber.js:192 Error: deadline-exceeded
    at new HttpsErrorImpl (index.cjs.js:58)
    at index.cjs.js:373
    at ZoneDelegate.push../node_modules/zone.js/dist/zone.js.ZoneDelegate.invokeTask (zone.js:421)
    at Zone.push../node_modules/zone.js/dist/zone.js.Zone.runTask (zone.js:188)
    at push../node_modules/zone.js/dist/zone.js.ZoneTask.invokeTask (zone.js:503)
    at ZoneTask.invoke (zone.js:492)
    at timer (zone.js:3034)

[rubenheymans]

i have extended my timeout to 540sec, but yet i hit the deadline exceed issue about a min of the program run

here my CF snippet

const runtimeOpts = {
  timeoutSeconds: 540,
//   memory: '1GB'  --> deployment error if defined
}
...
export const GCPSpeech2TranscriptLRR = functions.runWith(runtimeOpts).https.onRequest((req, res) => {

    cors(req, res, async () => {

        const gcsUri = `${storageServer}/${req.body.data.fullPath}`

        const encoding = 'FLAC';
        const languageCode = 'ms-MY'

        const config = {
            encoding: encoding,
            languageCode: languageCode,
            audioChannelCount: 2, // hit 'Invalid audio channel count' if not specify
            enableAutomaticPunctuation: true, // https://cloud.google.com/speech-to-text/docs/automatic-punctuation
        };
        const audio = {
            uri: gcsUri,
        };
        const request = {
            audio: audio,
            config: config,
        };

        // Detects speech in the audio file
        const [operation] = await client.longRunningRecognize(request);
        const [response] = await operation.promise();

        // resolve src/index.ts:175:18 - error TS7006: Parameter 'result' implicitly has an 'any' type.
        const transcription = response.results
            .map((result:any) => result.alternatives[0].transcript)
            .join('\n');
        res.status(200).send({ data: { status: true, message: transcription } })
    })
})

error msg in Chrome Console

Subscriber.js:192 Error: deadline-exceeded
    at new HttpsErrorImpl (index.cjs.js:58)
    at index.cjs.js:373
    at ZoneDelegate.push../node_modules/zone.js/dist/zone.js.ZoneDelegate.invokeTask (zone.js:421)
    at Zone.push../node_modules/zone.js/dist/zone.js.Zone.runTask (zone.js:188)
    at push../node_modules/zone.js/dist/zone.js.ZoneTask.invokeTask (zone.js:503)
    at ZoneTask.invoke (zone.js:492)
    at timer (zone.js:3034)

Well at least I can fix your: deployment error if defined

// tslint:disable-next-line:no-duplicate-imports import { VALID_MEMORY_OPTIONS } from 'firebase-functions';
memory: VALID_MEMORY_OPTIONS[3]

[yogskumar]

@Yogeshkumar9291 this is the exact issue I had. using .then I could not control the exit of the function before the promise completed succcessfully. I moved it to async/await structure to make sure that my functions don't exit before promises are resolved and that solved my issue. I suggest you give it a try.

let collectionReference = db.collection('userdatabog').where("date", ">", "20200315").limit(500)

const snapshot = await collectionReference.get()
        
const dataList = snapshot.docs.map((documentSnapshot) => {
    console.log("Noti Data " + documentSnapshot.data())
    return documentSnapshot.data();
});
console.log("DataList  " + JSON.stringify(dataList))
        

It isn't working..

[weilies]

i have Dough who from Firebase shared that i should call onCall instead of onRequest to use callable function
https://stackoverflow.com/questions/62587762/firebase-cloud-function-set-timeout-540s-but-ended-at-60s

But yet it still doesn't resolve my issue. Strange...

[JoshBot-Debug]

Same here August of 2020 and I'm getting the DEADLINE_EXCEEDED error.

[dobromyslov]

Same error when saving a batch of 100 writes.

[micadasystems]

I have the same error when using firebase-admin in an Electron app + Vue 2.

vue.runtime.esm.js?2b0e:1888 TypeError: Cannot read property 'DEADLINE_EXCEEDED' of undefined
    at eval (gax.js?359a:355)
    at Array.map (<anonymous>)
    at constructRetry (gax.js?359a:354)
    at Object.constructSettings (gax.js?359a:475)
    at eval (util.js?cf9e:21)
    at Object../node_modules/@google-cloud/firestore/build/src/util.js (chunk-vendors.js:406)
    at __webpack_require__ (app.js:849)
    at fn (app.js:151)
    at eval (validate.js?ae2a:19)
    at Object../node_modules/@google-cloud/firestore/build/src/validate.js (chunk-vendors.js:511)
logError @ vue.runtime.esm.js?2b0e:1888
globalHandleError @ vue.runtime.esm.js?2b0e:1879
handleError @ vue.runtime.esm.js?2b0e:1839
eval @ vue.runtime.esm.js?2b0e:1856
Promise.catch (async)
invokeWithErrorHandling @ vue.runtime.esm.js?2b0e:1856
callHook @ vue.runtime.esm.js?2b0e:4219
insert @ vue.runtime.esm.js?2b0e:3139
invokeInsertHook @ vue.runtime.esm.js?2b0e:6346
patch @ vue.runtime.esm.js?2b0e:6565
Vue._update @ vue.runtime.esm.js?2b0e:3945
updateComponent @ vue.runtime.esm.js?2b0e:4066
get @ vue.runtime.esm.js?2b0e:4479
Watcher @ vue.runtime.esm.js?2b0e:4468
mountComponent @ vue.runtime.esm.js?2b0e:4073
Vue.$mount @ vue.runtime.esm.js?2b0e:8415
eval @ main.js?56d7:6
./src/main.js @ app.js:1159
__webpack_require__ @ app.js:849
fn @ app.js:151
1 @ app.js:1172
__webpack_require__ @ app.js:849
checkDeferredModules @ app.js:46
(anonymous) @ app.js:925
(anonymous) @ app.js:928

My Firestore test collection has only 1 document with 1 field.

    const db = firebase.firestore()
    const ref = db.collection('countries');

    ref.get().then((res) => {
      console.log(res)
    })  

The same code running in a standalone nodejs app runs fine, so in my case prob has to do with Electron.

[matetukacs]

You may be able to work around your issue if you start by issuing a DocumentReference.get() call. We retry those automatically if the connection establishment takes a bit longer than it should.

@schmidt-sebastian Could I issue a listCollections() command instead? Does that also get retried automatically? We are running on kubernetes, and I'm thinking I could use listCollections as a sort of ping, which I could use in the readiness check to make sure kubernetes does not send requests to the process before it completes. What do you think?

[schmidt-sebastian]

listCollections() also gets retried automatically (and read-only operation does).

[MorenoMdz]

I had the same error where I only was only reading and writing small amounts of records (not even a few hundred) per firestore get()/set(). I discovered for me the issue was I had too many operations happening in parallel as i was performing these operations nested within a for loop. Increasing timeout and memory did not help.

To resolve the error (similar to @ashking resolution) I updated my code to handle the firestore database operation callbacks using async/wait. I effectively removed any parallel firestore operations. This is working really well and no more errors, and things happen quickly which is good as I was concerned I would then hit the timeout limit if things happens one at a time. But seems faster if anything.

Hope this helps anyone stuck on a similar issue.

Would you have an example of your code? Right now I am randomly having this issue where the DB won't save the subcollection items (here the Assignments), in this case we have a 1 year dates lock so it never gets past 400 documents, yet it is triggering the error, sometimes the DB will retry and create the documents minutes later, sometimes it wont.

  await Promise.all(
    challenge.totalUser.map(async user => await assignToUser(user))
  );

[superzadeh]

Chiming in as we're also having a lot of DEADLINE_EXCEEDED. We are running similar code what was mentioned in this this thread.

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
   at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
   at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24
   at processTicksAndRejections (internal/process/task_queues.js:79:11)
Caused by: Error: 
   at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23)
   at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32)
   at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32)
   at processTicksAndRejections (internal/process/task_queues.js:97:5)
   at DefaultFirestoreService.updateUserTimeline (/workspace/src/services/FirestoreService.ts:667:22)
   at CronController.timelineUpdateForUser (/workspace/src/controllers/CronController.ts:173:7)

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
   at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
   at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24
   at processTicksAndRejections (internal/process/task_queues.js:79:11)
Caused by: Error: 
   at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23)
   at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32)
   at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32)
   at processTicksAndRejections (internal/process/task_queues.js:97:5)
   at DefaultCronService.reminderNotificationForUser (/workspace/src/services/CronService.ts:208:22)
   at CronController.reminderNotificationForUser (/workspace/src/controllers/CronController.ts:142:7)

We are getting this error when fetching a document that isn't particularly large (below 1KB):

// this.firestoreClient is a Firestore() instance
const users = await this.firestoreClient.collection(collections.USERS).doc(userId).get()

It is happening very consistently but in an unpredictable way, and is wreaking total havoc in production. We see it most often in parts of the code where we are writing/reading in parallel/very close timespan from Firestore. Is there any solution available to date, or should we just migrate to another DB that can support our load (which isn't really high, we're never doing more than ~100 read/writes per second)? Seeing that the issue was opened a couple years ago, and appears to still be happening, we're not very confident on being able to get this resolved at all with Firestore. Moving everything to another database is a significant amount of work, that we didn't plan for, so having some sort of honest transparency from the team working on this would help a lot in making sure we make the right decision.

We are also facing very similar errors with Cloud Tasks, who is using the same GRPC library; and was wondering if the issues are not related somehow, and even if you pointed out several time that the GRPC lib is only doing its job, it seems evident that there is something wrong happening on that layer (or on how both clients -- Firestore and Cloud Tasks -- are using it).

PS: increasing the DEADLINE is not an option, the code is already running slow and failing; making it even slower is not viable workaround for production environments. Our users notice and complain about performance all the time already.

[MorenoMdz]

@superzadeh you sure there isn't anything else around that fetch query that might be looping over and increasing the function execution time? The issues I had were solved by splitting a few iteration-heavy functions into smaller ones that happen after some trigger events.

[superzadeh]

Thanks for the suggestion @MorenoMdz, our call site is only making calls to Firestore with async/await "sequentially" (no Promise.all or loops). This is a endpoint called by a CRON job, and working on our users collection one by one: each user has a CRON creating a Cloud Task for them, and our service is processing them one by one. We have 5 instances running, and the queue is capped at 200 max concurrent jobs, which more less means about 40 simultaneous jobs running per service instance (running on AppEngine Flex Environment, 2 CPUs and 6GB RAM). This is by far nothing amazing/pushing the limits in terms of scale, but enough to break Firestore?

Here's the code from our one of our functions calling the getUserData that ends up crashing with DEADLINE_EXCEEDED in the stack traces I shared earlier. Is there anything here we're doing wrong?

  // FirestoreService.ts
  async updateUserTimeline(userId: string): Promise<void> {
  
    // DEADLINE_EXCEEDED is thrown by this call 👇 
    const userData = await this.getUserData(userId)
    const tipOfTheDay = userData.tipOfTheDay
    //...
  }

  async getUserData(userId: string): Promise<firestore.DocumentData> {
    const user = await this.firestoreClient.collection(collections.USERS).doc(userId).get()
    return user.data()
  }

This code itself is called directly form an Express controller handling the route for the CRON job:

  // CronController.ts
  public timelineUpdateForUser = async (
    req: Request,
    res: Response,
    next: NextFunction,
  ) => {
    try {
      const userId = req.params.id
      await this.firestoreService.updateUserTimeline(userId)
      res.status(200).end()
    } catch (error) {
      this.loggingService.error(error)
      next(error)
    }
  }

Results in this:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded
   at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/src/call.ts:81:24)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client.ts:570:32)
   at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/src/client-interceptors.ts:389:48)
   at /workspace/node_modules/@grpc/grpc-js/src/call-stream.ts:249:24
   at processTicksAndRejections (internal/process/task_queues.js:79:11)
Caused by: Error: 
   at Firestore.getAll (/workspace/node_modules/@google-cloud/firestore/build/src/index.js:808:23)
   at DocumentReference.get (/workspace/node_modules/@google-cloud/firestore/build/src/reference.js:208:32)
   at DefaultFirestoreService.getUserData (/workspace/src/services/FirestoreService.ts:605:32)
   at processTicksAndRejections (internal/process/task_queues.js:97:5)
   at DefaultFirestoreService.updateUserTimeline (/workspace/src/services/FirestoreService.ts:667:22)
   at CronController.timelineUpdateForUser (/workspace/src/controllers/CronController.ts:173:7)

[superzadeh]

Any update or workarounds we could try? We are still having this issue happening frequently (every hour at least) in production. I'm also wondering whether the issue should re-openned (or I'm happy to open a new one too).

[ramzauchenna]

The workaround i used was to convert it to a batch commit rather than one at a time limited to 450 records in each batch. Never had the issue again.

[superzadeh]

Thanks @ramzauchenna, are you referring to batched writes or something else?
The issue in our case is happening on reads, which don’t appear to have an option to be batched using the Firestore SDK.

[schmidt-sebastian]

@superzadeh Are you only reading a single document? Are you otherwise able to use Firestore?

[superzadeh]

@superzadeh Are you only reading a single document? Are you otherwise able to use Firestore?

No, we are reading multiple documents (a different one per HTTP request that comes in). The DEADLINE_EXCEEDED errors only happen during peak traffic, otherwise Firestore works as expected.

We were wondering if the soft limit of 1 write per second could be causing this issue? We are only writing to different documents in parallel to these reads, however each document is within the subcollection of a document.

Ex:

- users collection
   +-- foouser_doc_id
          +-- Timeline Subcollection
                 +-- document 1
                 +-- document 2
   +-- baruser_doc_id
          +-- Timeline Subcollection
                 +-- document 1
                 +-- document 2

We write only once in document 1 and document 2, however since they are both in a subcollection of the user document baruser_doc_id, could that count and triggers the soft limit of 1 write/second on the baruser_doc_id document?

[raviganwal]

I solved this by login into, google cloud console and choose your project.
then,

1. from left side menu click on Cloud functions,
2. Click on any cloud function you wish to increase timeout.
3. press edit. and increase timeout time.

in as a front end I am using flutter app so I increased timeout there too,

HttpsCallable callable2 = FirebaseFunctions.instance.httpsCallable(
          NetworkUrl.stripeSubscription,
          options: HttpsCallableOptions(timeout: const Duration(minutes: 1)))<===

[Dave181295]

July 2023 same issue

[milaGGL]

Hi @Dave181295, thank you for reporting this issue. If the problem is showing up consistently and you are able to reproduce it, please create a new issue ticket with a minimal reproducible app, and we can start looking into it.

[jmw11x]

SAME ISSUE TODAY