Error: 13 INTERNAL: Received RST_STREAM with code 2 #679

SUM-Soft · 2020-06-10T19:27:14Z

Hi!

Getting the following error periodically (1-2 times a day)

A 2020-06-10T18:35:53.594123Z (node:14) UnhandledPromiseRejectionWarning: Error: 13 INTERNAL: Received RST_STREAM with code 2 
A 2020-06-10T18:35:53.594173Z     at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26) 
A 2020-06-10T18:35:53.594187Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52) 
A 2020-06-10T18:35:53.594267Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141) 
A 2020-06-10T18:35:53.594293Z     at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181) 
A 2020-06-10T18:35:53.594306Z     at Http2CallStream.outputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74) 
A 2020-06-10T18:35:53.594318Z     at Http2CallStream.maybeOutputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22) 
A 2020-06-10T18:35:53.594330Z     at Http2CallStream.endCall (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18) 
A 2020-06-10T18:35:53.594342Z     at ClientHttp2Stream.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22) 
A 2020-06-10T18:35:53.594356Z     at ClientHttp2Stream.emit (events.js:315:20) 
A 2020-06-10T18:35:53.594367Z     at emitCloseNT (internal/streams/destroy.js:69:8) 
A 2020-06-10T18:35:53.594394Z (node:14) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1) 
A 2020-06-10T18:35:53.594405Z (node:14) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code. 
A 2020-06-10T18:35:54.059739Z Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:341:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
    at Http2CallStream.outputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
    at Http2CallStream.maybeOutputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
    at Http2CallStream.endCall (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
    at ClientHttp2Stream.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:403:22)
    at ClientHttp2Stream.emit (events.js:315:20)
    at emitCloseNT (internal/streams/destroy.js:69:8) { 
A 2020-06-10T18:35:54.059769Z   code: 13, 
A 2020-06-10T18:35:54.059794Z   details: 'Received RST_STREAM with code 2', 
A 2020-06-10T18:35:54.059803Z   metadata: Metadata { internalRepr: Map {}, options: {} }, 
A 2020-06-10T18:35:54.059813Z   note: 'Exception occurred in retry method that was not classified as transient' 
A 2020-06-10T18:35:54.059821Z }

Environment details

OS: Cloud Run
Node.js version: node:lts-slim
npm version: 6.14.5
@google-cloud/datastore version: 6.0.0

Thanks!

The text was updated successfully, but these errors were encountered:

BenWhitehead · 2020-06-15T22:12:39Z

@schmidt-sebastian Does this look like the same error we're seeing in Firestore?

BenWhitehead · 2020-06-15T22:13:02Z

@SUM-Soft Can you confirm if you're using Cloud Datastore or Cloud Firestore in Datastore Mode?

schmidt-sebastian · 2020-06-15T22:23:33Z

It would shock me if this wasn't the exact same as googleapis/nodejs-firestore#1023

SUM-Soft · 2020-06-16T05:02:08Z

@SUM-Soft Can you confirm if you're using Cloud Datastore or Cloud Firestore in Datastore Mode?

Using Cloud Firestore in Datastore mode

mdobson · 2020-07-09T21:57:47Z

We see this issue in using Cloud Datastore within Cloud Run as well.

radvansky-tomas · 2020-07-10T09:11:31Z

I am seeing this constantly now...every time I am trying to write object to newly created collection (from auth trigger in firebase cloud functions).

damienromito · 2020-07-12T16:33:47Z

I got this on OnDelete triggers

Yesterday

Today

rdev-software · 2020-07-12T20:01:35Z

Show how you configure your admin app, if you dont have same issue as above

SUM-Soft · 2020-07-21T17:22:47Z

The error still occurs with @google-cloud/datastore version: 6.1.0
Any progress on this issue?

zdenda-online · 2020-10-15T22:53:06Z

We see the same. It usually happens for us when we do quite a heavy load of database writes in Node.js GCP function (nodejs10) during the migration of data. I tried to "relax" the calls putting random 1-20ms "sleep" in between writes but it did not help.

And yes, we are using Datastore in Firestore mode. Datastore client libraries version 6.2.0.

Any progress on this?

biendinh · 2020-10-26T09:09:35Z

Same here on firebase function.
Db: Cloud Firestore

jhcordeiro · 2020-10-30T17:16:30Z

Firebase functions with Node 10:

Error message:
2020-10-30 13:09:54.340 EDT 6trbr5cj3lf7 Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:327:49) at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181) at process.nextTick (/workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:124:78) at process._tickCallback (internal/process/next_tick.js:61:11)

Please, is there any progress with this issue? all my function calls started failing after 9:10am PST today

eli-stewart · 2020-12-10T06:42:02Z

I am seeing this as well. I am running a Typescript React App with a node.js server. The server is throwing this error fairly regularly. Something like once an hour. I also get this similar error: @firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server but about 20% as frequently.

I had a crash on my site on Monday where I was not able to read or write the database and I am trying to figure out if its related.

These are the 3 types of errors I am receiving of this type:
@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 2

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 14 Message: 14 UNAVAILABLE: Stream refused by server

@firebase/firestore: Firestore (7.19.1): Connection GRPC stream error. Code: 13 Message: 13 INTERNAL: Received RST_STREAM with code 0

zdenda-online · 2021-02-04T07:59:44Z

Any update?

By the way, I don't agree this should have priority p2 = moderate importance. This breaks the base use of the service.

schmidt-sebastian · 2021-02-04T20:29:45Z

We are working on this with multiple engineers and trying to come with mitigation strategies. Unfortunately, this is more difficult than anticipated.

zdenda-online · 2021-02-04T20:46:18Z

@schmidt-sebastian ok thanks for the info. Good luck with that :-) Was unsure as I see Open bug with no assignee.

binajmen · 2021-02-24T15:46:53Z

Thank you @schmidt-sebastian for the follow up.

Please keep us in the loop when things are moving. I'm having the same issue with Cloud Run + Firestore.

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: Protocol error
    at Object.callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:327:49)
    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /app/node_modules/@grpc/grpc-js/build/src/call-stream.js:130:78
    at processTicksAndRejections (internal/process/task_queues.js:75:11)

kickbk · 2021-03-01T15:04:42Z

Same issue here. Waiting for this to be resolved.

tohagan · 2021-03-09T02:08:47Z

Saw one of these in a firebase async onRequest call (Node 10 runtime) on 02/15/2021

Here's the stack ...

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
    at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:342:141)
    at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:305:181)
    at process.nextTick (/workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:124:78)
    at process._tickCallback (internal/process/next_tick.js:61:11)

dronpopdev · 2021-03-12T12:35:04Z

I got this on onCreate triggers. Cloud Firestore + Cloud Functions.

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET
at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
at /workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
at processTicksAndRejections (internal/process/task_queues.js:79:11)

More details. I have created 15 objects. Each object was created approximately once every 20 seconds. It happened on March 12th.

bcoe · 2021-03-12T23:25:25Z

@dronpopdev could you ensure the latest version of @grpc/grpc-js is in your package-lock.json (https://www.npmjs.com/package/@grpc/grpc-js) redeploy your application, and share the latest version of the error message -- if you're still running into issues.

We've been working to add improved error messages to the library.

LKNSI · 2021-03-30T22:41:06Z

Seeing these on non-google connections with gRPC. But while using a google library that has it's own gRPC connection within the context of the call made on our applications. Near identical message even when not using a google library.

~~Bidirectional Streaming:~~

~~i.e. Microservice -> (grpc connection) -> Server -> (Google Library that has a streaming gRPC connection (in our case speech to text) -> Response to microservice~~

~~"@grpc/grpc-js": "1.2.12"~~

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
at Object.callErrorFromStatus (/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/node_modules/@grpc/grpc-js/build/src/client.js:387:49)
at Object.onReceiveStatus (/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
at /node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
at processTicksAndRejections (internal/process/task_queues.js:79:11)

Okay so this will definitely apply to a lot of people - I found out that it was actually HAProxy timing out the connection, when using something gRPC related, either from google or locally produced - even if live messages are being clearly sent with GRPC_TRACE=all , and are being received properly. I guess in some cases this might either be a frontend issue or something else.

vishalvijay · 2021-05-15T11:11:26Z

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET
at Object.callErrorFromStatus (/workspace/node_modules/@grpc/grpc-js/build/src/call.js:31)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client.js:176)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336)
at Object.onReceiveStatus (/workspace/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299)
at (/workspace/node_modules/@grpc/grpc-js/build/src/call-stream.js:145)
at processTicksAndRejections (internal/process/task_queues.js:79)

Facing same issue with Firebase pubsub schedule function

functions
  .region('asia-south1')
  .pubsub.schedule('every 1 minutes')
  .onRun(async () => {
    await theWork();
  });

tfalvo · 2021-05-17T18:46:41Z

Facing exactly the same issue with Cloud Tasks:

Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: read ECONNRESET
    at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:176:52)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:336:141)
    at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
    at /usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
    at processTicksAndRejections (internal/process/task_queues.js:79:11)

This happens quite rarely, but when happening, it's not in a context of massive task creation, just a single task created :

await this.client.createTask({
      parent: queueName,
      task: {...}
});

dhruv-sehgal · 2021-05-21T11:08:23Z

@tfalvo , were you able to solve above problem. Facing similar problem with cloud task.

Chm-vinicius · 2021-06-09T19:08:26Z

Same issue here, when try to migrate db from realtimedatabe to firestore, on localhost with small sample of db the migration function works very well, but when i use entire legacy db i got this errors and emulators got shutdown:

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) at Object.callErrorFromStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\call.js:31:26) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client.js:176:52) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:336:141) at Object.onReceiveStatus (workspace\functions\node_modules\@grpc\grpc-js\build\src\client-interceptors.js:299:181) at workspace\functions\node_modules\@grpc\grpc-js\build\src\call-stream.js:145:78 at processTicksAndRejections (internal/process/task_queues.js:75:11) Caused by: Error at WriteBatch.commit (workspace\functions\node_modules\@google-cloud\firestore\build\src\write-batch.js:413:23) at DocumentReference.set (workspace\functions\node_modules\@google-cloud\firestore\build\src\reference.js:350:14) at userIterator (workspace\functions\dbMigrator.js:270:28) at workspace\functions\dbMigrator.js:276:5 at local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:604:16 at runFunction (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:583:15) at runHTTPS (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:603:11) at handler (local\AppData\Roaming\npm\node_modules\firebase-tools\lib\emulator\functionsEmulatorRuntime.js:532:23) at Layer.handle [as handle_request] (local\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\layer.js:95:5) at next (local\AppData\Roaming\npm\node_modules\firebase-tools\node_modules\express\lib\router\route.js:137:13)

odira1 · 2021-06-20T03:40:44Z

Had this error happen after a couple of deletes on datastore, resolved it by ensuring that the process exited properly with process.exit(0). hope this helps someone, cheers :)

crwilcox · 2021-07-22T21:35:23Z

In order for us to gather feedback better, please add any further details to the Firestore issue at googleapis/nodejs-firestore#1023. The root cause is likely the same and it seems better to gather feedback in a single place. Feel free to add there if you see an issue.

meredithslota · 2022-05-10T18:52:39Z

Closing this out as discussion and investigation have continued in the issue linked above, and we are optimistic that a fix has been found and implemented, see here: googleapis/nodejs-firestore#1023 (comment) If this issue is not fixed for you, please add your repro steps to that issue.

product-auto-label bot added the api: datastore Issues related to the googleapis/nodejs-datastore API. label Jun 10, 2020

yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Jun 11, 2020

radvansky-tomas mentioned this issue Jul 10, 2020

RST_STREAM error keeps showing up googleapis/nodejs-firestore#1023

Open

chrismaree mentioned this issue Oct 26, 2020

feat(serverless-hub) Added re-try logic to gcp calls UMAprotocol/protocol#2111

Merged

yoshi-automation added the 🚨 This issue needs some love. label Dec 7, 2020

crwilcox mentioned this issue Apr 19, 2021

Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) #813

Closed

moolen mentioned this issue Jun 14, 2021

RST_STREAM exception every 24 hours for gcpSecretsManager externalsecrets external-secrets/kubernetes-external-secrets#768

Closed

crwilcox added external This issue is blocked on a bug with the actual product. and removed 🚨 This issue needs some love. labels Jul 22, 2021

googleapis locked as resolved and limited conversation to collaborators Jul 22, 2021

meredithslota closed this as completed May 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: 13 INTERNAL: Received RST_STREAM with code 2 #679

Error: 13 INTERNAL: Received RST_STREAM with code 2 #679

SUM-Soft commented Jun 10, 2020

BenWhitehead commented Jun 15, 2020

BenWhitehead commented Jun 15, 2020

schmidt-sebastian commented Jun 15, 2020

SUM-Soft commented Jun 16, 2020

mdobson commented Jul 9, 2020

radvansky-tomas commented Jul 10, 2020

damienromito commented Jul 12, 2020

rdev-software commented Jul 12, 2020

SUM-Soft commented Jul 21, 2020

zdenda-online commented Oct 15, 2020

biendinh commented Oct 26, 2020

jhcordeiro commented Oct 30, 2020

eli-stewart commented Dec 10, 2020 •

edited

Loading

zdenda-online commented Feb 4, 2021

schmidt-sebastian commented Feb 4, 2021

zdenda-online commented Feb 4, 2021

binajmen commented Feb 24, 2021 •

edited

Loading

kickbk commented Mar 1, 2021

tohagan commented Mar 9, 2021

dronpopdev commented Mar 12, 2021

bcoe commented Mar 12, 2021

LKNSI commented Mar 30, 2021 •

edited

Loading

vishalvijay commented May 15, 2021

tfalvo commented May 17, 2021

dhruv-sehgal commented May 21, 2021 •

edited

Loading

Chm-vinicius commented Jun 9, 2021

odira1 commented Jun 20, 2021

crwilcox commented Jul 22, 2021

meredithslota commented May 10, 2022

Error: 13 INTERNAL: Received RST_STREAM with code 2 #679

Error: 13 INTERNAL: Received RST_STREAM with code 2 #679

Comments

SUM-Soft commented Jun 10, 2020

Environment details

BenWhitehead commented Jun 15, 2020

BenWhitehead commented Jun 15, 2020

schmidt-sebastian commented Jun 15, 2020

SUM-Soft commented Jun 16, 2020

mdobson commented Jul 9, 2020

radvansky-tomas commented Jul 10, 2020

damienromito commented Jul 12, 2020

rdev-software commented Jul 12, 2020

SUM-Soft commented Jul 21, 2020

zdenda-online commented Oct 15, 2020

biendinh commented Oct 26, 2020

jhcordeiro commented Oct 30, 2020

eli-stewart commented Dec 10, 2020 • edited Loading

zdenda-online commented Feb 4, 2021

schmidt-sebastian commented Feb 4, 2021

zdenda-online commented Feb 4, 2021

binajmen commented Feb 24, 2021 • edited Loading

kickbk commented Mar 1, 2021

tohagan commented Mar 9, 2021

dronpopdev commented Mar 12, 2021

bcoe commented Mar 12, 2021

LKNSI commented Mar 30, 2021 • edited Loading

vishalvijay commented May 15, 2021

tfalvo commented May 17, 2021

dhruv-sehgal commented May 21, 2021 • edited Loading

Chm-vinicius commented Jun 9, 2021

odira1 commented Jun 20, 2021

crwilcox commented Jul 22, 2021

meredithslota commented May 10, 2022

eli-stewart commented Dec 10, 2020 •

edited

Loading

binajmen commented Feb 24, 2021 •

edited

Loading

LKNSI commented Mar 30, 2021 •

edited

Loading

dhruv-sehgal commented May 21, 2021 •

edited

Loading