Getting GOAWAY when listening to streams after a while

[jmacedoit]

Hi everyone. We have a server listening over many long lived streams using firebase-admin. We recently bumped our versions of @google-cloud/firestore and firebase-admin to:

    "@google-cloud/firestore": "^2.1.0",
    "firebase-admin": "^8.0.0"

We managed to get rid of the DEADLINE_EXCEEDED we were previously getting but unfortunately we are now getting a new error:

Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY

This one happens on different conditions though... While we could reproduce DEADLINE_EXCEEDED with some load tests this seems to happen after a few hours always happens after 30 minutes or so (perhaps when the streams being listened to are idle for a while?) after the server process starts. Any idea of what might be the cause, and also, which combination of package versions might get us free from both DEADLINE_EXCEEDED and GOAWAY errors? Because when this happens, restarting the process is the only remedy.

Environment details

• OS: Happening locally on macOS 10.14.4 and also on a docker container based on the node:10-alpine image running on a Ubuntu 18.04.2 LTS machine.
• Node.js version: v10.15.0 on macOS, v10.xon the Ubuntu machine.
• npm version: 6.9.0
• @google-cloud/firestore version: ^2.1.0

Steps to reproduce

1. Start the server (which currently listens to ~80 streams)
2. Leave it running and wait
3. After a while (maybe after streams have no activity for some time) the error comes up and the streams "stop working", i.e, no new events are detected starting from there

Thanks!

[jmacedoit]

Detailed error log:

Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY
    at Http2CallStream.call.on (/Users/xxx/Projects/yyy/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:198:15)
    at Http2CallStream.endCall (/Users/xxx/Projects/yyy/node_modules/@grpc/grpc-js/build/src/call-stream.js:74:18)
    at /Users/xxx/Projects/yyy/node_modules/@grpc/grpc-js/build/src/call-stream.js:355:18
    at processTicksAndRejections (internal/process/task_queues.js:86:5)
  code: 'ERR_HTTP2_GOAWAY_SESSION',
  details:
   'Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY',
  metadata: Metadata { internalRepr: Map {} } }

[jakeleventhal]

I am having the same problem

[jmacedoit]

For everyone still having this problem, I think grpc-js 0.4.3 fixes it. Adding the explicit dependency to "@grpc/grpc-js": "^0.4.3" will probably do it (remember to remove node modules just to be sure it is installed). (Or just wait until nodejs-firestore gets updated to use that version)

[callmehiphop]

This should be resolved, you might need to delete any lock files to pull in the latest gax/grpc-js packages. Let us know if the issue persists and we'll be happy to take another look! 😎

[jmacedoit]

@callmehiphop Is the most recent version of nodejs-firestore already pulling "@grpc/grpc-js": "^0.4.3" ? In case it is not, shouldn't the issue remain opened until it does?

[callmehiphop]

@jmacedoit I can confirm that the latest version does pull in 0.4.3, all patch updates get pulled in automatically.

[LOG-TAG]

@jmacedoit still getting this error on cloud function randomly ! I'm using "firebase": "^6.2.3" ! Updating 6.5.0 will solve this ? Changing firebase version will effect the cloud function / firestore ? (most recent version of nodejs-firestore )

[LOG-TAG]

@jmacedoit still getting this error on cloud function randomly ! I'm using "firebase": "^6.2.3" ! Updating 6.5.0 will solve this ? Changing firebase version will effect the cloud function / firestore ? (most recent version of nodejs-firestore )

{ Error: Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY
at Http2CallStream.call.on (/srv/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
at emitOne (events.js:121:20)
at Http2CallStream.emit (events.js:211:7)
at process.nextTick (/srv/node_modules/@grpc/grpc-js/build/src/call-stream.js:71:22)
at _combinedTickCallback (internal/process/next_tick.js:132:7)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
code: 'ERR_HTTP2_GOAWAY_SESSION',
details: 'Getting metadata from plugin failed with error: New streams cannot be created after receiving a GOAWAY',
metadata: Metadata { options: undefined, internalRepr: Map {} } }

[jmacedoit]

@LOG-TAG You should always try to use the most recent version of packages when you find a bug to check whether it has been solved or not. This being said, ^6.2.3 should be pulling 6.5.0. Just to be safe, delete your node modules and update your dependency in package.json to ^6.5.0 (your package-lock.json may be constraining your version to an old one, this will force it to change).

If this doesn't work, post the output of npm ls here so we can see which firebase/firestore related dependencies you might have that are outdated.