Auth error:Error: read ECONNRESET

[sahas-]

I'm using 0.42.0 and upon (consistent) high volume pub/sub, I keep getting this error from both app engine and local.

Auth error:Error: read ECONNRESET
E1017 22:14:00.474261000 140735185842176 plugin_credentials.c:67] Getting metadata from plugin failed with error: read ECONNRESET
E1017 22:14:00.474294000 140735185842176 client_auth_filter.c:93] Client side authentication failure: read ECONNRESET
Error: read ECONNRESET at ../node_modules/google-cloud/node_modules**/grpc/src/node/src/client.js:417:17**

In the below snapshot, this happens at the peak

[stephenplusplus]

Is there a way you can provide some code that reproduces this consistently?

[sahas-]

Sorry, its internal code. However, I'm happy to share screens and demo you live, if that'll help.

[stephenplusplus]

No problem. If possible, removing anything sensitive from your code and reducing it to the simplest possible scenario that reproduces the error will help me debug more quickly. But if that's not possible, I'll need to know what types of API calls are being made and how frequently, so I can try to throw some code together that reproduces.

[sahas-]

I'll try to create a sample code to replicate this.
In the mean time,
My app is subscribed to a topic where messages are getting published. Essentially there are 2 different types of messages arrive, they are identified by attributes by the subscriber code to handle them in different code path. Otherwise, both subscriber and publishing applications belong to the same application and leverage the same service account.
In short, what I observed is that this happens on high concurrency of both pub and sub (~10-15)/sec on the same topic by the same application.

[stephenplusplus]

@sahas- I'm going to close, but please let me know if you can stand up some quick sample code that reproduces this. Generally speaking, there are a couple ways to handle similar scenarios that might relate;

1. throttling calls to the API using a library like async, and/or

2. setting a higher maxRetries option on the API calls you make through our library:

require('@google-cloud/pubsub')({ maxRetries: 5 })

The default value of 3 might not be enough to handle errors that could be caused by making a large amount of simultaneous requests.

Hopefully that is useful. If not, re-opening and figuring this specific issue out is only a failing test case away!

[weatherbell]

We are getting the same errors using the finalize cloud storage trigger. It makes no sense because we are providing the service account json key right in the code. The error occurs intermittently.

This is how we are initializing the datastore connection:

// Imports the Google Cloud client library
 const Datastore = require('@google-cloud/datastore');

 // Your Google Cloud Platform project ID
 const projectId = 'our-project-id';

 // Instantiates a client
 const datastore = Datastore({
   projectId: projectId,
   keyFilename: `./${projectId}.json`,
   namespace: 'images'
 });

[JustinBeckwith]

Greetings @weatherbell! If you're running into issues with Datastore, would you mind opening a new issue here?
https://github.com/googleapis/nodejs-datastore