-
Notifications
You must be signed in to change notification settings - Fork 653
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error: No connection established #1064
Comments
Thank you for the report! Just to confirm, does that mean that the script should have an hour of idle time with no network requests to experience that, or just an hour of runtime? |
Similar issue on @google-cloud/[email protected] |
We are seeing the exact same issues as @harscoet with |
Same issue when using |
We are looking into this and will either fix it ASAP, or release an update to |
Just to make sure I understand, are you making some requests that succeed, then going idle for a while, then making more requests and seeing these errors? |
Also, do you folks run your code on a physical machine, or in any kind of cloud e.g. cloud functions? |
@murgatroid99 we are seeing that our services are running fine for 10ish minutes and then we start dropping connections. At this time we see deadline exceeded errors. Once this happens we are unable to have successful requests with both @alexander-fenster we are seeing this using with services running within GKE containers |
We downgraded |
If my fix is right, this problem was actually in #1015 but it was overshadowed by other problems. |
I have published version 0.6.5 of |
@apatora @harscoet @reichjustin @weipublic Could you folks please check if the fix in v0.6.5 works for you all, and let us know? Thank you! |
@alexander-fenster I tried v0.6.5 and I still have the issue. Lots of |
Same here, with DataStore and PubSub, 0.6.5 doesn't fix the issue |
Same issues for us when using |
Can someone seeing this issue with grpc-js 0.6.5 run with the environment variables |
I would also like to understand how exactly this problem is manifesting. Are you seeing one or a few errors and then later calls succeed? Or do you see this error and then all subsequent calls fail with the same error? Or something else? |
would be reasonable to suggest using pinned @grpc/grpc-js versions ? just to avoid a large impact when those sensitive dependencies run into issues. |
We have seen those errors starting to occur around 1 hour after a fresh pod creation. After those started, all subsequent calls were starting to fail. Pintpointing lib to previous working version (0.5.2) resolved the issue. (using google Firestore) |
I tried the trick suggested in #1027 and run npx npm-force-resolutions && npm install
however, I'm getting the following error:
any ideas ? My app also uses other google cloud dependencies "dependencies": { |
I have published 0.6.6 with another possible fix for this problem. If anyone is willing to try it out and report the result that would be helpful. |
@rvillane We normally don't pin dependencies since it makes it much harder to release patches. I understand the frustration caused by these issues, we'll definitely go ahead and pin the version to some v0.5.x if the problem is not fixed soon enough. [updated: it said "0.4.x" originally, 0.5.x should be fine though] |
As far as I know there are no outstanding issues with 0.5.x |
Again, it would be helpful to see the confirmation of which version(s) of the library you have, as well as the stack trace of the errors you are getting. In addition to that, as mentioned previously in this issue it would be helpful to see the output of running your code with the environment variables |
@murgatroid99 I will try. It becomes difficult since it is completely random if and when it happens. |
While I try and get a stack trace. Here is the error from the functions log
|
|
|
|
At the very beginning of your script, before imports, you can write |
I am facing this issue and I am using Firebase admin |
As mentioned above I am using Firebase and am getting the connection failure error. I added the proc.env settings and here is the log 2019-11-20T16:48:52.351Z | subchannel | 172.217.163.170:443 READY -> IDLE |
@kaushaldeck Can you also confirm what version of |
Hope this helps
|
Yes. That trace log shows a bug that was fixed in |
I had this problem for few months. My issue is, the port 8080 is closed on my Mac. I opened the port and everything worked perfectly. |
I'm experiencing sustained CPU spikes to 100% after an hour since a pod is started. I've created a debug pod which doesn't handle any traffic and it still manifests the same way. After profiling the CPU I've found that calls to Is it intended for this code: https://github.com/grpc/grpc-node/pull/1251/files#diff-a96b991cf71efa63f54e0313d45cf39cR234-R257 to create a loop like the one I described? I'm currently using these versions of google libraries:
The libraries are using |
@alexandrugheorghe That looks like the same bug as reported in googleapis/nodejs-pubsub#890, and that user says that the problem was fixed in |
@murgatroid99 I'll update now and check it again in an hour. |
@murgatroid99 Sorry for the late reply. Pods have been stable, version |
I'm hoping this issue hasn't resurfaced, but I did encounter the same error in a project that had been idle for some time. I recycled the pod so I should have an idea of the timeout being the cause in the future, but it impacted all our instances in a test environment. The error: err:
message: '14 UNAVAILABLE: No connection established'
name: Error
stack: >
Error: 14 UNAVAILABLE: No connection established
at Object.callErrorFromStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call.ts:81:24)
at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client.ts:334:36)
at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
at Object.onReceiveStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
at Http2CallStream.outputStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:230:22)
at Http2CallStream.maybeOutputStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:280:14)
at Http2CallStream.endCall (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:263:12)
at Http2CallStream.cancelWithStatus (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:597:10)
at ChannelImplementation.tryPick (/app/packages/animus/node_modules/@grpc/grpc-js/src/channel.ts:387:22)
at ChannelImplementation._startCallStream (/app/packages/animus/node_modules/@grpc/grpc-js/src/channel.ts:433:10)
at Http2CallStream.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/call-stream.ts:573:18)
at BaseUnaryInterceptingCall.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:374:15)
at BaseUnaryInterceptingCall.start (/app/packages/animus/node_modules/@grpc/grpc-js/src/client-interceptors.ts:437:11)
at ServiceClientImpl.makeUnaryRequest (/app/packages/animus/node_modules/@grpc/grpc-js/src/client.ts:315:10)
at ServiceClientImpl.<anonymous> (/app/packages/animus/node_modules/@grpc/grpc-js/src/make-client.ts:174:15)
at args (/app/packages/animus/node_modules/dialogflow/src/v2beta1/sessions_client.js:175:35)
at /app/packages/animus/node_modules/google-gax/src/normalCalls/timeout.ts:54:13
at OngoingCallPromise.call (/app/packages/animus/node_modules/google-gax/src/call.ts:82:23)
at NormalApiCaller.call (/app/packages/animus/node_modules/google-gax/src/normalCalls/normalApiCaller.ts:46:15)
at funcPromise.then.then (/app/packages/animus/node_modules/google-gax/src/createApiCall.ts:103:26)
at Object.dynatraceOnServiceExecutionIndicator [as doInvoke] (/opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:1803:20)
at Object.a.safeInvoke (/opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:1854:29)
at /opt/dynatrace/oneagent/agent/bin/1.195.54.20200529-113801/any/nodejs/nodejsagent.js:7079:25
at process._tickCallback (internal/process/next_tick.js:68:7)"
code: 14 We install dependencies from the {
// deps...
"@grpc/grpc-js": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/@grpc/grpc-js/-/grpc-js-1.0.4.tgz",
"integrity": "sha512-Qawt6HUrEmljQMPWnLnIXpcjelmtIAydi3M9awiG02WWJ1CmIvFEx4IOC1EsWUWUlabOGksRbpfvoIeZKFTNXw==",
"requires": {
"google-auth-library": "^6.0.0",
"semver": "^6.2.0"
},
"dependencies": { /** deps... */ }
}
} Will have another issue opened if this has resurfaced. I'll be back with more if it's an issue we come across again. |
Hello everyone, I've been trying to create a secret in the GCP from running the GCP APIs from AWS lambda. I tried setting the environment variable at the run time like
The secrets manager's version is 3.2.3
|
I am facing the same issue again:
package-lock.json
|
@sidcool1234 i'm too. |
@sidcool1234 , @ducvu91 did you guys find any work-around or still persisting with this issue ? Please let me know if you guys found any work-around. |
@AkhilA4 not yet, Sometimes it happens and it's random, there's no specific case so I can't find where the cause is. |
OK @ducvu91. Since last week, I'm encountering this issue every time I run the createSecret GCP API. Will try to create a new issue to get some attention for the issue. Not sure what needs to be done from dev end if this gets encountered in production environment. |
I was getting this issue because I setup the Firestore emulator and wasn't turning it on. If you recently enabled the emulators, you have to start them with firebase emulators:start . Then you'll start getting the information from your local database. |
Seeing the same issue with
UPDATE:
After reading above I guess getting |
Hey Error: 1 CANCELED: Call cancelled Is the problem related to a certain version and/or incompatible versions? |
Problem description
Since
google-gax
forced thegrpc-js
version to 0.6.4 (googleapis/gax-nodejs@628db9e), I am having the following issue :Reproduction steps
The bug only happens after one hour. We are using
grpc-js
inside@google-cloud/bigtable
and our bigtable instance is autoscaling.The text was updated successfully, but these errors were encountered: