-
Notifications
You must be signed in to change notification settings - Fork 227
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Getting GOAWAY error #27
Comments
This issue was moved to grpc/grpc-node#138 |
Thanks for reporting. I moved the issue to the gRPC repo, since it shouldn't be logging anything unless a user opts-in for that. As far as the error itself (too many pings), is this anything we need to be worried about @callmehiphop? |
Could you describe your deployment environment? The error is likely from a proxy between your client and the GCP service. |
We run on GKE in GCP. We didn't create any proxy between my client and pubsub and I am unaware of any proxies that exist by default. |
Thanks for the info. One more question, how soon did you see the error, after the stream was opened? |
I'm getting the same with version 0.16.1 |
@wenbozhu in my most recent logs I initiate the connection Here is an example:
|
I can also report getting this on version 0.15.0 on node 8.9 on GKE as well. Same behavior of the long initial duration and then the hourly report. |
We believe the issue is caused by a bug introduced in grpc 1.7.0 (fixed in 1.8.0). The error should only happen when there is no outstanding request on a channel. Let us know if this is not the case. |
We were using 1.8.0 (just double checked) :-/ |
Same for us, our deployed grpc version is 1.8.0 per the lock file. |
Sorry, the fix was not shipped with 1.8.0. I'm creating a new release (v1.8.4) that includes this fix. I'll update this thread after it's done. |
Yes please! 😄 |
@y-zeng we've locked our dependency on gRPC to the 1.7.x range, due to a bug introduced in 1.8.0: grpc/grpc-node#130. Do you know if that issue has been resolved? |
I've published the gRPC core v1.8.4 release (https://github.com/grpc/grpc/releases/tag/v1.8.4) which included the PING bug fixes. @murgatroid99 could you please do a new grpc-node release with v1.8.4? |
@murgatroid99 do you know if there is any update on grpc/grpc-node#130? |
I expect to be able to publish the grpc-node 1.8.4 by tomorrow, and it has a change that should improve that situation. Specifically, it makes some network errors (I think including that one) report a status of UNAVAILABLE, which should be retried. |
I can see that the grpc-node 1.8.4 is still not published, and there is also no 1.8.4 tag for grpc on npm. When can we expect it to be published? |
grpc-node 1.8.4 has been published. @mkamioner, @baronfel Could you please update grpc-node and verify the fix? Thanks! |
Unfortunately after updating to grpc-node 1.8.4 the bug still occurs :-( What is the next step @y-zeng ? |
@y-zeng ping... Bueller? Bueller?... ;-) |
No longer relevant for us. We moved to SQS. I highly recommend getting this SDK stuff together. We were on pubsub for over a year and the we decided that the SDK is just not stable enough right now and we need something that is production ready |
Sorry to hear about this. We had been tracking the issue with you and other users on the grpc repo. Would still be useful to have the log in case you have it. Thanks. |
UPDATE: Probably best to mention we're on v 0.13.2 if the lib... Just for anyone else that might find this thread. We have recently had this issue, and I resolved it by having a new We had:
which we then wrapped and used like this:
But then we changed it to this (just the changed functions below):
UPDATE 2 Although this removed the error, we instead had the issue where messages started to get stuck in the queue (see here: #11). So I have reverted this change, updated the libs to 0.17.0, and and pegged GRPC to 1.7.3. We still see the error, but way less frequently, and no stuck messages. |
We have what we think is a solution that we will publish in a gRPC patch release soon. |
We have published gRPC version 1.10.1 with another change that should help with this. |
I solved this problem as follow.
google-gax depends on gRPC, but google-gax v0.16.0 used in pubsub v0.18.0 uses the gRPC library with this bug. We need to upgrade to google-gax |
Any news on this issue or how to solve it? We're experiencing random peaks of these errors during the day for a couple minutes, and then they seem to resolve. We're seeing this using Spanner. |
@webnator is this still the case using updated dependencies? It might help to confirm what versions you are running. |
I have been getting the following error a lot after upgrading to
0.15.0
Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings"
It seems that the
grpc
module is printing it (it is the only place in my code base where this string exists) and pubsub is the only component using grpc... Anyone else having this problem? What is the impact and how can we stop it?Thanks!
Mo
Environment details
Steps to reproduce
The text was updated successfully, but these errors were encountered: