gRPC "socket closed" service exception still occurring

[meredithslota]

Hey there @saranshdhingra could you share what OS you're running on and what version of the gRPC library you're using where you do not see the "socket closed" issue? I'm inclined to think this is an issue in the network stack below gRPC or perhaps how gRPC is interacting with the native network stack.

We are running on Ubuntu 18.04 with gRPC v1.28.1 and see this issue with some frequency (on the order of 10s of minutes during a long-lived bi-directional streaming connection).

I'm curious if this issue was solved or if it just did not repro in your environment and hence was closed? Thanks!

Originally posted by @jaryder in #2427 (comment)

[jaryder]

To clarify, the issue I described is happening in the gRPC client. Thanks!

[lukasgit]

@meredithslota this issue is still ongoing and it's important to let others follow the original thread here #2427

[meredithslota]

@lukasgit Apologies for the confusion; the linked issue #2427 was marked closed by @saranshdhingra back in April 2022 and I didn't want the last comment to be lost. It's cross-referenced so that previous repro information can still be useful, but if you have current repro steps, please include them here. Thanks!

[bcoe]

@saranshdhingra could you help triage this issue, it is out of SLO.

[saranshdhingra]

@bcoe @meredithslota I will try to look into this issue this week. It's been a bit difficult to replicate it.
I'll try to find the last environment setup I had for this and continue from there.

Thanks a lot for your patience on this @lukasgit

[vishwarajanand]

Reduced priority since it doesn't block shipping releases.

[shivgautam]

@saranshdhingra - Did we find anything on this issue?

[huyhoangle86]

I got both issue, "socket closed " and "go away " in python grpc server side. seems like nothing has been resolved lately

[saranshdhingra]

So, I am not really sure how I can proceed with this issue as I have not been able to replicate it.

Given that it's a response given by the backend APIs, all we can do at the Client Library's front is retry it, which we are doing.

If someone, can find an instance where the UNAVAILABLE code is not leading to a retry, then I would certainly try to dig deeper. I have tested the retry functionality and that seems to be working as intended.

For anyone who wants to control the retry logic can use the grpcRetryFunction option when they make API calls and test if the retry is in fact happening or not.

But, if there is something at the GRPC extension/product API level, I am afraid, the conversation needs to happen in the respective repository.

After all this while, not being able to give a concrete answer to this is not what I had hoped, but this is the best I can say when I haven't been able to replicate it.

I will be closing this for now, but if anyone encounters an instance where the library doesn't retry in the case where it is supposed to, please reopen this, I would certainly look into it.

[wazelin]

A similar issue sporadically occurs when using Bigtable client, @saranshdhingra.

throw(/var/www/html/vendor/google/gax/src/ServerStream.php:84)
Google\ApiCore\ApiException.__construct(/var/www/html/vendor/google/gax/src/ApiException.php:267)
Google\ApiCore\ApiException.create(/var/www/html/vendor/google/gax/src/ApiException.php:148)
Google\ApiCore\ApiException.createFromStdClass(/var/www/html/vendor/google/gax/src/ServerStream.php:84)
Google\ApiCore\ServerStream.readAll(/var/www/html/vendor/google/cloud-bigtable/src/ResumableStream.php:100)
Google\Cloud\Bigtable\ResumableStream.readAll(/var/www/html/vendor/google/cloud-bigtable/src/ChunkFormatter.php:168)
Google\Cloud\Bigtable\ChunkFormatter.readAll
Generator.current(/var/www/html/vendor/google/cloud-bigtable/src/Table.php:331)
Google\Cloud\Bigtable\Table.readRow(/var/www/html/vendor/oat-sa/bundle-bigtable-document-manager/Driver/BigtableDocumentDriver.php:86)

I'm not entirely sure if it should be like that, but I'm guessing the problem could be addressed by adding the same retry mechanism configuration to this piece of configuration.