-
Notifications
You must be signed in to change notification settings - Fork 438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Spanner] GRPC randomly throws ServiceException("Socket closed") #1208
Comments
I followed the stacktrace and found the following comment for code: 14 in Google\Rpc\Code.php /**
* The service is currently unavailable. This is most likely a
* transient condition, which can be corrected by retrying with
* a backoff.
* See the guidelines above for deciding between `FAILED_PRECONDITION`,
* `ABORTED`, and `UNAVAILABLE`.
* HTTP Mapping: 503 Service Unavailable
*
* Generated from protobuf enum <code>UNAVAILABLE = 14;</code>
*/
const UNAVAILABLE = 14; Should we be retrying this? |
When using Spanner, should we turn these parameters in GKE?
|
Can you determine whether the issue arises from various service calls (like |
Unfortunately, we can only confirm this error on |
No problem! I'll investigate and get back to you once I know more. |
So our team was looking though the code and realized that the error occurs on But seems like the try/catch right below that is actually handling the case. |
We are looking in to whether wrapping the /cc @snehashah16 (our Spanner contact) for some assistance around your question regarding GKE parameters and the differences seen in the stress environment. |
@dwsupplee any progress investigating this one? |
@danoscarmike We've been having trouble replicating the issue locally, but have a change set moving the valid calls which we could resume on into the try/catch block. We are attempting to validate it doesn't have any negative side effects and will be pushing it up shortly. |
Would it be possible to check out this branch to see if it helps mitigate the issue before we release? |
@dwsupplee Sure, we'll try it out today and let you know how it goes. |
@dwsupplee I tried out the latest branch you provided and ran it for about 2 hours. Hopefully this gets merged soon? |
Awesome, thanks for testing this out @taka-oyama. We will get this merged today and a release following shortly! |
@dwsupplee
google-cloud-php/Spanner/src/Result.php Line 181 in 81a923e
Why not include this part in a try block? |
@castaneai, from what I can remember our original guidance from the Spanner team was to only retry when a resume token is available. With that said, it does seem reasonable to retry on the first valid call since we haven't received any rows yet. I'm certainly open to adding this in. |
This error still occurs from time to time. |
I'm seeing this exact issue still on v0.91 |
@dwsupplee @jdpedrie this issue still randomly persists. We're part of the Google Cloud Startup program and launching this year... a fix would be greatly appreciated so we can move to production. Log output:
|
@lukasgit would you mind opening up a fresh issue for us? Sometimes comments on closed issues can get lost. |
Thanks! |
We occasionally see this error in production.
This error occurs randomly across various places where we use spanner.
How can I resolve this?
The text was updated successfully, but these errors were encountered: