Bigtable: stream terminated by RST_STREAM with error code: INTERNAL_ERROR is not retried by client

[aravindsrinivasan]

Package: cloud.google.com/go/bigtable v1.13.0

When our bigtable instance has high throughput we see the following error on occasion:

rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

This to me looks like a transient error that should be retried by the client. I see the Spanner client retrying this exact error:

google-cloud-go/spanner/retry.go

Lines 59 to 70 in 5a2ed6b

	// Retry returns the retry delay returned by Cloud Spanner if that is present.
	// Otherwise it returns the retry delay calculated by the generic gax Retryer.
	func (r *spannerRetryer) Retry(err error) (time.Duration, bool) {
	if status.Code(err) == codes.Internal &&
	!strings.Contains(err.Error(), "stream terminated by RST_STREAM") &&
	// See b/25451313.
	!strings.Contains(err.Error(), "HTTP/2 error code: INTERNAL_ERROR") &&
	// See b/27794742.
	!strings.Contains(err.Error(), "Connection closed with unknown cause") &&
	!strings.Contains(err.Error(), "Received unexpected EOS on DATA frame from server") {
	return 0, false
	}

Could also do the same in the Bigtable client? Or is this error actually not retryable in the context of Bigtable?

[aravindsrinivasan]

Any update here?

[nainya]

Any updates here?

[bhshkh]

Not yet picked up. Will pick up soon