storage: stream error: stream ID xxx; INTERNAL_ERROR with increased load on calling Attr to check object existence

[ivymilkyway]

Our application is streaming tons of data from GCS files.
With increased workload, we’ve seen increased stream error like
Get "https://storage.googleapis.com/storage/v1/b/bagomatic-prod/o/bag%2Fstandard_v1%2F5G21A6P05L4100129%3A1565945553%3A1565945573.bag?alt=json&prettyPrint=false&projection=full": stream error: stream ID 3787; INTERNAL_ERROR

from calling the Attrs method to check the existence of an object.

func (o *ObjectHandle) Attrs(ctx context.Context) (attrs *ObjectAttrs, err error) {
https://github.com/googleapis/google-cloud-go/blob/master/storage/storage.go#L803

I've seen previous discussion regarding the stream error: stream IDxxx; INTERNAL_ERROR. seems adding retries on this error solves issue for reads. However for Attrs call, it doesn't seem to retry on those errors.

We have added simple exponential backoff retries o this method, seems to reduce the error rate by some extent, however still seeing them with increased workload.

Our application first check the existence of the object, and then start range reader to stream from the file. Interestingly we didn't see lots of errors from the range reader, most of the errors from the Attr call, which is low volume comparing to the range reader since it's called only once per request.

[andrewsg]

Thanks for your report. I'll check with our team on whether retries can support this error on actions other than reads, and look into why you might be experiencing such a high error rate on this call.

[I-Cat]

Thanks for your report. I'll check with our team on whether retries can support this error on actions other than reads, >> and look into why you might be experiencing such a high error rate on this call.

@andrewsg

The code comes back as expecting prog.go:1:1:

{"error":{"code":401,"message":"Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.","errors":[{"message":"Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object.","domain":"global","reason":"required","locationType":"header","location":"Authorization"}]}}

Not Found
== No file

[frankyn]

@tritone and @BrennaEpp is this issue addressed by the retry work y'all are doing? (I believe so but want to double check).

[tritone]

We have not added this particular error to the default ShouldRetry method. However, we now provide a method to specify a custom error predicate for retries: https://pkg.go.dev/cloud.google.com/go/storage@main#WithErrorFunc (note that this feature is merged but not yet released).

You can use ObjectHandle.Retryer to customize the retry behavior for ObjectHandle.Attrs, including using a custom ErrorFunc which will mark this error as retryable.

I'm going to close this issue for now but feel free to re-open if you have further concerns.