doc(storage): update Writer.ChunkSize docs (#5850)

Clarify that each Writer allocates a buffer internally, and
state more strongly that callers should use ChunkSize to avoid
memory issues.

Co-authored-by: Noah Dietz <[email protected]>

storage/storage.go

@@ -1128,6 +1128,9 @@ func (o *ObjectHandle) ReadCompressed(compressed bool) *ObjectHandle {
 // attribute is specified, the content type will be automatically sniffed
 // using net/http.DetectContentType.
 //
+// Note that each Writer allocates an internal buffer of size Writer.ChunkSize.
+// See the ChunkSize docs for more information.
+//
 // It is the caller's responsibility to call Close when writing is done. To
 // stop writing without saving the data, cancel the context.
 func (o *ObjectHandle) NewWriter(ctx context.Context) *Writer {

storage/writer.go

@@ -61,21 +61,26 @@ type Writer struct {
 	// ChunkSize controls the maximum number of bytes of the object that the
 	// Writer will attempt to send to the server in a single request. Objects
 	// smaller than the size will be sent in a single request, while larger
-	// objects will be split over multiple requests. The size will be rounded up
-	// to the nearest multiple of 256K.
+	// objects will be split over multiple requests. The value will be rounded up
+	// to the nearest multiple of 256K. The default ChunkSize is 16MiB.
 	//
-	// ChunkSize will default to a reasonable value. If you perform many
-	// concurrent writes of small objects (under ~8MB), you may wish set ChunkSize
-	// to a value that matches your objects' sizes to avoid consuming large
-	// amounts of memory. See
+	// Each Writer will internally allocate a buffer of size ChunkSize. This is
+	// used to buffer input data and allow for the input to be sent again if a
+	// request must be retried.
+	//
+	// If you upload small objects (< 16MiB), you should set ChunkSize
+	// to a value slightly larger than the objects' sizes to avoid memory bloat.
+	// This is especially important if you are uploading many small objects
+	// concurrently. See
 	// https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload#size
 	// for more information about performance trade-offs related to ChunkSize.
 	//
 	// If ChunkSize is set to zero, chunking will be disabled and the object will
 	// be uploaded in a single request without the use of a buffer. This will
 	// further reduce memory used during uploads, but will also prevent the writer
-	// from retrying in case of a transient error from the server, since a buffer
-	// is required in order to retry the failed request.
+	// from retrying in case of a transient error from the server or resuming an
+	// upload that fails midway through, since the buffer is required in order to
+	// retry the failed request.
 	//
 	// ChunkSize must be set before the first Write call.
 	ChunkSize int