service/s3/s3manager: Reduce downloader memory allocations & GC runs

[vvelikodny]

Version of AWS SDK for Go?

v0.19.0

Version of Go (go version)?

go version go1.13.4 darwin/amd64

What issue did you see?

in case the user of the library doesn't create a buffer with a particular size and pass it to the S3 downloader, it downloads file part-by-part from 1st byte up to file length. Thus WriteAtBuffer increase buffer size based on the current max downloaded part, but we know file size at the start after this code:

https://github.com/aws/aws-sdk-go-v2/blob/master/service/s3/s3manager/download.go#L314

d.getChunk()

so I propose download file backward => from the end of the file, so we will allocate the right memory at the start as the last part of the file will be downloaded at the beginning of the process and then buffer never be reallocated. It gives good optimization and GC doesn't work during the whole process, at the beginning only.

Before pull request:

After pull request:

trace_2_bad_gc.txt
trace_2_good_gc.txt

Steps to reproduce

My sample code:

package main

import (
	"context"
	"fmt"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/aws/external"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/s3manager"
)

func HandleDemoRecord(bucket, region, item string) error {
	fmt.Printf("Bucket = %s, Key = %s\n", bucket, item)

	config, err := external.LoadDefaultAWSConfig()
	if err != nil {
		return err
	}
	config.Region = region

	headObject, err := getHeadObject(bucket, item, config)
	if err != nil {
		return err
	}

	fmt.Printf("obj: %v\nbytes:%+v\n", headObject, *headObject.ContentLength)

	downloader := s3manager.NewDownloader(config, func(d *s3manager.Downloader) {
		d.Concurrency = 5
	})

	var buf []byte
	// allocate before
	//buf := make([]byte, int(*headObject.ContentLength))

	fmt.Println("Starting download of file")
	buffer := aws.NewWriteAtBuffer(buf)
	numBytesDownloaded, err := downloader.Download(buffer, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(item),
	})

	if err != nil {
		return err
	}

	fmt.Println("Downloaded ", numBytesDownloaded, " bytes")

	return nil
}

func getHeadObject(bucket string, key string, config aws.Config) (*s3.HeadObjectResponse, error) {
	client := s3.New(config)

	input := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}

	request := client.HeadObjectRequest(input)

	headObjectResponse, err := request.Send(context.TODO())
	if err != nil {
		return nil, err
	}

	fmt.Printf("Downloaded: %v\n", request)

	return headObjectResponse, nil
}

func main() {
	err := HandleDemoRecord("vvelikodny-s3-upload", "ca-central-1", "mysql-workbench-community-8.0.18-macos-x86_64.dmg")
	if err != nil {
		panic(err)
	}
}

here I investigated also memory issues with huge allocations on start, but think it should be in the next issue/pull request.

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

[vvelikodny]

issue still presented in new SDK #483 (comment)