Skip to content

Commit

Permalink
Switch reading from S3 to io.Copy from io.ReadFull (#4225)
Browse files Browse the repository at this point in the history
* Switch reading from S3 to io.Copy from io.ReadFull

If the Content-Length header wasn't being sent back, the current
behavior could panic. It's unclear when it will not be sent; it appears
to be CORS dependent. But this works around it by not trying to
preallocate a buffer of a specific size and instead just read until EOF.

In addition I noticed that Close wasn't being called.
https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#GetObjectOutput
specifies that Body is an io.ReadCloser so I added a call to Close.

Fixes #4222

* Add some extra efficiency
  • Loading branch information
jefferai authored Mar 30, 2018
1 parent 0c433c2 commit 82b493a
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions physical/s3/s3.go
Original file line number Diff line number Diff line change
Expand Up @@ -188,16 +188,20 @@ func (s *S3Backend) Get(ctx context.Context, key string) (*physical.Entry, error
if resp == nil {
return nil, fmt.Errorf("got nil response from S3 but no error")
}
defer resp.Body.Close()

data := make([]byte, *resp.ContentLength)
_, err = io.ReadFull(resp.Body, data)
data := bytes.NewBuffer(nil)
if resp.ContentLength != nil {
data = bytes.NewBuffer(make([]byte, 0, *resp.ContentLength))
}
_, err = io.Copy(data, resp.Body)
if err != nil {
return nil, err
}

ent := &physical.Entry{
Key: key,
Value: data,
Value: data.Bytes(),
}

return ent, nil
Expand Down

0 comments on commit 82b493a

Please sign in to comment.