s3: error for out of region bucket - 301 response missing Location header

[ncw]

If I create a bucket in one region, then try to access it from a different region s3 sends a redirect, however the library doesn't recognize it. Here is what I get from the debug

Here I created a bucket in the default region and am now trying to list it from us-west-1

$ rclone ls s3west:rclone
2015/08/18 20:18:22 DEBUG: Request Signiture:
---[ CANONICAL STRING  ]-----------------------------
GET
/mybucket
delimiter=&max-keys=1024&prefix=
host:s3-us-west-1.amazonaws.com
x-amz-date:20150818T191822Z

host;x-amz-date
e3b0c44298fc1c149xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20150818T191822Z
20150818/us-west-1/s3/aws4_request
fbeae995314xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----------------------------------------------------
2015/08/18 20:18:22 DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-us-west-1.amazonaws.com/mybucket?delimiter=&max-keys=1024&prefix= HTTP/1.1
Host: s3-us-west-1.amazonaws.com
User-Agent: rclone/v1.18
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXX/20150818/us-west-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=76c168xxxxxxxxxxxxxxxxxxxxxxxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Amz-Date: 20150818T191822Z
Accept-Encoding: gzip


-----------------------------------------------------
2015/08/18 20:18:23 DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 0 status code 0

-----------------------------------------------------
2015/08/18 20:18:23 DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-us-west-1.amazonaws.com/mybucket?delimiter=&max-keys=1024&prefix= HTTP/1.1
Host: s3-us-west-1.amazonaws.com
User-Agent: rclone/v1.18
Authorization: AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXXXXXXXXXXXXX/20150818/us-west-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=76c168xxxxxxxxxxxxxxxxxxxxxxxxxx
X-Amz-Content-Sha256: e3b0c44298fc1c149xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Amz-Date: 20150818T191822Z
Accept-Encoding: gzip


-----------------------------------------------------
2015/08/18 20:18:23 DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 0 status code 0

This then repeats to make 10 times before returning an error

-----------------------------------------------------
2015/08/18 20:19:13 S3 bucket rclone: Couldn't read bucket "mybucket": RequestError: send request failed
caused by: Get : 301 response missing Location header

What I think is happening is that s3 sends a redirect but the library doesn't deal with it correctly, and retries instead.

[jasdel]

Hi @ncw thanks for submitting this bug. The SDK should not be retrying request when they error with this request. The 301 error returned is due to the request being made to the wrong region. The Go SDK does require that the request are made to the correct region.

The redirect provided by S3 is not intended to actually be followed in this case. That is why the Location header is not set. If your application has access to the bucket you could do S3.GetBucketLocaltion() to get the bucket's region. Or if not not performing a HEAD request on the bucket to receive the Bucket's location in the x-amz-bucket-region header.

curl -I "https://bucketname.s3.amazonaws.com"

Or in Go, This functionality isn't currently in the SDK but can be implemented simply with the below.

resp, err := http.Head("https://bucketname.s3.amazonaws.com")
if err != nil {
    fmt.Println("ERROR", err)
    return
}
fmt.Println("Region:", resp.Header.Get("X-Amz-Bucket-Region"))

[jasdel]

I pushed 9411a1e which fixes the multiple retries when the request should of not retried at all. Thanks for reporting this issue, if you have an more issues or feedback please let us know.

[ncw]

@jasdel thanks for the explanation and fix. I'll update my docs to indicate the above.

PS http://rclone.org is now using the new SDK - the managed multipart upload was a real timesaver thanks!

[jasdel]

Great let us know if you run into any issues with the SDK, new features/tools we can add, or things we can do better.

[vpartington]

Thanks @jasdel for pointing me in the right direction. I spent a few days wondering why the HeadBucket call in AWS SDK for Go v2 does not work for this. 😞

Instead of using http.Head you can also use the manager.GetBucketRegion function in AWS SDK for Go v2.

[johnpyp]

Though it may be far too late to revisit this, maybe this error should actually be retried by the AWS SDKs for consistency?

The AWS CLI does seem to retry this error, and the information is actually provided in the headers (or body) to complete the redirect - no need for an additional call. For example...

aws s3 ls s3://cwaopendata/Astronomy --debug # Public aws dataset, called from a us-east-1 account

Results in this response (formatted):

# The Request/response log
# https://cwaopendata.s3.us-east-1.amazonaws.com:443
# "GET /?list-type=2&prefix=Astronomy&delimiter=%2F&encoding-type=url HTTP/1.1" 301 None

# Response Body
<?xml version="1.0" encoding="UTF-8"?>\n
<Error>
	<Code>PermanentRedirect</Code>
	<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
	<Endpoint>cwaopendata.s3-ap-northeast-1.amazonaws.com</Endpoint>
	<Bucket>cwaopendata</Bucket>
	<RequestId>YGK9CR634REV7SHC</RequestId>
	<HostId>x6rtN9iikJDKUQ6buaZWh29vVfhL8Su8ppMVhLBq8n4cCrRXD2LvvHw6bLxRDgPiiXB9e4ia490=</HostId>
</Error>

# Response headers
{
    "x-amz-bucket-region": "ap-northeast-1",
    "x-amz-request-id": "YGK9CR634REV7SHC",
    "x-amz-id-2": "x6rtN9iikJDKUQ6buaZWh29vVfhL8Su8ppMVhLBq8n4cCrRXD2LvvHw6bLxRDgPiiXB9e4ia490=",
    "Content-Type": "application/xml",
    "Transfer-Encoding": "chunked",
    "Date": "Mon, 12 Aug 2024 21:58:58 GMT",
    "Server": "AmazonS3",
}

botocore itself handles this with a "redirect from error handler": S3RegionRedirectorv2, which is enabled by default. and not easily configurable.

---

As an aside, even if it is too late to make this the default behavior for the go library for backwards compatibility reasons, it seems like it would be a nice configurable affordance for end-user libraries. I was pointed here through issues in rclone and datafusion-cli, which both seem like they'd benefit from supporting the implicit redirect.