Strange behavior when trying to create an S3 bucket in us-east-1

[ghost]

Version info:
boto3 = 0.0.19 (from pip)
botocore = 1.0.0b1 (from pip)
Python = 2.7.9 (from Fedora 22)

I have no problem creating S3 buckets in us-west-1 or us-west-2, but specifying us-east-1 gives InvalidLocationConstraint

>>> conn = boto3.client("s3")
>>> conn.create_bucket(
    Bucket='testing123-blah-blah-blalalala', 
    CreateBucketConfiguration={'LocationConstraint': "us-east-1"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 200, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 255, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

Also trying with a s3 client connected directly to us-east-1:

>>> conn = boto3.client("s3", region_name="us-east-1")
>>> conn.create_bucket(Bucket='testing123-blah-blah-blalalala', CreateBucketConfiguration={'LocationConstraint': "us-east-1"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 200, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/site-packages/botocore/client.py", line 255, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid

When I do not specify a region, the bucket is created in us-east-1 (verified in the web console):

>>> conn.create_bucket(Bucket='testing123-blah-blah-blalalala')
{u'Location': '/testing123-blah-blah-blalalala', 'ResponseMetadata': {'HTTPStatusCode': 200, 'HostId': 'Qq2CqKPm4PhADUJ8X+ngxxEE3yRrsT3DOS4TefgzUpYBKzQO/62cQy20yPa1zs7l', 'RequestId': '06B36B1D8B1213C8'}}

...but the bucket returns None for LocationConstraint:

>>> conn.get_bucket_location(Bucket='testing123-blah-blah-blalalala')
{'LocationConstraint': None, 'ResponseMetadata': {'HTTPStatusCode': 200, 'HostId': 'nBGHNu30A/m/RymzuoHLiE2uWuzCsz3v1mcov324r2sMYX7ANq1jOIR0XphWiUIAxDwmxTOW8eA=', 'RequestId': '53A539CC4BCA08C4'}}

us-east-1 is listed as a valid region when I enumerate the regions:

>>> conn = boto3.client("ec2", region_name="us-east-1")
>>> [x["RegionName"] for x in conn.describe_regions()["Regions"]]
['eu-central-1', 'sa-east-1', 'ap-northeast-1', 'eu-west-1', 'us-east-1', 'us-west-1', 'us-west-2', 'ap-southeast-2', 'ap-southeast-1']

[jamesls]

This is an artifact of the underlying S3 API and the fact that it doesn't accept us-east-1. To create an S3 bucket in us-east-1 you can just not specify any CreateBucketConfiguration: s3.create_bucket(Bucket='foo').

Comparing our docs to the S3 API Reference docs (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html) it looks like we aren't documenting enum types, which I think would have helped here. We'll look into updating our docs.

[ghost]

I disagree that this shouldn't be fixed within Boto3/core, even if the REST API behaves this way. Having one special region that should be assumed default and raises a confusing Exception when explicitly named violates the Zen of Python

Explicit is better than implicit

Creating a boto3.client("ec2") will raise an exception for not specifying a region_name, so it's not as if the rest of Boto3 assumes us-east-1 as default.

Of course it's trivial for me just to add the below quirk now that I know of this behavior:

conn.create_bucket(
  BucketName='whatevs', 
  CreateBucketConfiguration=None if region == 'us-east-1' else {'LocationConstraint': region})

But why not add the quirk to Boto3's s3.create_bucket() instead of forcing every developer that ever creates S3 buckets to discover the behavior and re-implement the quirk in their own code?

I haven't looked at the Boto3/core code yet, but I would imagine that adding something like this would fix it and prevent numerous other people from running into the same issue:

if CreateBucketConfiguration is not None and \
CreateBucketConfiguration["LocationConstraint"] == "us-east-1":
    CreateBucketConfiguration = None

[jamesls]

Sorry about that. I wasn't suggesting we weren't open to any client side updates, just wanted to explain the current behavior. The other thing I forgot to mention is we don't actually have specific code for this method (or any method in general). We actually dynamically generate methods at runtime based on a shared JSON Description. So this essentially means that we surface the exact REST API in boto3 (and all the semantics of the underlying REST API). We can however add customizations if necessary.

My main hesitation here is that this isn't technically us-east-1, at least in the sense of how other AWS services refer to us-east-1. Our docs refer to this as "US Standard" and say that this can either be "facilities in Northern Virginia or the Pacific Northwest" (http://docs.aws.amazon.com/general/latest/gr/rande.html). This potentially false consistency could be confusing for customers as well.

We'll consider this though. Thanks for the feedback.

[devmage]

It's been a year. Have you figured out what to do about this problem? It still presents a poor developer experience, having to work around an API inconsistency in an interface (boto) that is ostensibly meant to hide API behaviors.

(he said, having spent a day debugging a problem caused by this bug)

[devmage]

On boto3 1.3.1 (and possibly boto but I haven't tested there), it's not as simple as passing CreateBucketConfiguration=None to the call to create_bucket, as that yields a validation error:

  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 262, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 528, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python2.7/dist-packages/botocore/client.py", line 581, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python2.7/dist-packages/botocore/validate.py", line 270, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
ParamValidationError: Parameter validation failed:
Invalid type for parameter CreateBucketConfiguration, value: None, type: <type 'NoneType'>, valid types: <type 'dict'>

Similarly, one cannot pass an empty dict CreateBucketConfiguration={}, as that raises:

ClientError: An error occurred (MalformedXML) when calling the CreateBucket operation: The XML you provided was not well-formed or did not validate against our published schema

It seems that you must omit the field. Similarly, if not using other CreateBucketConfiguration values, it seems that you must omit the entire parameter:

        if region == 'us-east-1':
            s3_conn.create_bucket(
                Bucket=bucket,
            )
        else:
            s3_conn.create_bucket(
                Bucket=bucket,
                CreateBucketConfiguration={'LocationConstraint': region},
            )

[edwardotis]

I sent amazon feedback to fix their documentation on the default region. I encourage others to do so. I can't believe this poor design decision on Amazon's part.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html

[tanzeelrana]

is this issue fixed?

[ghost]

@tanzeelrana:
I don't know if the client API still has this problem, and I don't have access to an AWS account anymore, but I wrote some code a few months ago that uses the higher level Bucket API, and it did not have this problem. I'm going to close the issue.

[ramyala]

why was this closed? This issue still exists.

[dazza-codes]

After about 5 years, this is still not even fixed in the documentation, e.g.

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-creating-buckets.html#create-an-amazon-s3-bucket
• boto3 1.16.44

heavy sigh - why is this issue closed?

The documentation fails to explain that us-east-1 will raise an exception if it is used explicitly. The sample code should include additional details of how to avoid that exception, thanks to snippets from this issue, e.g. it doesn't allow an empty dict nor a default None argument to the CreateBucketConfiguration parameter, it must be excluded entirely.

                if region_name == "us-east-1":
                    s3_client.create_bucket(Bucket=bucket_name)
                else:
                    location = {"LocationConstraint": region_name}
                    s3_client.create_bucket(
                        Bucket=bucket_name,
                        CreateBucketConfiguration=location
                    )

[b0urb0n]

March 2021 - checking in. This is still an issue. 🙄

[damianarata]

Hey guys, I created this other ticket on AWS-CLI. I saw this error on the web console too, looks like an AWS internal error. Comment on the ticket to make AWS fix it or clarify the error.