Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resource/aws_s3_bucket: Provider produced inconsistent result after apply (S3 Bucket creation eventual consistency) #11891

Closed
bflad opened this issue Feb 4, 2020 · 3 comments · Fixed by #11894
Closed

resource/aws_s3_bucket: Provider produced inconsistent result after apply (S3 Bucket creation eventual consistency) #11891

bflad opened this issue Feb 4, 2020 · 3 comments · Fixed by #11894
Labels
bug Addresses a defect in current functionality. service/s3 Issues and PRs that pertain to the s3 service.
Milestone
v2.48.0

Comments

@bflad
Copy link
Contributor

bflad commented Feb 4, 2020

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.20
Terraform AWS Provider v2.47.0

Affected Resource(s)

  • aws_s3_bucket

Terraform Configuration Files

resource "aws_s3_bucket" "example" {
  bucket = "example"
}

Debug Output

Can be found in daily TeamCity acceptance testing.

Expected Behavior

S3 Bucket created successfully.

Actual Behavior

Terraform returns the following error:

        Error: Provider produced inconsistent result after apply
        
        When applying changes to aws_s3_bucket.example, provider "aws" produced an
        unexpected new value for was present, but now absent.
        
        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.

This bug report is solely focused on this specific message:

        When applying changes to aws_s3_bucket.example, provider "aws" produced an
        unexpected new value for was present, but now absent.

Other messages in that section for this resource will require separate fixes and should be reported as separate bug reports. Other resources require separate fixes and should be reported as separate bug reports.

Steps to Reproduce

  1. terraform apply

Important Factoids

This error is possible because Read function in the aws_s3_bucket resource uses the HeadBucket API call, which can return a status code 404 on immediate read after creation instead of a parsable error such as NoSuchBucket.
@bflad bflad added bug Addresses a defect in current functionality. service/s3 Issues and PRs that pertain to the s3 service. labels Feb 4, 2020
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Feb 4, 2020
@bflad bflad removed the needs-triage Waiting for first response or review from a maintainer. label Feb 4, 2020
bflad added a commit that referenced this issue Feb 4, 2020
@bflad
resource/aws_s3_bucket: Retry read after creation for 404 status code
c192261 
Reference: #11891

Previously in the acceptance testing (inconsistently across various testing due to eventual consistency):

```
--- FAIL: TestAccAWSAthenaNamedQuery_basic (3.56s)
    testing.go:640: Step 0 error: errors during apply:

        Error: Provider produced inconsistent result after apply

        When applying changes to aws_s3_bucket.test, provider "aws" produced an
        unexpected new value for was present, but now absent.

        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.

--- FAIL: TestAccAWSCodeBuildProject_Environment_Certificate (7.72s)
    testing.go:640: Step 0 error: errors during apply:

        Error: Provider produced inconsistent result after apply

        When applying changes to aws_s3_bucket.test, provider "aws" produced an
        unexpected new value for was present, but now absent.

        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.
```

The HeadBucket/HeadObject S3 APIs work differently than other AWS APIs where they can exclusively return only status code information and not a relevant error code. This update accounts for that discrepency by retrying on 404 status codes on the resource read immediately after bucket creation.

Output from acceptance testing (failures from other eventual consistency issues):

```
--- PASS: TestAccAWSS3Bucket_shouldFailNotFound (18.57s)
--- PASS: TestAccAWSS3Bucket_Cors_Delete (31.55s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (33.16s)
--- PASS: TestAccAWSS3Bucket_forceDestroy (34.90s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (36.16s)
--- PASS: TestAccAWSS3Bucket_basic (36.41s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (36.72s)
--- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (37.42s)
--- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (39.28s)
--- PASS: TestAccAWSS3Bucket_objectLock (62.84s)
--- FAIL: TestAccAWSS3Bucket_Cors_Update (70.43s)
    testing.go:640: Step 2 error: After applying this step, the plan was not empty:
--- PASS: TestAccAWSS3Bucket_namePrefix (34.06s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutStorageClass (74.89s)
--- PASS: TestAccAWSS3Bucket_Logging (75.40s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (77.09s)
--- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (59.98s)
--- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (80.69s)
--- FAIL: TestAccAWSS3Bucket_RequestPayer (50.88s)
    testing.go:640: Step 2 error: Check failed: Check 2/3 error: aws_s3_bucket.bucket: Attribute 'request_payer' expected "Requester", got "BucketOwner"
--- FAIL: TestAccAWSS3Bucket_acceleration (21.28s)
    testing.go:640: Step 0 error: Check failed: Check 2/2 error: aws_s3_bucket.bucket: Attribute 'acceleration_status' expected "Enabled", got ""
--- FAIL: TestAccAWSS3Bucket_LifecycleBasic (86.20s)
    testing.go:640: Step 3 error: After applying this step and refreshing, the plan was not empty:
--- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (60.54s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (62.41s)
--- PASS: TestAccAWSS3Bucket_UpdateAcl (61.70s)
--- PASS: TestAccAWSS3Bucket_region (30.65s)
--- PASS: TestAccAWSS3Bucket_Versioning (105.29s)
--- PASS: TestAccAWSS3Bucket_generatedName (35.22s)
--- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (34.39s)
--- PASS: TestAccAWSS3Bucket_Policy (76.26s)
--- PASS: TestAccAWSS3Bucket_WebsiteRedirect (83.43s)
--- PASS: TestAccAWSS3Bucket_Website_Simple (83.85s)
--- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (149.59s)
--- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (112.51s)
--- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (234.41s)
--- PASS: TestAccAWSS3Bucket_Replication (242.83s)
--- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (182.05s)
```

Re-run of failed tests:

```
--- PASS: TestAccAWSS3Bucket_RequestPayer (59.87s)
--- PASS: TestAccAWSS3Bucket_Cors_Update (61.90s)
--- PASS: TestAccAWSS3Bucket_acceleration (62.81s)
--- PASS: TestAccAWSS3Bucket_LifecycleBasic (89.19s)
```
@bflad bflad mentioned this issue Feb 4, 2020
Merged
bflad added a commit that referenced this issue Feb 5, 2020
@bflad
resource/aws_s3_bucket: Retry read after creation for 404 status code (…
ba169a4 
…#11894)

Reference: #11891

Previously in the acceptance testing (inconsistently across various testing due to eventual consistency):

```
--- FAIL: TestAccAWSAthenaNamedQuery_basic (3.56s)
    testing.go:640: Step 0 error: errors during apply:

        Error: Provider produced inconsistent result after apply

        When applying changes to aws_s3_bucket.test, provider "aws" produced an
        unexpected new value for was present, but now absent.

        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.

--- FAIL: TestAccAWSCodeBuildProject_Environment_Certificate (7.72s)
    testing.go:640: Step 0 error: errors during apply:

        Error: Provider produced inconsistent result after apply

        When applying changes to aws_s3_bucket.test, provider "aws" produced an
        unexpected new value for was present, but now absent.

        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.
```

The HeadBucket/HeadObject S3 APIs work differently than other AWS APIs where they can exclusively return only status code information and not a relevant error code. This update accounts for that discrepency by retrying on 404 status codes on the resource read immediately after bucket creation.

Output from acceptance testing (failures from other eventual consistency issues):

```
--- PASS: TestAccAWSS3Bucket_shouldFailNotFound (18.57s)
--- PASS: TestAccAWSS3Bucket_Cors_Delete (31.55s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithEmptyPrefixes (33.16s)
--- PASS: TestAccAWSS3Bucket_forceDestroy (34.90s)
--- PASS: TestAccAWSS3Bucket_forceDestroyWithObjectLockEnabled (36.16s)
--- PASS: TestAccAWSS3Bucket_basic (36.41s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenAES256IsUsed (36.72s)
--- PASS: TestAccAWSS3Bucket_Cors_EmptyOrigin (37.42s)
--- PASS: TestAccAWSS3Bucket_ReplicationExpectVersioningValidationError (39.28s)
--- PASS: TestAccAWSS3Bucket_objectLock (62.84s)
--- FAIL: TestAccAWSS3Bucket_Cors_Update (70.43s)
    testing.go:640: Step 2 error: After applying this step, the plan was not empty:
--- PASS: TestAccAWSS3Bucket_namePrefix (34.06s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutStorageClass (74.89s)
--- PASS: TestAccAWSS3Bucket_Logging (75.40s)
--- PASS: TestAccAWSS3Bucket_ReplicationWithoutPrefix (77.09s)
--- PASS: TestAccAWSS3Bucket_disableDefaultEncryption_whenDefaultEncryptionIsEnabled (59.98s)
--- PASS: TestAccAWSS3Bucket_LifecycleExpireMarkerOnly (80.69s)
--- FAIL: TestAccAWSS3Bucket_RequestPayer (50.88s)
    testing.go:640: Step 2 error: Check failed: Check 2/3 error: aws_s3_bucket.bucket: Attribute 'request_payer' expected "Requester", got "BucketOwner"
--- FAIL: TestAccAWSS3Bucket_acceleration (21.28s)
    testing.go:640: Step 0 error: Check failed: Check 2/2 error: aws_s3_bucket.bucket: Attribute 'acceleration_status' expected "Enabled", got ""
--- FAIL: TestAccAWSS3Bucket_LifecycleBasic (86.20s)
    testing.go:640: Step 3 error: After applying this step and refreshing, the plan was not empty:
--- PASS: TestAccAWSS3Bucket_WebsiteRoutingRules (60.54s)
--- PASS: TestAccAWSS3Bucket_enableDefaultEncryption_whenTypical (62.41s)
--- PASS: TestAccAWSS3Bucket_UpdateAcl (61.70s)
--- PASS: TestAccAWSS3Bucket_region (30.65s)
--- PASS: TestAccAWSS3Bucket_Versioning (105.29s)
--- PASS: TestAccAWSS3Bucket_generatedName (35.22s)
--- PASS: TestAccAWSS3Bucket_Bucket_EmptyString (34.39s)
--- PASS: TestAccAWSS3Bucket_Policy (76.26s)
--- PASS: TestAccAWSS3Bucket_WebsiteRedirect (83.43s)
--- PASS: TestAccAWSS3Bucket_Website_Simple (83.85s)
--- PASS: TestAccAWSS3Bucket_ReplicationConfiguration_Rule_Destination_AccessControlTranslation (149.59s)
--- PASS: TestAccAWSS3Bucket_tagsWithNoSystemTags (112.51s)
--- PASS: TestAccAWSS3Bucket_ReplicationSchemaV2 (234.41s)
--- PASS: TestAccAWSS3Bucket_Replication (242.83s)
--- PASS: TestAccAWSS3Bucket_tagsWithSystemTags (182.05s)
```

Re-run of failed tests:

```
--- PASS: TestAccAWSS3Bucket_RequestPayer (59.87s)
--- PASS: TestAccAWSS3Bucket_Cors_Update (61.90s)
--- PASS: TestAccAWSS3Bucket_acceleration (62.81s)
--- PASS: TestAccAWSS3Bucket_LifecycleBasic (89.19s)
```
@bflad bflad added this to the v2.48.0 milestone Feb 5, 2020
@bflad
Copy link
Contributor Author

bflad commented Feb 5, 2020

The fix for this this specific instance of Provider produced inconsistent result after apply for the aws_s3_bucket resource has been merged and will release with version 2.48.0 of the Terraform AWS Provider, tomorrow. Other messages of this error with the aws_s3_bucket resource or other similar errors with other resources will require separate bug reports for triage.

@ghost
Copy link

ghost commented Feb 7, 2020

This has been released in version 2.48.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Mar 27, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Mar 27, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Addresses a defect in current functionality. service/s3 Issues and PRs that pertain to the s3 service.
Projects
None yet
Milestone
v2.48.0
1 participant
@bflad