[WIP] Add additional checksum algorithms in aws_s3_object resource and data source

[mediba-h-sugawara]

Description

Add additional checksum algorithms (CRC32, CRC32C, SHA1, SHA256) to S3 object resource and data source.

Relations

Related #23901
Related #25647

References

• New – Additional Checksum Algorithms for Amazon S3
• Using supported checksum algorithms
• PutObject API Reference
• HeadObject API Reference

Output from Acceptance Testing

$ make testacc TESTS=TestAccS3Object_checksum\|TestAccS3Object_source\|TestAccS3ObjectDataSource_checksum\|TestAccS3ObjectDataSource_allParams PKG=s3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/s3/... -v -count 1 -parallel 20 -run='TestAccS3Object_checksum|TestAccS3Object_source|TestAccS3ObjectDataSource_checksum|TestAccS3ObjectDataSource_allParams'  -timeout 180m
=== RUN   TestAccS3ObjectDataSource_allParams
=== PAUSE TestAccS3ObjectDataSource_allParams
=== RUN   TestAccS3ObjectDataSource_checksumCRC32
=== PAUSE TestAccS3ObjectDataSource_checksumCRC32
=== RUN   TestAccS3ObjectDataSource_checksumCRC32C
=== PAUSE TestAccS3ObjectDataSource_checksumCRC32C
=== RUN   TestAccS3ObjectDataSource_checksumSHA1
=== PAUSE TestAccS3ObjectDataSource_checksumSHA1
=== RUN   TestAccS3ObjectDataSource_checksumSHA256
=== PAUSE TestAccS3ObjectDataSource_checksumSHA256
=== RUN   TestAccS3Object_source
=== PAUSE TestAccS3Object_source
=== RUN   TestAccS3Object_sourceHashTrigger
=== PAUSE TestAccS3Object_sourceHashTrigger
=== RUN   TestAccS3Object_checksumCRC32
=== PAUSE TestAccS3Object_checksumCRC32
=== RUN   TestAccS3Object_checksumCRC32FromContent
=== PAUSE TestAccS3Object_checksumCRC32FromContent
=== RUN   TestAccS3Object_checksumCRC32C
=== PAUSE TestAccS3Object_checksumCRC32C
=== RUN   TestAccS3Object_checksumCRC32CFromContent
=== PAUSE TestAccS3Object_checksumCRC32CFromContent
=== RUN   TestAccS3Object_checksumSHA1
=== PAUSE TestAccS3Object_checksumSHA1
=== RUN   TestAccS3Object_checksumSHA1FromContent
=== PAUSE TestAccS3Object_checksumSHA1FromContent
=== RUN   TestAccS3Object_checksumSHA256
=== PAUSE TestAccS3Object_checksumSHA256
=== RUN   TestAccS3Object_checksumSHA256FromContent
=== PAUSE TestAccS3Object_checksumSHA256FromContent
=== CONT  TestAccS3ObjectDataSource_allParams
=== CONT  TestAccS3Object_checksumCRC32FromContent
=== CONT  TestAccS3Object_checksumSHA1FromContent
=== CONT  TestAccS3Object_checksumSHA256FromContent
=== CONT  TestAccS3Object_checksumSHA1
=== CONT  TestAccS3Object_checksumSHA256
=== CONT  TestAccS3Object_checksumCRC32C
=== CONT  TestAccS3ObjectDataSource_checksumSHA256
=== CONT  TestAccS3Object_checksumCRC32CFromContent
=== CONT  TestAccS3ObjectDataSource_checksumCRC32C
=== CONT  TestAccS3Object_checksumCRC32
=== CONT  TestAccS3ObjectDataSource_checksumSHA1
=== CONT  TestAccS3Object_sourceHashTrigger
=== CONT  TestAccS3Object_source
=== CONT  TestAccS3ObjectDataSource_checksumCRC32
--- PASS: TestAccS3ObjectDataSource_checksumSHA256 (202.11s)
--- PASS: TestAccS3ObjectDataSource_checksumCRC32C (202.27s)
--- PASS: TestAccS3ObjectDataSource_checksumCRC32 (208.73s)
--- PASS: TestAccS3ObjectDataSource_checksumSHA1 (208.74s)
--- PASS: TestAccS3ObjectDataSource_allParams (213.26s)
--- PASS: TestAccS3Object_checksumSHA256FromContent (224.77s)
--- PASS: TestAccS3Object_checksumSHA1FromContent (234.12s)
--- PASS: TestAccS3Object_checksumCRC32FromContent (234.42s)
--- PASS: TestAccS3Object_checksumCRC32CFromContent (234.71s)
--- PASS: TestAccS3Object_source (235.57s)
--- PASS: TestAccS3Object_sourceHashTrigger (293.14s)
--- PASS: TestAccS3Object_checksumCRC32C (294.43s)
--- PASS: TestAccS3Object_checksumSHA1 (295.33s)
--- PASS: TestAccS3Object_checksumCRC32 (296.25s)
--- PASS: TestAccS3Object_checksumSHA256 (297.32s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/s3 300.476s

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[github-actions]

Welcome @mediba-h-sugawara 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTOR guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

[md5]

Speaking as a user of this provider and an interested third party, this looks like a nice addition. That being said, there were a couple of things that surprised me about this PR.

The first thing was the addition of code in the provider itself to calculate the checksums for the new algorithms. Based on the linked blog post, it looks like this functionality was added to the SDKs. Indeed, it looks like it was added to the AWS SDK for Go V2 in aws/aws-sdk-go-v2#1600, but it does not seem to have been added to the V1 SDK. Since the aws_s3_object resource and data source are using the V1 SDK, I understand why the choice to add the checksum code here was made, but perhaps it would be better to move to the V2 SDK?

The other thing that I noticed is that this PR does not pass along user-provided values for the checksum_* fields other than checksum_algorithm. The linked blog post shows an example of passing on a precomputed checksum using the Python SDK and I think this would be helpful for the use case where a user has already stored the checksum alongside the file to be uploaded, particularly in the case of large objects.

I think if these files were updated to use the V2 SDK, the resourceObjectUpload function could be simplified to just pass all the checksum_* fields to the SDK without looking at their values at all. If you support user-provided values for the checksum fields, hasObjectContentChanges will also need to be updated, since right now it is only checking for changes to checksum_algorithm.

[mediba-h-sugawara]

Thanks for your comment! 👍
I refactor from v1 sdk to v2 sdk now. 😎

[md5]

Hopefully that dependency update is a straightforward change to make with no unforseen downsides. The maintainers would know best if there's a specific reason that change has not been made previously.

It looks like you may also want to add ConflictsWith for all the new checksum_* properties based on that blog post. It seems to only make sense to specify checksum_algorithm when you do not provide a precomputed value or to only specify the precomputed checksum for a single algorithm, but that may be an incorrect assumption on my part.

[adamMo]

hey @mediba-h-sugawara do you have some news?

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.