Support ChecksumSHA256 in aws_s3_object data source output

[maunzCache]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

I do have a project setup were i need to handle lambda update triggers by providing lambda layer code from an S3 bucket. I cannot provide the code on runtime e.g. terraform apply because the internal hash calculation seems to differ on different machines which may be a bug but that is not my point. I just found out that AWS S3 API supports "custom" checksums additionally to the ETag. Those have been introduced in February of 2022 . Thus the AWS API now supports additional parameters to access those values which are available in the go API as well e.g. https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#GetObjectOutput (Note: They messed up theirs docs but it is there!). You can see it in the GetObjectOutput struct:

    // The base64-encoded, 32-bit CRC32 checksum of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this
    // may not be a checksum value of the object. For more information about how
    // checksums are calculated with multipart uploads, see Checking object integrity
    // (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32 *string `location:"header" locationName:"x-amz-checksum-crc32" type:"string"`

    // The base64-encoded, 32-bit CRC32C checksum of the object. This will only
    // be present if it was uploaded with the object. With multipart uploads, this
    // may not be a checksum value of the object. For more information about how
    // checksums are calculated with multipart uploads, see Checking object integrity
    // (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32C *string `location:"header" locationName:"x-amz-checksum-crc32c" type:"string"`

    // The base64-encoded, 160-bit SHA-1 digest of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this
    // may not be a checksum value of the object. For more information about how
    // checksums are calculated with multipart uploads, see Checking object integrity
    // (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA1 *string `location:"header" locationName:"x-amz-checksum-sha1" type:"string"`

    // The base64-encoded, 256-bit SHA-256 digest of the object. This will only
    // be present if it was uploaded with the object. With multipart uploads, this
    // may not be a checksum value of the object. For more information about how
    // checksums are calculated with multipart uploads, see Checking object integrity
    // (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA256 *string `location:"header" locationName:"x-amz-checksum-sha256" type:"string"`

I would really love to have access to those parameters on the aws_s3_object data source. It would help a lot.

New or Affected Resource(s)

• aws_s3_object ( terraform-provider-aws/internal/service/s3/object_data_source.go )

Potential Terraform Configuration

data "aws_s3_object" "my_object" {
  bucket = "myBucket"
  key    = "myFile.zip"
}

resource "aws_lambda_layer_version" "my_lambda" {
  description = "Layer containing the my_object layer. (Deployed from S3)"

  compatible_runtimes = "python3.8"
  s3_bucket           = data.aws_s3_object.layer_from_s3.bucket
  s3_key              = data.aws_s3_object.layer_from_s3.key
  s3_object_version   = data.aws_s3_object.layer_from_s3.version_id
  source_code_hash    = data.aws_s3_object.layer_from_s3.checksum_sha256
}

References

• None seen

[xnick123]

This would be useful to me too in conjunction with the aws_signer_signing_job, as i want to know the checksum ob the artifact it uploaded to S3.

[saxonww]

I'm automating lambda layer and function updates from objects delivered to an S3 bucket right now, and doing so is a little clunky without support for this.

What I'm doing at the moment is adding the sha256 value to the object's metadata when I copy it:

aws s3 cp --metadata sha256=$(sha256sum layer.zip | cut -d' ' -f1 | xxd -r -p | base64) layer.zip s3://bucket/layer.zip

Then I'm using data.aws_s3_object.layer_zip.metadata["Sha256"] - it gets capitalized for some reason - as the source_code_hash value for aws_lambda_function. It works.

Aside from being clunky, this is not great because I have to trust that the metadata on the object is correct. In this case I think the worst that can happen is the metadata value is wrong, and the object gets re-deployed, though.

I'm delivering layer.zip to S3 outside of Terraform because it's non-deterministic, so the hash is changing every time anyway. My choices are I think to do this (and control updates via my CI system), or implement something where I download the file to have Terraform read it via data.local_file. The latter doesn't have the potential for mismatched metadata, but it does mean an extra copy that I don't want.

[github-actions]

This functionality has been released in v5.17.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.