[v2] S3 high level checksums

[aemous]

Issue #, if available:

• Support S3 additional checksums in high-level S3 commands #6750

Low-level aws s3api commands support checksums other than MD5 for verifying end-to-end data integrity. This pull request ports over similar functionality to high-level aws s3 commands.

Description of changes:

• Added --checksum-algorithm flag for file uploads using aws s3 cp, aws s3 sync, and aws s3 mv.
  • Supported algorithms are CRC32, SHA256, SHA1, CRC32C.
• Added --checksum-mode flag for file downloads using aws s3 cp, aws s3 sync, and aws s3 mv.
• Added unit and functional tests to assert the correctness of these changes.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[hssyoo]

Looks like a solid start.

My main feedback here is that --checksum-algorithm should be supported for copies (ie s3s3). I can definitely see strong use cases for wanting to copy the object checksums between different S3 paths (or even generating different checksums). In order to support this, we're going to need to port this PR for the vended s3transfer package. Let's make this port a separate PR to reduce noise in this one.

We should also prefer f-strings for string formatting in all new code. ie f"var: {my_var}" instead of "var: %s" % myvar.

[hssyoo]

Playing around with it some more, I'm not sure --checksum-mode ENABLED provides anything useful without the debug flag:

aws s3 cp s3://hssyoo-brawn-test/README.rst ./ --checksum-mode ENABLED

download: s3://hssyoo-brawn-test/README.rst to ./README.rst

Compare this to the low-level get-object:

aws s3api get-object --bucket hssyoo-brawn-test --key README.rst --checksum-mode ENABLED ./README.rst

{
    ...
    "ChecksumCRC32C": "ac1Isw==",
    ...
}

With the current changes, the --checksum-mode parameter in high-level S3 commands:

• Doesn't do client-side validation of the object
• Doesn't output the checksum the user could use to validate the object

We should decide what behavior we want to support.

EDIT: Sorry, validation is automatically supported with botocore. Ignore.

[hssyoo]

Looks good! 🏆