Notes on Amazon S3

Getting started: Configure AWS credential environment variables for use in other GitHub Actions

• aws-actions/configure-aws-credentials: Configure AWS credential environment variables for use in other GitHub Actions.
• AWS Developer Forums: Possible to sync a single file with aws ...
• sync — AWS CLI 1.19.96 Command Reference

Bucket policy

• Bucket policy examples - Amazon Simple Storage Service
• An easier way to control access to AWS resources by using the AWS organization of IAM principals | AWS Security Blog

Owner access: bucket-owner-full-control

• Resolve 403 errors from S3 objects uploaded by other accounts
• Controlling ownership of uploaded objects using S3 Object Ownership - Amazon Simple Storage Service

Miscellaneous links

• InventoryAndAnalyticsExamplePolicy (See dlemieux1 for example)
• Example walkthroughs: Managing access to your Amazon S3 resources - Amazon Simple Storage Service

IAM

• Make sure s3:PutObjectAcl is included in the IAM policy. This does not need to be defined in the Bucket policy.

• Troubleshooting IAM policies - AWS Identity and Access Management

• Permissions boundaries for IAM entities - AWS Identity and Access Management

Pricing

• Amazon S3 Simple Storage Service Pricing - Amazon Web Services

Cost-saving opportunities?

• with S3 intelligent tiers why we need life cycle policies? to retire yes we may need but tiers can be automatically managed with S3 intelligent tier - Course: AWS Certified Solutions Architect Associate SAA-C02

Verify checksum

Amazon's official method: Check the integrity of an object uploaded to Amazon S3 which verifies the object's integrity during the upload with the use of the low-level aws s3api put-object command.

But I want to use aws s3 sync.

Get the ETag:

Can be fetched by using ListObjectsV2 - Amazon Simple Storage Service:

$ aws s3api list-objects-v2 --bucket opendrr-api-prebuilt-cache-1 --profile opendrr-api-s3-reader

which returns the following JSON:

{
    "Contents": [
        {
            "Key": "boundaries/opendrr-boundaries.sql",
            "LastModified": "2021-06-17T06:05:14+00:00",
            "ETag": "\"20b8d16a40cbe5daf6f4d5dfd051840b-131\"",
            "Size": 1098382563,
            "StorageClass": "STANDARD"
        }
    ]
}

To extract ETag from this JSON output with jq:

$ aws s3api list-objects-v2 --bucket opendrr-api-prebuilt-cache-1 --profile opendrr-api-s3-reader \
    | jq -r '.Contents[] | select(.Key == "boundaries/opendrr-boundaries.sql").ETag'
"20b8d16a40cbe5daf6f4d5dfd051840b-131"

Calculate MD5 checksum

s3etag.sh: A Bash script to compute ETag values for S3 multipart uploads on OS X by @emersonf in Nov 2013, and revised for Linux by @skchronicles for Linux in Apr 2020. This works AWESOME!

$ ./s3etag.sh 
Usage: ./s3etag.sh file partSizeInMb
$ ./s3etag.sh opendrr-boundaries.sql 8
opendrr-boundaries.sql	20b8d16a40cbe5daf6f4d5dfd051840b-131

The s3etag.sh gist was referenced by a recent post (May 2020) How to get the md5sum of a file on Amazon's S3 - Stack Overflow

All about AWS S3 ETags - Teppen.io is another great write-up, but for some reasons, its Python implementation, Calculating the S3 ETag for a local file - Teppen.io, returns wrong checksum result for the file I tested.

Miscellaneous links

• Amazon S3 actions - Amazon Simple Storage Service
• Troubleshoot issues copying an object between S3 buckets