migrate to AWS SDKv2, updating only singnatures and making sure tests are passing

[wakeful]

Description

This PR includes changes to upgrade the outdated AWS SDK to the newer SDKv2 version.

• I focused on upgrading all calls from the legacy AWS SDKv1.
• Updated function signatures to match the new SDKv2.
• Fixed issues to ensure all tests are passing.

What's missing:
- Finalise porting modules/aws/auth.go (support for MFA and assume role). I complete this over the next few days.

Fixes #1432.

TODOs

Read the Gruntwork contribution guidelines.

• Update the docs.
• Run the relevant tests successfully, including pre-commit checks.
• Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable.
• Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

migrated to AWS SDK v2.

[james03160927]

Hi @wakeful, thanks for making this contribution. It seems like some of the relevant tests are failing. Can you please run them and make sure they don't fail?

[wakeful]

hey @james03160927 I updated tests, rebase with latest master & bump all aws SDK v2 pkg - please try again if you get any errors 🙏 share the output log.

[so0k]

this is awesome @wakeful - but this should probably come with a major bump, for example for s3 fileupload I had to replace s3manager... and I can't remember other situations where the change in terratest bled through into my code (which could be more a problem with my code)...

Until this gets merged, I'm using your fork of terratest

replace github.com/gruntwork-io/terratest => github.com/wakeful/terratest v0.47.2-0.20241015095311-22a3870b391e

I'd like to see this merged (and I'd like to contribute some more utility functions such as testing Cloudfront Functions, Sfn State Machines, ...)

ref all the utilities I wrote here
https://github.com/envtio/base/tree/main/integ/aws

but I'm not sure if there's an interest for this by gruntworks (not sure if there's a slack or discord where this can be discussed?)

[wakeful]

thanks for testing @so0k!

I also have few more ideas / new helpers but first we need to merge this one.

but I'm not sure if there's an interest for this by gruntworks (not sure if there's a slack or discord where this can be discussed?)

recently there was a blog post about A stronger Terragrunt community with official Discord server - I'm hoping at some point other tools will also get a dedicated channel there.

[james03160927]

Seeing this error on packer unit test

    packer_basic_example_test.go:231: operation error EC2: ModifyImageAttribute, https response error StatusCode: 400, RequestID: 6679d41e-890b-42d7-a5ee-6a040ce75f2c, api error OperationNotPermitted: You can’t publicly share this image because block public access for AMIs is enabled for this account. To publicly share the image, you must call the DisableImageBlockPublicAccess API.

[wakeful]

hey @james03160927.

can’t publicly share this image because block public access for AMIs is enabled for this account

Based on the error above, it seems that in your AWS account (or AWS organisation) there is a policy blocking the sharing of the AMI with everyone. I’ve decided to modify the test to check if the AMI is private. This way, we won’t require any changes in your testing environment.

The fix has been pushed, and the branch has been rebased with the latest master.

[james03160927]

Can you rebase your change with the latest commit? It requires bigger circleCi resource to run the tests which I submitted here. #1472

[wakeful]

@james03160927 done 👍

[james03160927]

Still seeing failures in the following tests:
TestPackerDockerExampleLocal

TestPackerDockerExampleLocal 2024-11-18T23:49:01Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:01Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": read tcp 127.0.0.1:39468->127.0.0.1:8080: read: connection reset by peer. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:03Z retry.go:91: HTTP GET to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:03Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:03Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": dial tcp 127.0.0.1:8080: connect: connection refused. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:05Z retry.go:91: HTTP GET to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:05Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:05Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": dial tcp 127.0.0.1:8080: connect: connection refused. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:07Z retry.go:91: HTTP GET to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:07Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:07Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": dial tcp 127.0.0.1:8080: connect: connection refused. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:09Z retry.go:91: HTTP GET to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:09Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:09Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": dial tcp 127.0.0.1:8080: connect: connection refused. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:11Z retry.go:91: HTTP GET to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:11Z http_helper.go:58: Making an HTTP GET call to URL http://localhost:8080
TestPackerDockerExampleLocal 2024-11-18T23:49:11Z retry.go:103: HTTP GET to URL http://localhost:8080 returned an error: Get "http://localhost:8080": dial tcp 127.0.0.1:8080: connect: connection refused. Sleeping for 2s and will try again.
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Running command docker with args [compose --project-name testpackerdockerexamplelocal down]
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Container testpackerdockerexamplelocal-web_app-1  Stopping
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Container testpackerdockerexamplelocal-web_app-1  Stopping
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Container testpackerdockerexamplelocal-web_app-1  Stopped
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Container testpackerdockerexamplelocal-web_app-1  Removing
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Container testpackerdockerexamplelocal-web_app-1  Removed
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Network testpackerdockerexamplelocal_default  Removing
TestPackerDockerExampleLocal 2024-11-18T23:49:13Z logger.go:67: Network testpackerdockerexamplelocal_default  Removed
--- FAIL: TestPackerDockerExampleLocal (127.25s)

TestTerraformPackerExample

5s and will try again.
TestTerraformPackerExample 2024-11-18T23:58:59Z retry.go:91: HTTP GET to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:58:59Z http_helper.go:58: Making an HTTP GET call to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:58:59Z retry.go:103: HTTP GET to URL http://54.201.25.30:8080 returned an error: Get "http://54.201.25.30:8080": dial tcp 54.201.25.30:8080: connect: connection refused. Sleeping for 5s and will try again.
TestTerraformPackerExample 2024-11-18T23:59:04Z retry.go:91: HTTP GET to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:04Z http_helper.go:58: Making an HTTP GET call to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:04Z retry.go:103: HTTP GET to URL http://54.201.25.30:8080 returned an error: Get "http://54.201.25.30:8080": dial tcp 54.201.25.30:8080: connect: connection refused. Sleeping for 5s and will try again.
TestTerraformPackerExample 2024-11-18T23:59:09Z retry.go:91: HTTP GET to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:09Z http_helper.go:58: Making an HTTP GET call to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:09Z retry.go:103: HTTP GET to URL http://54.201.25.30:8080 returned an error: Get "http://54.201.25.30:8080": dial tcp 54.201.25.30:8080: connect: connection refused. Sleeping for 5s and will try again.
TestTerraformPackerExample 2024-11-18T23:59:14Z retry.go:91: HTTP GET to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:14Z http_helper.go:58: Making an HTTP GET call to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:14Z retry.go:103: HTTP GET to URL http://54.201.25.30:8080 returned an error: Get "http://54.201.25.30:8080": dial tcp 54.201.25.30:8080: connect: connection refused. Sleeping for 5s and will try again.
TestTerraformPackerExample 2024-11-18T23:59:19Z retry.go:91: HTTP GET to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:19Z http_helper.go:58: Making an HTTP GET call to URL http://54.201.25.30:8080
TestTerraformPackerExample 2024-11-18T23:59:20Z retry.go:103: HTTP GET to URL http://54.201.25.30:8080 returned an error: Get "http://54.201.25.30:8080": dial tcp 54.201.25.30:8080: connect: connection refused. Sleeping for 5s and will try again.

TestTerraformAwsRdsExample/mysql

=== NAME  TestTerraformAwsRdsExample/mysql
    apply.go:15: 
        	Error Trace:	/home/circleci/project/modules/terraform/apply.go:15
        	            				/home/circleci/project/test/terraform_aws_rds_example_test.go:108
        	Error:      	Received unexpected error:
        	            	FatalError{Underlying: error while running command: exit status 1; �[31m╷�[0m�[0m
        	            	�[31m│�[0m �[0m�[1m�[31mError: �[0m�[0m�[1mRequired plugins are not installed�[0m
        	            	�[31m│�[0m �[0m
        	            	�[31m│�[0m �[0m�[0mThe installed provider plugins are not consistent with the packages
        	            	�[31m│�[0m �[0mselected in the dependency lock file:
        	            	�[31m│�[0m �[0m  - registry.terraform.io/hashicorp/aws: the cached package for registry.terraform.io/hashicorp/aws 4.67.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
        	            	�[31m│�[0m �[0m
        	            	�[31m│�[0m �[0mTerraform uses external plugins to integrate with a variety of different
        	            	�[31m│�[0m �[0minfrastructure services. To download the plugins required for this
        	            	�[31m│�[0m �[0mconfiguration, run:
        	            	�[31m│�[0m �[0m  terraform init
        	            	�[31m╵�[0m�[0m}
        	Test:       	TestTerraformAwsRdsExample/mysql

[wakeful]

I resolved the conflicts and will work on the failing tests later today.

[wakeful]

The issue in TestTerraformPackerExample and TestPackerDockerExampleLocal was related to a missing HTTP server. It seems that the Sinatra gem no longer installs an HTTP server by default, or it may require extra configuration. I added puma and configured Sinatra to use it. Both tests are now passing for me.

The TestTerraformAwsRdsExample/mysql issue seems more like a Terraform cache problem. I reran it locally, and it passed without any issues

[wakeful]

@james03160927 Can you rerun the tests again? After my investigation, they are not related to my changes. They also failed for me on the master branch, so I fixed them.