tests/provider: Skip unsupported tests with PreChecks

[gdavison]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

We currently use PreCheck functions in acceptance tests to ensure that acceptance tests can be executed in the current partition, account, and/or region. If an endpoint does not exist or a sample operation returns certain error values, the test is skipped.

These are some examples of what a precheck looks for:

1. Endpoint existence: If a service's endpoint does not exist, the service cannot be tested. For instance, testAccPartitionHasServicePreCheck("codestar", t) verifies that the CodeStar service is available.
2. Feature availability: Not all features of a service may exist in all partitions, accounts, and/or regions. Often you must add a custom precheck function to test for a feature. For instance, database performance insights are not supported in databases running in GovCloud. A custom function ensures that performance insights are available before testing the feature. NOTE: Ideally, the precheck should verify actual functionality instead of being a hardcoded partition check. That way, when AWS adds this feature to the RDS service in the future, the precheck won't need to be modified to start running tests as soon as the functionality is available.
3. Environment & credentials: Many existing precheck functions check to make sure that credentials, environment variables, regions, alternate accounts, alternate regions, etc. are available for tests that require them.

In the acceptance test, add a call to an endpoint precheck (e.g., testAccPartitionHasServicePreCheck("codestar", t) below) and/or a custom precheck function that tests for necessary underlying conditions for the test to be run:

func TestAccAWSCodeStarNotificationsNotificationRule_Status(t *testing.T) {
	// code...

	resource.ParallelTest(t, resource.TestCase{
		PreCheck:     func() { testAccPreCheck(t); testAccPartitionHasServicePreCheck("codestar", t) },
		Providers:    testAccProviders,
		CheckDestroy: testAccCheckAWSCodeStarNotificationsNotificationRuleDestroy,
		Steps: []resource.TestStep{
			// code ...
		},
	})
}

An example of a custom precheck:

func testAccPreCheckAWSCodePipeline(t *testing.T) {
	conn := testAccProvider.Meta().(*AWSClient).codepipelineconn

	input := &codepipeline.ListPipelinesInput{}

	_, err := conn.ListPipelines(input)

	if testAccPreCheckSkipError(err) {
		t.Skipf("skipping acceptance testing: %s", err)
	}

	if err != nil {
		t.Fatalf("unexpected PreCheck error: %s", err)
	}
}

These checks can only run against the default provider configuration. In some cases, a PreCheck needs to run against an alternate provider configuration, such as for an alternate region or account.

Add functions to the testing suite that will configure AWS API connections using the authentication configuration settings for acceptance tests.

[YakDriver]

You do not need access to GovCloud to assist with adding prechecks!

When verifying that a precheck works properly, the test should be skipped when preconditions are not met. However, it is also important to ensure that the test is not skipped when preconditions are met. For example, if you add a precheck for a service and do not access to GovCloud, verifying that tests aren't skipped in the standard/AWS partition is sufficient.

What a skipped test will look like:

    provider_test.go:511: skipping tests; partition aws-us-gov does not support codestar service
--- SKIP: TestAccAWSCodeStarNotificationsNotificationRule_Status (1.61s)

What a test looks like that is not skipped:

--- PASS: TestAccAWSCodeStarNotificationsNotificationRule_basic (18.75s)

Endpoints still failing (e.g., no such host):

• appmesh (tests/provider: Add prechecks (AppMesh) #15503)
• appsync (tests/provider: Add precheck (appsync) #15504)
• budgets (tests/provider: Add prechecks (budgets) #15506)
• cloud9 (tests/provider: Add precheck (Cloud9) #15508)
• cloudfront (tests/provider: Add precheck (CloudFront) #15509)
• codeartifact
• codepipeline (us-gov-east-1)
• codestar-notifications
• datapipeline
• elastictranscoder
• fsx
• gamelift
• globalaccelerator
• kinesisvideo
• lightsail
• macie
• mediapackage
• mediastore
• models.lex
• mq
• opsworks
• qldb
• sdb
• servicediscovery
• servicequotas
• shield
• waf
• worklink

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!