Binary acceptance test driver

[kmoe]

This PR adds a new optional test driver for Terraform provider acceptance tests that runs the Terraform CLI.

To minimise work for provider developers at this stage, the API follows as closely as possible the functionality of the legacy test driver, minimising changes needed to acceptance test code.

Since we are targeting a 1.x.x release of the provider SDK, no breaking changes will be made, and the legacy test driver will only be removed in version 2.0.0.

Usage

The new test driver must be enabled by initialising the test helper in the TestMain() function in all provider packages that run acceptance tests. Most providers have only one package.

Import the new "github.com/hashicorp/terraform-plugin-sdk/acctest" package and add the following TestMain function to provider_test.go:

func TestMain(m *testing.M) {
	acctest.UseBinaryDriver("provider_name", Provider)
	resource.TestMain(m)
}

Some providers already have a TestMain() defined, usually for the purpose of enabling test sweepers (as described in Extending Terraform: Sweepers). These additional occurrences should be removed.

Known issue: TypeSet attribute checks

A limitation of the binary acceptance test driver is that attribute checks addressing TypeSet elements will fail.

These attribute checks are rarely used in the legacy test framework, as the hashcode used for addressing Set elements in the flatmap representation must be calculated during the test. Please see #196.

A follow-up improvement to the SDK type system will fix this issue and provide an idiomatic way to check complex type attributes.

Opt-out per TestCase

Initialising the binary test helper using acctest.UseBinaryDriver() causes all tests to be run using the new binary driver. Until SDK version 2.0.0, the DisableBinaryDriver boolean property can be used to use the legacy test driver for individual TestCases.

Multi-provider testing

Providers that require resources from other providers during acceptance tests have two options:

• Standard provider discovery mechanism. During acceptance test runs, Terraform CLI will attempt to locate providers on the local filesystem, and will download any missing providers in the usual way. This is the default behaviour.
• Binaries in the provider repo. Relying on the Terraform CLI provider discovery mechanism can be avoided by including auxiliary provider binaries in terraform.d/plugins/$GOOS_$GOARCH/$provider_name, relative to the provider source code directory root. The environment variable TF_ACC_PROVIDER_ROOT_DIR must be set to the path of the source code directory root to use this feature. Providers can be included in binary or zip format.

It is no longer necessary to import other Terraform providers as Go modules: these imports should be removed.

Further examples will be added to this section as I raise PRs to enable the binary test driver in HashiCorp-owned providers.

Closes #13

[appilon]

I think you accidentally committed vendor

Fortunately looks like an easy fix just remove the last commit:

2f0ac66

[appilon]

Do we know if any of "the big providers" even use this? What value does it provide, I'm not sure it's a modern practice I would be okay if we made providers "delete" those tests or rewrite them?

[kmoe]

Copying some data here from a conversation elsewhere:

There are 806 occurrences of ID refresh tests in terraform-providers providers:

$ grep -r --exclude-dir="vendor" --include="*_test.go" "IDRefresh" * | wc -l
806

and more than half of these are in "big" providers:

$ grep -r --exclude-dir="vendor" --include="*_test.go" "IDRefresh" * | grep -e aws -e azure -e google -e kubernetes -e oracle | wc -l
454

[appilon]

One minor aspect that needs to be covered is how to providers that call into our resource.TestMain() can get setup with the new framework. It is used to run sweepers (maybe this is what you were referring to on the todo list)

[appilon]

the providers map is something I managed to cut as cruft in the version2 branch, wondering if we can get away without it (so I don't need to restore it in that branch)? Can't remember if I authored this comment but seems like its lingering around for ImportStateVerify

[radeksimko]

I suppose we can keep this in when merging into master (as opt-in) and then remove it in version2 branch. This would align with the risk profile you set by removing it in version2 I expect?

We could add a comment here to remember to do it after merging?

[appilon]

I agree we should remove the conditional check that acts as an "opt-in" in version2. Unfortunately it's the map passed to RunLegacyTest called providers that I cut from version2 and once this is merged to master I will need to rebase version2 and restore it. Unless of course we trace providers through RunLegacyTest and find a way to remove it.

It's not a big deal I'm sure we will get the merge conflicts solved one way or another.

[appilon]

amazing that you reverse engineered whatever this was? I have never written an "IDRefresh" test myself and I don't think its covered in the extend docs.

[appilon]

The beautiful new binary driver is over my head at this point. It's insulated from causing much harm though as it's still opt in! So I say let's give it a shot!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.