End-to-end tests when multiple depended resources are required

[wiktorn]

In some cases (like this one) there are examples that contain all the necessary resources to successfully do the apply. But in many cases this is not the case (example - missing secrets).

Possible options

Creating a end-to-end example might not be as valuable to the reader, so the general idea is to provide similar framework as it is available in test/modules.

For now, I came with two possible approaches:

1. Separate "fixture" module
   In tests/modules_e2e/<module_name>/fixtures/<fixture_name> create a terraform module that provisions all dependent resources (like NEGs or secret manager + secrets, to refer to examples above) and create a fixture.tfvars file which will be provided on input of the module under test.

The second part is, that for each test case, in tests/modules_e2e/<module_name>/<test_case_name>.tfvars we would define module inputs as it is now done for test cases in tests/modules/. During the execution of the test, a third e2e.tfvars file will be injected, that is exactly the same as for tests/examples_e2e.

2. Provision module under test, and all other resources as a one terraform module. That's kind of simple - the only variables injected are the ones coming from e2e.tfvars and the rest has to be defined in the module it self.

Evaluation

I like the approach described in point 1 above, as it allows easy reuse of fixture for different test cases (we might want to consider whether they should be on module level, or global). The other advantage (?) is that during the test, we only apply module itself, without dependencies, so we don't get any failures on non-empty plan, for resources that are part of the fixture. Though this will be pretty complex approach.

The other things to consider are, whether to create separate folder structure for that, or keep all those tests in tests/modules. We could reuse existing *.tfvars test cases, if we could ensure that e2e.tfvars variables will override some of the values provided (by making sure that it is last -var-file argument).

The second approach is straightforward, but there will be no way to distinguish failures due to bad fixture from bad module.

Thoughts?

[ludoo]

Can we look at how many are in this situation?

Because I would not instinctively discard option 0 of providing the resources in the example itself, if those are few and simple. It moves some of the examples a bit closer to blueprints, but on the other hand it makes it easier to copy/paste and have working code. If the majority of examples only need a simple extra resource, which we are already doing in places as you noted, I would skip all the complexity.

My main worry is this is becoming a framework within the framework, which requires deep specific knowledge, and inevitably distracts resources away from the main goal (modules, blueprints, FAST), or adds rigidity to changes by making them very complex to implement.

As a practical example, we have a lot of important stuff we would like to have where we don't have enough resources: improving the new project factory, backporting the folder factory, fixing the know security issue with CI/CD, improving MT documentation, cleaning up old blueprints, etc. All the focus that is going in this framework will a) use resources that IMHO could be better spent towards our main goals, and b) make those changes even more slow and complicated to implement.

We need this of course, I would not just make it a primary focus of our development efforts now that we have it.