chore: allow e2e suites to run separately (#179)

Signed-off-by: Sam Berning <[email protected]>

Issue #, if available:

*Description of changes:*

Had a discussion with @davidhsingyuchen about how to run tests in only
one of the e2e test suites, and this is the approach we landed on.

- This will allow users to more easily test only the e2e test scenarios
they are working on
- We can potentially run the two suites in parallel in CI on different
runners

One drawback of this approach is that each run of the full e2e tests
will take 3-5 minutes longer, due to running the
`SynchronizedBeforeSuite` an additional time.

*Testing done:*

```
make test-e2e
make test-e2e-container
make test-e2e-vm
```

- [x] I've reviewed the guidance in CONTRIBUTING.md


#### License Acceptance

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.

Signed-off-by: Sam Berning <[email protected]>

CONTRIBUTING.md

@@ -228,15 +228,24 @@ Keeping a good unit test coverage will be part of pull request review. You can r
 
 See `test-e2e` section in [`Makefile`](./Makefile) for more reference.
 
+In this repository, there are two suites of E2E tests: `container` tests and `vm` tests.
+
 If the e2e test scenarios you are going to contribute
 
 - are in generic container development workflow
 - can be shared by finch-core by replacing test subject from "finch" to "limactl ..."
 - E.g.: pull, push, build, run, etc.
 
-implement them in common-tests repo and then import them in [`./e2e/e2e_test.go`](./e2e/e2e_test.go) in finch CLI and finch-core. The detailed flow can be found [here](https://github.com/runfinch/common-tests#sync-between-tests-and-code).
+they belong with the `container` tests. Implement them in common-tests repo and then import them in [`./e2e/container/container_test.go`](./e2e/container/container_test.go) in finch CLI and `./e2e/e2e_test.go` in finch-core. The detailed flow can be found [here](https://github.com/runfinch/common-tests#sync-between-tests-and-code).
+
+Otherwise, it means that the scenarios are specific to finch CLI (e.g., version, VM lifecycle, etc.), and belong with the `vm` tests. You should implement them under `./e2e/vm/` (e.g., `./e2e/vm/version_test.go`) and import them in `./e2e/vm/vm_test.go`.
+
+If you want to run just one of the two suites, append the suite name to the end of the Makefile target:
 
-Otherwise, it means that the scenarios are specific to finch CLI (e.g., version, VM lifecycle, etc.), and you should implement them under `./e2e/` (e.g., `./e2e/version.go`) and import them in `./e2e/e2e_test.go`.
+```sh
+make test-e2e-container
+make test-e2e-vm
+```
 
 To save time while developing e2e tests, use the [`Focus`](https://onsi.github.io/ginkgo/#focused-specs) decorator while running tests, but be sure to remove it before PR-ing your changes.
 

Makefile

@@ -244,11 +244,21 @@ test-unit:
 
 # test-e2e assumes the VM instance doesn't exist, please make sure to remove it before running.
 #
-# Container tests have to be run before VM tests according to the current code setup.
+# Container tests and VM tests can be run in any order, but they must be run sequentially.
 # For more details, see the package-level comment of the e2e package.
 .PHONY: test-e2e
-test-e2e:
+test-e2e: test-e2e-vm-serial
+
+.PHONY: test-e2e-vm-serial
+test-e2e-vm-serial: test-e2e-container
+	go test -ldflags $(LDFLAGS) -timeout 30m ./e2e/vm -test.v -ginkgo.v --installed="$(INSTALLED)"
+
+.PHONY: test-e2e-container
+test-e2e-container:
 	go test -ldflags $(LDFLAGS) -timeout 30m ./e2e/container -test.v -ginkgo.v --installed="$(INSTALLED)"
+
+.PHONY: test-e2e-vm
+test-e2e-vm:
 	go test -ldflags $(LDFLAGS) -timeout 30m ./e2e/vm -test.v -ginkgo.v --installed="$(INSTALLED)"
 
 .PHONY: gen-code

e2e/container/container_test.go

@@ -24,13 +24,17 @@ func TestContainer(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	// The VM should be cleaned up in SynchronizedAfterSuite of e2e/vm.
 	ginkgo.SynchronizedBeforeSuite(func() []byte {
 		command.New(o, "vm", "init").WithTimeoutInSeconds(600).Run()
 		tests.SetupLocalRegistry(o)
 		return nil
 	}, func(bytes []byte) {})
 
+	ginkgo.SynchronizedAfterSuite(func() {
+		command.New(o, "vm", "stop").WithTimeoutInSeconds(60).Run()
+		command.New(o, "vm", "remove").WithTimeoutInSeconds(60).Run()
+	}, func() {})
+
 	ginkgo.Describe(description, func() {
 		tests.Pull(o)
 		tests.Rm(o)

e2e/vm/vm_test.go

@@ -27,7 +27,11 @@ func TestVM(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	// The VM should be spined up in SynchronizedBeforeSuite of e2e/container.
+	ginkgo.SynchronizedBeforeSuite(func() []byte {
+		command.New(o, "vm", "init").WithTimeoutInSeconds(600).Run()
+		return nil
+	}, func(bytes []byte) {})
+
 	ginkgo.SynchronizedAfterSuite(func() {
 		command.New(o, "vm", "stop").WithTimeoutInSeconds(60).Run()
 		command.New(o, "vm", "remove").WithTimeoutInSeconds(60).Run()