test/e2e: Add wait for namespace service account

[stevenhorsman]

In testing I saw that the default ServiceAccount for the newly created namespace wasn't always available straight away, especially as the cluster gets more namespaces in.

To try and fix this I've added a new waitForNamespaceToBeUseable method and done a poll in there every 5 seconds to check if the 'default' serviceAccount is in the namespace yet

Fixes: #1657

[stevenhorsman]

When testing this locally I see that is can sometime take a while between the namespace being active and the service account being created:

time="2024-01-12T17:47:37Z" level=info msg="Creating namespace 'coco-pp-e2e-test-15bf0f25'..."
time="2024-01-12T17:47:37Z" level=info msg="Wait for namespace 'coco-pp-e2e-test-15bf0f25' be ready..."
time="2024-01-12T17:47:42Z" level=info msg="Wait for default serviceaccount in namespace 'coco-pp-e2e-test-15bf0f25'..."
time="2024-01-12T17:47:42Z" level=trace msg="default serviceAccount not found after 0 seconds"
time="2024-01-12T17:47:47Z" level=trace msg="default serviceAccount not found after 5 seconds"
time="2024-01-12T17:47:52Z" level=trace msg="default serviceAccount not found after 10 seconds"
time="2024-01-12T17:47:57Z" level=trace msg="default serviceAccount not found after 15 seconds"
time="2024-01-12T17:48:02Z" level=trace msg="default serviceAccount not found after 20 seconds"
time="2024-01-12T17:48:07Z" level=trace msg="default serviceAccount not found after 25 seconds"
time="2024-01-12T17:48:12Z" level=trace msg="default serviceAccount not found after 30 seconds"
time="2024-01-12T17:48:17Z" level=trace msg="default serviceAccount not found after 35 seconds"
time="2024-01-12T17:48:22Z" level=trace msg="default serviceAccount not found after 40 seconds"
time="2024-01-12T17:48:27Z" level=trace msg="default serviceAccount not found after 45 seconds"
time="2024-01-12T17:48:32Z" level=trace msg="default serviceAccount not found after 50 seconds"
time="2024-01-12T17:48:37Z" level=trace msg="default serviceAccount not found after 55 seconds"
time="2024-01-12T17:48:42Z" level=trace msg="default serviceAccount not found after 60 seconds"
time="2024-01-12T17:48:47Z" level=trace msg="default serviceAccount not found after 65 seconds"
time="2024-01-12T17:48:52Z" level=trace msg="default serviceAccount not found after 70 seconds"
time="2024-01-12T17:48:57Z" level=trace msg="default serviceAccount not found after 75 seconds"
time="2024-01-12T17:49:02Z" level=info msg="default serviceAccount exists, namespace 'coco-pp-e2e-test-15bf0f25' is ready for use"
=== RUN   TestLibvirtCreateSimplePod
=== RUN   TestLibvirtCreateSimplePod/SimplePeerPod_test
    assessment_runner.go:229: Waiting for containers in pod: simple-test are ready
=== RUN   TestLibvirtCreateSimplePod/SimplePeerPod_test/PodVM_is_created
time="2024-01-12T17:49:47Z" level=info msg="Deleting pod simple-test..."
time="2024-01-12T17:49:52Z" level=info msg="Pod simple-test has been successfully deleted within 60s"
--- PASS: TestLibvirtCreateSimplePod (50.05s)
    --- PASS: TestLibvirtCreateSimplePod/SimplePeerPod_test (50.05s)
        --- PASS: TestLibvirtCreateSimplePod/SimplePeerPod_test/PodVM_is_created (0.00s)

so I think this was the cause of the failures before.

[liudalibj]

thanks @stevenhorsman
this is helpful to avoid we meet the error assessment_runner.go:223: pods "simple-test" is forbidden: error looking up service account coco-pp-e2e-test-947078b3/default: serviceaccount "default" not found again.

[huoqifeng]

LGTM