HTTPBin example: Setup step is flaky

[dgzlopes]

I tried to run the HTTPBin example and got hit by some problems.

First, the 10s timeout was hit while waiting for the pod to be ready:

ERRO[0013] aborting test. Pod httpbin not ready after 10 seconds
        at setup (file:///Users/dgzlopes/go/src/github.com/grafana/xk6-disruptor/examples/httpbin/disrupt-pod.js:24:8(36))
        at native  hint="script exception"

I changed the timeout and re-run the script after this happened and got another error:

ERRO[0003] GoError: namespaces "httpbin-ns" already exists
        at reflect.methodValueCall (native)
        at setup (file:///Users/dgzlopes/go/src/github.com/grafana/xk6-disruptor/examples/httpbin/disrupt-pod.js:18:14(7))
        at native  hint="script exception"

If the test finishes abruptly, the teardown phase isn't executed, and the namespace and its contents from the past run hang around. I removed this manually and run the test again. Another error:

ERRO[0064] setup() execution timed out after 60 seconds  hint="You can increase the time limit via the setupTimeout option"

Right. Self-explanatory 😄

Also: grafana/xk6-kubernetes#79

[pablochacin]

Hi @dgzlopes The flakiness you mention has Three elements that depends on the test environment:

1. The timeout for the creation of resources. This depends totally on the environment. I think it would probably worth documenting this error in the example and also explain how to increase it by changing the timeout variable.

2. "httpbin-ns" already exists: This is trickier to solve without making the code more complex because there are two options here: 1) the previous test didn't finished properly and the namesapce must be deleted manually or 2) the delete from the previous execution is still ongoing. It is possible to check the status of the namespace and either wait for it to be deleted or force the deletion, but this add complexity to the code. Another way (maybe better) is to use a random namespace for each test.

3. The setup failure also depends on the time it takes for creating an ephemeral container in a certain environment, including the image download. As the message explains, the only option is to increase the timeout. I'm not sure if it is clear from the message which option is that.

[dgzlopes]

Yeah, agree 👍

(1) Documentation would be pretty nice, yup.
(2) I wonder if it is related to grafana/xk6-kubernetes#79

Just wondering if we should instruct users trying the examples to use Kubectl before launching the k6 script. That way, the example is much more reproducible in almost every case, and we can get fancy with provisioning and using the k8s extension in the future.