TestFunctional/Tunnel on none: svc should have ingress after tunnel is created, but it was empty!

[tstromberg]

This is regularly occurring in our CI system:

https://storage.googleapis.com/minikube-builds/logs/3767/Linux-None.txt

    --- PASS: TestFunctional/Status (0.36s)
        cluster_status_test.go:35: Checking if cluster is healthy.
    --- FAIL: TestFunctional/Tunnel (3.97s)
        tunnel_test.go:45: starting tunnel test...
        tunnel_test.go:62: deploying nginx...
        tunnel_test.go:83: getting nginx ingress...
        tunnel_test.go:98: svc should have ingress after tunnel is created, but it was empty!

[tstromberg]

I did some searching through to see when this issue started becoming prevalent:

find /var/lib/jenkins/jobs/Linux_Integration_Tests_none -name log | xargs grep "svc should have ingress after tunnel is created" > /tmp/hits

Sorting this list, you can see that the problem becomes prevalent starting at build 3085 (Mar 1, built from #3774), and consistent by build 3089 (Mar 2, built from upstream:

NOTE: 3074 was also built from #3774

Linux_Integration_Tests_none/builds/1289/log
Linux_Integration_Tests_none/builds/1356/log
Linux_Integration_Tests_none/builds/1581/log
Linux_Integration_Tests_none/builds/1591/log
Linux_Integration_Tests_none/builds/1598/log
Linux_Integration_Tests_none/builds/3074/log
Linux_Integration_Tests_none/builds/3085/log
Linux_Integration_Tests_none/builds/3086/log
Linux_Integration_Tests_none/builds/3089/log
Linux_Integration_Tests_none/builds/3090/log
Linux_Integration_Tests_none/builds/3091/log
...

Based on the initial evidence, I wouldn't be surprised if #3774 (2f1f52e) had some unintentional impact here. If nothing else, it gives a good place to begin with git bisect.

[tstromberg]

Any luck?

[balopat]

Should be working now, fixed in #4105

[balopat]

In the end everything was working just fine with tunnel in None driver as well. We just needed more wait for kubernetes to bring up the ingress than the 2 seconds that we waited. I fixed it by setting up a better polling/error handling in the integration test.