[Test Flake] tf_job_simple_test needs retries for ks init to deal with git connection issues

[jlewi]

See stack trace below. The test failed because ks init had a connection error trying to read from
git. We should have retries to deal with this.

W
ERROR|2018-07-05T06:16:46|test_helper.py:98| Subprocess failed;
level=info msg="Using context \"gke_kubeflow-ci_us-east1-d_e2e-834a\" from kubeconfig file \"/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-110
9-866b43a-2372-834a/.kube/config\""
level=error msg="unable to find SHA1 for repo: Get https://api.github.com/repos/ksonnet/parts/commits/master: read tcp 10.36.0.175:60958->192.30.253.116:4
43: read: connection reset by peer"
Traceback (most recent call last):
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/testing/py/kubeflow/testing/test_helper.py", line 96,
 in wrap_test
   test_case.test_func(test_case)
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/kubeflow/testing/tf_job_simple_test.py", line 72, in
test_tf_job_simple
   util.run(["ks", "init", "tf-job-simple-app"])
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/testing/py/kubeflow/testing/util.py", line 74, in run
   " ".join(command), process.returncode), "\n".join(output))
CalledProcessError: Command 'cmd: ks init tf-job-simple-app exited with code 1' returned non-zero exit status 1
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
   "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
   exec code in run_globals
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/kubeflow/testing/tf_job_simple_test.py", line 99, in
<module>
   test_suite.run()
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/testing/py/kubeflow/testing/test_helper.py", line 73,
 in run
   wrap_test(test_case)
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/testing/py/kubeflow/testing/test_helper.py", line 96,
 in wrap_test
   test_case.test_func(test_case)
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/kubeflow/testing/tf_job_simple_test.py", line 72, in
test_tf_job_simple
   util.run(["ks", "init", "tf-job-simple-app"])
 File "/mnt/test-data-volume/kubeflow-presubmit-kubeflow-e2e-gke-1109-866b43a-2372-834a/src/kubeflow/testing/py/kubeflow/testing/util.py", line 74, in run
   " ".join(command), process.returncode), "\n".join(output))
subprocess.CalledProcessError: Command 'cmd: ks init tf-job-simple-app exited with code 1' returned non-zero exit status 1

[pdmack]

@jlewi I think we should consider adding --skip-default-registries to ks init to avoid hitting the ksonnet repo in the test workflows. These are ephemeral apps anyway and it's not clear to me that fetching the ksonnet incubator SHA is actually necessary to launching kubeflow. The ks "bits" are in the lib dir AFAICT.

DEBUG github: fetching SHA1 for ksonnet/parts - master
DEBUG github: fetching contents for ksonnet/parts/incubator/registry.yaml - 40285d8a14f1ac5787e405e1023cf0c07f6aa28c

[jlewi]

Good idea.