Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helm test failure not reported to gubernator #355

Closed
jlewi opened this issue Jan 30, 2018 · 2 comments
Closed

Helm test failure not reported to gubernator #355

jlewi opened this issue Jan 30, 2018 · 2 comments
Labels
area/testing kind/bug

Comments

@jlewi
Copy link
Contributor

jlewi commented Jan 30, 2018

See #351; helm test is failing but this is not properly reported to gubernator.

Here are the logs from Airflow from the run_tests step.

[2018-01-30 13:21:58,659] {base_task_runner.py:98} INFO - Subtask: INFO:root:Creating gs://kubernetes-jenkins/pr-logs/pull/tensorflow_k8s/354/tf-k8s-presubmit/442/artifacts/junit_e2e.xml
[2018-01-30 13:21:58,660] {base_task_runner.py:98} INFO - Subtask: Traceback (most recent call last):
[2018-01-30 13:21:58,660] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[2018-01-30 13:21:58,661] {base_task_runner.py:98} INFO - Subtask:     "__main__", fname, loader, pkg_name)
[2018-01-30 13:21:58,661] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
[2018-01-30 13:21:58,661] {base_task_runner.py:98} INFO - Subtask:     exec code in run_globals
[2018-01-30 13:21:58,662] {base_task_runner.py:98} INFO - Subtask:   File "/var/lib/data/runs/tf_k8s_tests/2018-01-30T13_12_48/tensorflow_k8s/py/deploy.py", line 220, in <module>
[2018-01-30 13:21:58,662] {base_task_runner.py:98} INFO - Subtask:     main()
[2018-01-30 13:21:58,662] {base_task_runner.py:98} INFO - Subtask:   File "/var/lib/data/runs/tf_k8s_tests/2018-01-30T13_12_48/tensorflow_k8s/py/deploy.py", line 217, in main
[2018-01-30 13:21:58,663] {base_task_runner.py:98} INFO - Subtask:     args.func(args)
[2018-01-30 13:21:58,663] {base_task_runner.py:98} INFO - Subtask:   File "/var/lib/data/runs/tf_k8s_tests/2018-01-30T13_12_48/tensorflow_k8s/py/deploy.py", line 117, in test
[2018-01-30 13:21:58,664] {base_task_runner.py:98} INFO - Subtask:     t.failure = "helm test failed;\n" + e.output
[2018-01-30 13:21:58,664] {base_task_runner.py:98} INFO - Subtask: TypeError: cannot concatenate 'str' and 'NoneType' objects

So because an exception is occurring we never set t.failure and the test isn't reported as failed.

@jose5918 can you fix this is in the same PR as #351? We can't fix this issue without the fix to #351 because without that fix the tests won't pass.
@jose5918
Copy link
Contributor

@jlewi Sure

@jose5918 jose5918 mentioned this issue Jan 30, 2018
Merged
jlewi pushed a commit that referenced this issue Jan 31, 2018
@jose5918 @jlewi
Fix helm test (#356)
cabc1c0 
* Helm test was failing because validation for a tfjob required that replicaSpecs for a Parameter server specify a template. Helm test failure also was not reported.

Changes made:

* Updated e2e tests and examples to include a template for the PS replicaSpec
* Check for None before concatenating the error.
* Fixes #351 and #355
@gaocegege
Copy link
Member

It seems that problem has been solved, could we close it?

@jlewi jlewi closed this as completed Feb 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/testing kind/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jlewi @jose5918 @gaocegege