Add an e2etest for testing restart policy

[ChanYiLin]

Fixes #639

This PR does the following things:

1. add replicas restart policy test by comparing the start time of the container in the pod
2. fixed the pod_selector in py/tf_job_client.py by modifying
   labels["tf-replica-type"] = replica_type to
   labels["tf-replica-type"] = str.lower(replica_type)
3. wait a time interval before fetching the pods' phase in py/k8s_util.py
4. fixed bugs in clean pod policy test

---

This change is 

[ChanYiLin]

/hold
only add v1alpha2 test currently

[ChanYiLin]

/test kubeflow-tf-operator-presubmit

[coveralls]

Coverage remained the same at 64.496% when pulling 16907d1 on ChanYiLin:master into 89e6f66 on kubeflow:master.

[gaocegege]

Thanks for your contribution! Feel free to assign me or richardsliu to review.

[ChanYiLin]

@gaocegege
sure, no problem!
after I finish v1beta1, I'll then need your review :)

[ChanYiLin]

/retest

[ChanYiLin]

/hold cancel

[ChanYiLin]

/assign @gaocegege @richardsliu

[gaocegege]

\lgtm

While I think we need @richardsliu 's opinion.

@richardsliu We can only review https://github.com/kubeflow/tf-operator/pull/873/files#diff-bb0480aabc8947b0a59583d963fd9f31R1

[gaocegege]

/lgtm

[richardsliu]

Thank you for contributing!

[ChanYiLin]

/retest

[ChanYiLin]

@richardsliu
HI, I have refactored the code and added more tests.
Please help me to take a look, thanks!

[richardsliu]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: richardsliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [richardsliu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment