annotation of sidecar.istio.io/inject: "false" in TFJob got igored

[yhwang]

I am verifying v1.4-rc1. I have tfjob like this:

apiVersion: "kubeflow.org/v1"
kind: TFJob
metadata:
  name: tfjob-simple
  namespace: yh-wang
  annotations:
    sidecar.istio.io/inject: "false"
spec:
   tfReplicaSpecs:
     Worker:
       replicas: 2
       restartPolicy: OnFailure
       template:
         metadata:
           annotations:
             sidecar.istio.io/inject: "false"
         spec:
           containers:
             - name: tensorflow
               image: gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0
               command:
                 - "python"
                 - "/var/tf_mnist/mnist_with_summaries.py"

and apply it to the user namespace : my-namespace. I tried to add the annotation in two places. no luck.
I always got istio-proxy sidecar:

  Normal   Created    10s   kubelet            Created container tensorflow
  Normal   Started    9s    kubelet            Started container tensorflow
  Normal   Pulling    9s    kubelet            Pulling image "docker.io/istio/proxyv2:1.9.6"
  Normal   Pulled     8s    kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.6" in 829.923573ms
  Normal   Created    8s    kubelet            Created container istio-proxy
  Normal   Started    7s    kubelet            Started container istio-proxy

is this a known behavior change?

[andreyvelich]

Thank you for testing Training Operators @yhwang!

This should be fixed in: #2033 and included in v1.4-branch.
cc @kubeflow/wg-training-leads

[yhwang]

Thanks for the info. Then I will wait for the rc2 and verify it again then. Let me close this issue for now.

[yhwang]

verified this again with v1.4.0-rc.2. works properly.