Coordinator field not set when jobset was installed with version before coordinator support then upgraded

[avrittrohwer]

What happened:

In a cluster with jobset API installed with a version before coordinator support, then upgraded to a version with coordinator support, when submitting a jobset yaml with coordinator field set, the coordinator field is nil in the resulting stored jobset

What you expected to happen:

Coordinator field is not nil

How to reproduce it (as minimally and precisely as possible):

This is reproducible in a kind cluster with the follwing jobset:

apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
  name: js
spec:
  coordinator:
    replicatedJob: leader
    jobIndex: 0
    podIndex: 0
  replicatedJobs:
  - name: leader
    replicas: 1
    template:
      spec:
        parallelism: 1
        completions: 1
        template:
          spec:
            containers:
            - name: progrsm
              image: gcr.io/k8s-staging-perf-tests/sleep:v0.1.0
              args:
              - 10m

1. kind create cluster
2. kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.4.0/manifests.yaml
3. Expect to fail: kubectl apply -f js.yaml: Error from server (BadRequest): error when creating "js.yaml": JobSet in version "v1alpha2" cannot be handled as a JobSet: strict decoding error: unknown field "spec.coordinator"
4. Upgrade to version with coordinator: kubectl apply --server-side -f https://github.com/kubernetes-sigs/jobset/releases/download/v0.7.0/manifests.yaml
5. kubectl apply -f js.yaml
6. Coordinator field is nil: kubectl get jobsets js -o yaml | grep coordinator

Anything else we need to know?:

• If I change the field in the spec to coordinatorz, apply fails: strict decoding error: unknown field "spec.coordinatoz"
• If I change the coordinator.replicatedJob field to def-not-exists, the apply succeeds. So I think the coordinator field is getting set to nil most likely in the CRD conversion webhook and before the validation webhook

Environment:

• Kubernetes version (use kubectl version):

Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.1

• JobSet version (use git describe --tags --dirty --always): v0.4.0 and v0.7.0
• Cloud provider or hardware configuration: kind
• Install tools: kind
• Others:

[kannon92]

cc @ahg-g.

[avrittrohwer]

After some more testing, the issue is only reproducible in the kind cluster when I submit the jobset before the v0.4.0 manager pod is deleted by the deployment which makes sense why it would behave that way. I originally saw this in a GKE cluster and used kind as a minimal repro, I will keep investigating in the GKE cluster

[avrittrohwer]

The issue is that kueue was also installed in my cluster

Kueue v0.9.0 is the first version with a copy of jobset API with coordinator field: https://github.com/kubernetes-sigs/kueue/blob/release-0.9/vendor/sigs.k8s.io/jobset/api/jobset/v1alpha2/jobset_types.go

v0.8.0 does not have the field: https://github.com/kubernetes-sigs/kueue/blob/release-0.8/vendor/sigs.k8s.io/jobset/api/jobset/v1alpha2/jobset_types.go

[ahg-g]

Do you want to open an issue in the Kueue repo to update their JobSet definition?