Migrate release.ci.jenkins.io to arm64 (controller only)

[dduportal]

This issue is a follow-up of #3823 which goal is to track the operations to migrate the release.ci.jenkins.io controller to a linux/arm64 system in order to decrease the operational cost.

Requirements:

• Ensure there is a dedicated arm64 nodepool only for this controller (VM and subnet isolation)

• Migrate release.ci data volume from LRS to ZRS (pvc-ed6adfb7-5a25-4315-af8a-627c4913863d)

  • to avoid timeout add those annotations in kubernetes-management/config/jenkins_release.ci.jenkins.io.yaml => Faster restart and upgrade: decrease the attach/mount times (ref. https://github.com/jenkinsci/helm-charts/tree/main/charts/jenkins#long-volume-attachmount-times) by adding those annotations in kubernetes-management/config/jenkins_release.ci.jenkins.io.yaml: feat(privatek8s/release.ci) add podSecurityContextOverride to avoid long volume attach/mount times kubernetes-management#5136

    controller:
      podSecurityContextOverride:
        runAsUser: 1000
        runAsNonRoot: true
        supplementalGroups: [1000]

  • Set release.ci to shutdown mode

  • Set the statefulset to zero replicas to ensure a proper and controlled volume unmount

  • create a snapshot (from UI) named jenkins-releaseci-snapshot-<datetime>

  • create a disk from the snapshot (from UI) named jenkins-releaseci-snap

  • create a PV with a yaml file with the above created disk :

    apiVersion: v1
    kind: PersistentVolume
    metadata:
      annotations:
        pv.kubernetes.io/provisioned-by: disk.csi.azure.com
      name: jenkins-releaseci-snap
    spec:
      capacity:
        storage: 64Gi
      accessModes:
        - ReadWriteOnce
      persistentVolumeReclaimPolicy: Retain
      storageClassName: managed-csi-premium-zrs-retain
      csi:
        driver: disk.csi.azure.com
        volumeHandle: jenkins-releaseci-snap
        volumeAttributes:
          fsType: ext4

    note that the volumeHandle is shorten to the diskname not the full handle volumeHandle: jenkins-releaseci-snap

    kubectl apply -f  .tmp/jenkins-releaseci-pv-snap.yaml

  • create a PVC

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: jenkins-releaseci-snap
      namespace: jenkins-infra
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 64Gi
      volumeName: jenkins-releaseci-snap
      storageClassName: managed-csi-premium-zrs-retain

    kubectl apply -f  .tmp/jenkins-releaseci-pvc-snap.yaml

  • Change the storage class with a datasource as per top of the file https://github.com/jenkins-infra/kubernetes-management/pull/4720/files (DO NOT MIGRATE TO ARM64 YET) - feat(privatek8s/release.ci): migrate to ZRS volume and extend to 64Gi kubernetes-management#5138

    • note: As PVC fields are mostly immutable fields we need to ensure PVC is deleted before applying.

  • Merge the PR (infra.ci will handle the migration)

• Before ARM64 migration, prepare a test job to be used to check the agent spawning and to be a test indicator

  • Details:
    • having a pipeline running a mvn -v on each kind of agent from podtemplate.d : chore(release/agent-health): add an infra test pipeline to check agents health release#525

• Once the above requirements tasks are finished, then migrate to ARM64 with the same method used for infra.ci.jenkins.io (in feat(publick8s): migration of weekly.ci to ARM64 with Volume migration kubernetes-management#4720) by adding the proper couple of nodeSelector and tolerations. chore(release.ci): migrate to arm64 kubernetes-management#5140

---

Definition of done

• the volume is a ZRS volume in the azure UI
• Delete the temporarily and manually created disk volume and PVC (claim) also the azure snapshot
• the pod start and is on an ARM64 node
• the jenkins CI release.ci.jenkins.io is responding
• trying to launch an agent from the release.ci (need to prepare a test job)
• check that the build queue is being handled

Originally posted by @smerle33 in #3823 (comment)