infra.ci.jenkins.io on arm64 (controller and agents)

[smerle33]

Service(s)

Azure, infra.ci.jenkins.io

Summary

as for the publick8s, we should create an ARM64 nodepool within the kubernetes privateK8S in order to migrate jenkins agent to arm64.

Also need to migrate infra.ci.jenkins.io to arm (with care about PV/PVC migration to the correct zone for arm zone 1)

• create an ARM64 nodepool
• create ARM64 agent in this nodepool
• migrate the infra.ci.jenkins.io instance in the arm64 nodepool

Reproduction steps

No response

[smerle33]

• done:
  • [use ARM64 agent on https://github.com/jenkins-infra/kubernetes-management/tree/main] provide the same tools as update on the allinone image (https://github.com/jenkins-infra/packer-images)
  • https://github.com/jenkins-infra/packer-images/releases/tag/1.44.0
  • [use ARM64 agent on https://github.com/jenkins-infra/kubernetes-management/tree/main] modify the pipeline of https://github.com/jenkins-infra/kubernetes-management/tree/main to use the ARM64 allinone image instead of the docker-helmfile one. (jenkins-infra/kubernetes-management@2ea17f9)
• wip:
  • [use ARM64 agent on https://github.com/jenkins-infra/kubernetes-management/tree/main] cleanup jenkins-infra/docker-helmfile project (remove job, archive repository, update docker-hub as deprecated)
  • [use ARM64 agent on terraform usages] update shared pipeline library feat(terraform) Remove parameter agentContainerImage in favor of agentLabel pipeline-library#817 to use node label instead of podtemplate in order to switch to ARM64 allinone version
• todo:
  • lists of terraform repositories:
    • https://github.com/jenkins-infra/azure
    • https://github.com/jenkins-infra/azure-net
    • https://github.com/jenkins-infra/aws
    • https://github.com/jenkins-infra/digitalocean
    • https://github.com/jenkins-infra/fastly
    • https://github.com/jenkins-infra/cloudflare
    • https://github.com/jenkins-infra/datadog

[smerle33]

Plan to migrate infra.ci.jenkins.io to arm64
See post-mortem

• Migrate infra volume (pvc-23-redacted- managed-csi-premium-retain)
  • Create a new Storage class for ZRS (as feat(publick8s): add a storage class for ZRS azure#526 on publick8s)
  • Set infra.ci to shutdown mode
  • Set the statefulset to zero replicas before merging the PR below to ensure a proper and controlled volume unmount
  • Create MANUALLY a disk volume from a snapshot as a ZRS storage named jenkins-infraci-snap (will be used as source for cloning by the helmchart) https://github.com/jenkinsci/helm-charts/blob/8bfdd9dfe1ae7a5e45f2bba6c21dcb32b7cfbb43/charts/jenkins/values.yaml#L1206-L1210
  • Change the storage class with a datasource as per top of the file https://github.com/jenkins-infra/kubernetes-management/pull/4720/files (DO NOT MIGRATE TO ARM64 YET) - chore(infraci): move persistence volume to ZRS in prevision for ARM64 migration kubernetes-management#5116
  • Set back the statefulset
• Create a new nodepool dedicated to controllers with a taint (agents and controllers should be on the same nodepool) - feat(privatek8s): add a nodepool dedicated to jenkins controller azure#658
  ----> postpone until more statistic of usage are extracted
• once the above task will be merged, migrate to ARM64 as per https://github.com/jenkins-infra/kubernetes-management/pull/4720/files bottom of the file, add a node Selector specific to the previously created nodepool

++++++++++++++++++++++++++++
Definition of done

• the volume is a ZRS volume in the azure UI - Delete the temporarily and manually created disk volume and PVC (claim) also the azure snapshot
• the pod start and is on an ARM64 node
• the jenkins CI infra.ci.jenkins.io is responding

check to add :

• check that the build queue is being handled

---

Post-Mortem:

we need to provide a PVC and PV matching the source disk :

• create a snapshot (from UI)
• create a disk from the snapshot (from UI)
• create a PV with a yaml file with the above created disk :

apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: disk.csi.azure.com name: jenkins-infraci-snap spec: capacity: storage: 64Gi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain storageClassName: managed-csi-premium-zrs-retain csi: driver: disk.csi.azure.com volumeHandle: jenkins-infraci-snap volumeAttributes: fsType: ext4

note that the volumeHandle is shorten to the diskname not the full handle volumeHandle: jenkins-infraci-snap

• create a PVC

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-infraci-snap namespace: jenkins-infra spec: accessModes: - ReadWriteOnce resources: requests: storage: 64Gi volumeName: jenkins-infraci-snap storageClassName: managed-csi-premium-zrs-retain

• kubectl to create them :

kubectl apply -f  .tmp/jenkins-infra-pv-snap.yaml
kubectl apply -f  .tmp/jenkins-infra-pvc-snap.yaml

as it was getting a timeout we had to add this to the jenkins helmchart values :

controller:
  podSecurityContextOverride:
    runAsUser: 1000
    runAsNonRoot: true
    supplementalGroups: [1000]

The Migration part cannot be done by merging the PR as pretented above : - [ ] Change the storage class with a datasource as per top of the file https://github.com/jenkins-infra/kubernetes-management/pull/4720/files (DO NOT MIGRATE TO ARM64 YET) - https://github.com/jenkins-infra/kubernetes-management/pull/5116 as the infra.ci is down so it need to be executed locally :

gh pr checkout 5116
helmfile -f "clusters/privatek8s.yaml" diff --suppress-secrets --skip-deps --context=2 --concurrency=8 -l name=jenkins-infra
helmfile -f "clusters/privatek8s.yaml" apply --suppress-secrets --skip-deps --context=2 --concurrency=8 -l name=jenkins-infra

The PR still need to be merged not to introduced changes when infra will start successfully

[smerle33]

update: infra.ci is now officially running on arm64.

next steps: cleanup (next week)

[dduportal]

Update: fixing a few errors discovered after the controller migration:

• Datadog connection errors in infra.ci logs:
  • No agent were running on the node where infraci ran: fix(privatek8s/datadog) ensure datadog agent runs on all nodes by updating tolerations kubernetes-management#5132
  • Azure Linux support for datadog + admission controller refusing webhooks: fix(publick8s,privatek8s) ensure datadog setup works on AKS (Azure Linux, Admission Controller) kubernetes-management#5133
• Failure to spin up VM agents due to subnet changes: fix(infra.ci.jenkins.io) update NSG to allow arm64 controller to reach agents (new subnet) azure#665

=> Build queue is now empty \o/

[smerle33]

(edited): issue tracking the release.ci's migration to arm64: #4042

[smerle33]

CleanUp (infra.ci, weekly.ci, release.ci)

• remove PV/PVC from snap
• remove disk not mounted (from snap)
• remove snapshots