ws-manager: fix event workers hang forever

[jenting]

Fix workers hang if there are over 100 VolumeSnapshot is ready, and we restart the ws-manager.
The m.notifyPod channel is no receiver, and it causes the 100 event workers to hang forever.
So, the ws-manager can't handle any workspace pod event and volume snapshot event changes.

Description

Fix event workers hang if there are 100+ VolumeSnapshot ready and ws-manager restarts.

Related Issue(s)

Fixes #13007

How to test

• Prepare Pod yaml manifest and save it as pod.yaml.

  apiVersion: v1
  kind: Pod
  metadata:
    name: test
  spec:
    containers:
    - name: test
      image: alpine:latest
      volumeMounts:
      - name: volv
        mountPath: /data
    volumes:
    - name: volv
      persistentVolumeClaim:
        claimName: test

• Prepare PVC yaml manifest and save it as pvc.yaml.

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: test
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: rook-ceph-block
    resources:
      requests:
        storage: 1Mi

• Prepare VolumeSnapshot yaml manifest and save it as vs-0.yaml.

  apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshot
  metadata:
    name: test-0
    annotations:
      gitpod/id: test-0
  spec:
    volumeSnapshotClassName: csi-rbdplugin-snapclass
    source:
      persistentVolumeClaimName: test

• Prepare 100 VolumeSnapshot yaml manifest.

  for i in {1..100}; do cp vs-0.yaml vs-${i}.yaml; sed -i "s/-0/-${i}/g" vs-${i}.yaml; done

• Prepare Pod and PVC.

  kubectl create -f pod.yaml -f pvc.yaml

• Wait the Pod is running and PVC is bound.

• Prepare 100 VolumeSnapshots.

  for i in {1..100}; do kubectl apply -f vs-${i}.yaml; done

• Waits for all the 100 VolumeSnapshots become ready.

  kubectl get vs | grep true | wc -l

• Restart ws-manager

  kubectl rollout restart deploy ws-manager

• Make sure the workspace can start.

Release Notes

None

Documentation

None

Werft options:

• /werft with-preview

[jenting]

/werft run with-preview

👍 started the job as gitpod-build-jenting-fix-ws-manager-workers-hang.2
(with .werft/ from main)

[jenting]

/werft run with-preview

👍 started the job as gitpod-build-jenting-fix-ws-manager-workers-hang.5
(with .werft/ from main)