Investigate if possible to improve PVC mount time

[sagor999]

Currently mounting PVC takes about 15 seconds, while mounting same disk to VM via gcloud command about 3 seconds.

Investigate why there is a discrepancy and if there is a way to improve csi driver controller to reduce that time.

[sagor999]

I was able to reduce PVC attach time from 9s to 4s (which is about the same as running gcloud commands directly).
It involved changing two lines of code in gcp-csi-driver:
https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/25be424af1abfe1f9ba5ce68f0562aedf0c98bdc/pkg/gce-cloud-provider/compute/gce-compute.go#L781
changing polling here from 5s to 1s

https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/blob/25be424af1abfe1f9ba5ce68f0562aedf0c98bdc/pkg/gce-cloud-provider/compute/gce-compute.go#L742
Changing polling here from 3s to 1s

I will now work on creating PR upstream to allow specifying those poll times via command line arguments.

[princerachit]

Maybe we can use a backoff strategy instead of constant 1s as it might introduce unnecessary load?

[sagor999]

https://cloud.google.com/compute/docs/api-rate-limits
GCP has a global API rate limit per project:

Rate per project (ListRequestsPerMinutePerProject): 25 requests/second (maximum of 1500 requests/minute)

Since our clusters are all in the same project, I can easily see us hitting that rate limit.
Though we can request an increase for this quota due to unique nature of gitpod and the fact that we do need to mount PVC, or create disk\snapshot as fast as possible.

[sagor999]

kubernetes-sigs/gcp-compute-persistent-disk-csi-driver#956

[sagor999]

PR was merged into upstream.
DD was updated with findings.
Closing this issue as completed now.