Adding GPU devices to the OCP prod cluster

[Milstein]

• We need to setup few GPU devices to the NERC OCP prod cluster for course and other workload. Also need to understand if MIG can possible in this setup.

[joachimweyl]

@Milstein is this now resolved?

[jtriley]

The 2x V100 nodes have been added to the cluster, however, I currently have scheduling disabled. There is a machineconfig required to properly configure networking on these hosts. This will cause all worker nodes to reboot so this needs to be done during a maintenance event. Planning to do this before the upgrade during the 9/27 maintenance.

[jtriley]

re: MIG there are issues with using MIG that I've mentioned previously related to setting quotas/limits and also we don't currently have MIG-supported devices in the cluster (e.g. A100s). That said, I believe we will have some new A100 hosts landing in the next few weeks/month.

[jtriley]

I'm also looking into an issue with argocd not applying resources currently due to long-standing sync issues. This is preventing the necessary operators required for GPU computing on the prod cluster from being installed (see OCP-on-NERC/nerc-ocp-config#277)

[jtriley]

I'm also looking into an issue with argocd not applying resources currently due to long-standing sync issues. This is preventing the necessary operators required for GPU computing on the prod cluster from being installed (see OCP-on-NERC/nerc-ocp-config#277)

Turns out this was a misconfiguration on my part that should be resolved by OCP-on-NERC/nerc-ocp-config#283

[joachimweyl]

@jtriley Was this resolved by OCP-on-NERC/nerc-ocp-config#283?

[jtriley]

I'm going to close this issue given that we added them and configured networking properly during our last maintenance. There is still some ongoing work to set the proper quotas for GPUs from coldfront/openshift-acct-mgt. I will have a PR out soon for that.