Support GPU similar to CPU

[larsbijl]

Is your feature request related to a problem? Please describe.

CPU (cores) are first-class citizens in Host but not GPU's. Add support for GPU core count?

Describe the solution you'd like

We have machines with 8 GPU's in them. it would be nice to be able to specify a service with GPU cores and have similar to taskset affinity to the GPU's.

[donalm]

Hey @larsbijl how are you managing those 8-GPU nodes in production currently?

[larsbijl]

We implemented GPU similar to CPU in our version of the opencue scheduler.

[donalm]

Ah - are you maintaining that as a private fork?

[larsbijl]

yes, we made many changes to the UI for our facilities workflow which wouldn't be relevant to others, which are mixed into the same repo. Though we are open to contributing this change back when we get time to do so.

[donalm]

Yeah please consider contributing the GPU patch. If an alternative approach gets merged you might have significant work to do to stay in sync with upstream.

[donalm]

Hey @larsbijl we have some time to work on this feature now - are you interested in contributing your code at this time? Would we be able to help you to prepare it to get merged?

If not, would you like to offer us some guidance on how you approached this before we dig into it ourselves?

[larsbijl]

Hey @donalm, I will make time this weekend to put in a MR with our implementation.

it will be missing the migration (never wrote one) and some of the front end implementation as that had diverged from our local branch.

[donalm]

That'd be amazing - thanks @larsbijl !

[larsbijl]

A little update on this. I have the majority of this ported over.

Some outstanding issues that I will need some help on.

1. for simplicity I modified the Initial migration to incorporate all the changed needed in both the tables, functions and triggers.
   To keep backward compatibility for users it will need to make it into a migration.

2. Our cuegui and rqd have diverged too much for easy merge. I've ported what I can, but it will likely be missing elements.

3. We don't use windows, the GPU RQD side uses nvidia-smi directly. we will want to find an OS-agnostic method.

4. tests. we will definitely need to write some tests.