Improvements/ideas for GPU support

[jasonmads]

I'm not 100% how to word this, but here it goes.

Looking at #924 I have these ideas.

GPU hosts don't pick up non-GPU frames: If a host has CPU and GPU it should be able to run CPU and GPU frames regardless of type, with possible preference to GPU. Additionally some frame types may be able to utilize CPUs and GPUs,

GPU jobs are not like CPU jobs: OS will migrate CPU jobs to idle CPUs as needed. GPU jobs are usually assigned to 1 or more GPUs at startup and they continue to run on only those GPS until finished. I propose something similar to how a frame number passed to the command for GPU assignment. This would be managed by rqd. rqd only needs to tell cuebot number of free GPUs.

Are these bad ideas? Are there better ideas?

[bcipriano]

@splhack @IdrisMiles @larsbijl Any thoughts?

[splhack]

GPU hosts don't pick up non-GPU frames

True. Because Cuebot has the logic to avoid scheduling CPU-only job onto GPU hosts. This totally makes sense, because, theoretically, GPU host can be saturated by only CPU jobs if Cuebot doesn't have the logic. It is wasting GPU.

So the recommendation would be, create dedicated facility for GPU to schedule GPU job to GPU hosts, create another dedicated facility for CPU to schedule CPU-only job to CPU hosts.

--

I propose something similar to how a frame number passed to the command for GPU assignment.

Could you elaborate on the detail?

RQD already tells Cuebot number of free GPU units.

[jasonmads]

Ah, I see the reasoning for GPU only jobs on GPU host. If there was a need for CPU booking it could be arranged manually.

For GPU assignment, from what I understand, RQD keeps track of CPU cores and book jobs to cores. RQD uses taskset to do this. Similar functionality can be done with CUDA_VISIBLE_DEVICES environment variable for GPUs.

There is also the issue of CPUs to GPUs, how or should that be controlled. If we are just treating GPU machine as GPU only, then CPU issues can become an issue.
Example
20CPU 4GPU machine
First job take 10CPU 1 GPU
second job takes 10CPU 1GPU
no more CPUs, 2 GPU idle

Should CPUs be divided up per GPU? Each GPU would get 5CPU all the time.
or is it up to the admins to setup when jobs launch? Setting the 5CPU for GPU jobs, but if you have a machine with fewer or more CPU per GPU would create problems.

Edit: Also thinking some software takes a option for which GPUs to run on. This could be passed with some token %GPU_IDX% or something.

[splhack]

First job take 10CPU 1 GPU

What is the reason of why the job takes 10 CPU cores in this example?

--
In the current Cuebot implementation, the default reservation number of CPU cores per frame is 2. (I don't know why though.)

OpenCue/cuebot/src/main/java/com/imageworks/spcue/VirtualProc.java

Lines 142 to 143 in c22fe12

	&& proc.coresReserved <= 200) {
	proc.coresReserved = 200;

So, back to the example, 20 CPU cores and 4 GPU units machine can run 4 sets of GPU job frames (2 CPU cores and 1 GPU unit x 4 = 8 CPU cores and 4 GPU units total) concurrently, as long as you didn't set explicitly the number of CPU cores or memory reservation.

To repeat, the current Cuebot implementation implicitly expects GPU machine should have twice as many CPU cores as GPU units.

--

Also thinking some software takes a option for which GPUs to run on

We can use these environment variables in the rendering job.

• CUE_GPU_CORES
  • Comma separated GPU unit index. E.g. 1, 1,2,3, 2,4,6,7, ...
• CUE_GPUS
  • The number of reserved GPU units. E.g. 1, 2, ...

[jasonmads]

First job take 10CPU 1 GPU

What is the reason of why the job takes 10 CPU cores in this example?

Just an example of using CPUs before GPUs. I think every GPU should be evenly assigned CPUs. There could situations where this isn't ideal, but the GPU is the main resource that should be considered.

In the current Cuebot implementation, the default reservation number of CPU cores per frame is 2. (I don't know why though.)

I think this has to do with hyper threading, where each hyper thread is seen as 2 cores to the system but actually just 1 physical core.

So, back to the example, 20 CPU cores and 4 GPU units machine can run 4 sets of GPU job frames (2 CPU cores and 1 GPU unit x 4 = 8 CPU cores and 4 GPU units total) concurrently, as long as you didn't set explicitly the number of CPU cores or memory reservation.

To repeat, the current Cuebot implementation implicitly expects GPU machine should have twice as many CPU cores as GPU units.

Why not use all the CPUs as I mention above. The tasks I am running are helped by CPU for initial data processing before the data is sent to the GPU. Additionally some tasks can benefit from more than one GPU, how many CPUs are then also assigned?

Also thinking some software takes a option for which GPUs to run on

We can use these environment variables in the rendering job.

• CUE_GPU_CORES

  • Comma separated GPU unit index. E.g. 1, 1,2,3, 2,4,6,7, ...

• CUE_GPUS

  • The number of reserved GPU units. E.g. 1, 2, ...

If something like
command --gpus $CUE_GPUS
works I think that is fine, no need for a %GPU_IDX% token.

[thunders82]

GPU hosts don't pick up non-GPU frames

True. Because Cuebot has the logic to avoid scheduling CPU-only job onto GPU hosts. This totally makes sense, because, theoretically, GPU host can be saturated by only CPU jobs if Cuebot doesn't have the logic. It is wasting GPU.

Hi @splhack ,
I'm having a similar logic as @jasonmads.

We have specific tasks requiring intensive CPU and GPU, meaning we have nodes with multiple CPU and GPU. Currently, if we have no job requiring a GPU on the farm then we'll waste those CPUs.

Could this be a possible behaviour to implement ?
Prio to GPU jobs on GPU nodes :
In the event of sending a GPU job on the farm and the GPU node is used by a CPU only job --> Kill the CPU only job and retry it on another machine to give priority to the GPU job.

Edit : We can only assign CPU job on nodes with GPU if there is no GPU task pending (not assigned) on the farm.

This way we don't waste resources. What do you think ?

Thanks

[splhack]

@thunders82
recommend to speak it up in #1001
(just to clarify, the GPU scheduling logic (avoiding CPU job on GPU host) was already there when I revised the GPU support.)

[thunders82]

Thank you for point it out @splhack !