Add multiple GPU support #760

[splhack]

Plan

• Keeping original code and order as much as possible. This pull request concentrates on adding multiple GPU support. No intention of doing cosmetic changes.

  • Keeping naming convention. Examples
    • cores -> gpus
    • coreUnits -> gpuUnits
    • memory -> gpuMemory
    • mem_min -> gpu_mem_min
  • The order is vary from code to code, keeping it.
    • memory, gpu, cores -> memory, gpuMemory, cores, gpus
    • cores, memory, gpu -> cores, memory, gpus, gpuMemory
  • Place similar methods close to each other
    • cores-gpus, memory-gpuMemory

• This is the first iteration/attempt of shipping multiple GPU support. It may not cover all the use-cases.

  • First target is symmetric set of GPUs. Assuming all GPUs are same on one host.
    • A host can have multiple variety of types of GPUs. Multiple vendors, different amount of memory from GPU unit to GPU unit, etc, etc. Not supported.
  • The amount of GPU memory.
    • GPU doesn't share memory with the other GPU unit. So, 8 units of GPU with 16GiB doesn't mean the host has linear 128GiB GPU memory. Scheduling with combined both minimum GPU units and minimum GPU memory may not work well.

Commits

0. Add GetDefault and SetDefault to AllcationInterface #939 and Fix PyOutline test #942

   • To fix CI and avoid VERSION.in conflict

1. Add DB migration for supporting multiple GPU

   • Create V10 migration sql from feat: Add multiple GPU support #760
   • Changes from feat: Add multiple GPU support #760
     • Add more GPU memory stats
     • Use BIGINT for memory
     • Clean up unrelated changes
     • Create new indexes
   • Code review helper
     • Original feat: Add multiple GPU support #760 V1 diff https://gist.github.com/splhack/fab852153505c65742909917888c51bc
     • V9 vs V10 pg_dump https://github.com/splhack/OpenCue/pull/1/files

2. Sync with V10 migration

   • Rename
     • int_gpu_free -> int_gpu_mem_free
     • int_gpu_min -> int_gpu_mem_min
     • int_gpu_reserved -> int_gpu_mem_reserved
     • int_gpu_total -> int_gpu_mem_total
     • int_gpu_idle -> int_gpu_mem_idle
     • int_gpu_max -> int_gpu_mem_max
   • gradlew test passed

3. Bump minor version number

4. Add Job Spec DTD 1.12

   • Diff from 1.11 https://gist.github.com/splhack/565df692e0320693d60b2b15e7c9b920

5. Update proto files for multiple GPU support

   • Keep backward compatibility
     • Rename message fields (gpu -> gpu_memory) since it doesn't break binary compatibility
     • Keep rpc (SetMinGpu) but mark as deprecated

6. Sync with proto changes

   • Add warning of Job spec 'gpu's
   • gradlew test passed

7. Replace gpu with gpus and gpu_memory

   • Add counterpart logic, gpus for cores, gpuMemory for memory
   • gradlew test passed

8. [RQD] Support multiple GPUs with nvidia-smi

9. [PyOutline] Support gpus and gpu_memory

   • until 1.11
     • PyOutline has not been supporting gpu
   • from 1.12
     • gpu_memory = The amount of GPU memory
     • gpus = The number of GPU units

10. [PyCue] Support gpus and gpu_memory

11. [cuegui] Sync with proto changes

    • Minimum changes to pass python setup.py test

[splhack]

I found an issue of the inconsistency of gpu usage in #760. User may find difficult to track issue due to this.

• until DTD 1.11
  • gpu = The amount of GPU memory
• from DTD 1.12
  • gpu = The number of GPUs

Possible scenario:

1. User has gpu in their Job PyOutline script.
2. They updates PyOutline, which means user starts to use DTD 1.12.
3. The meaning of gpu was changed from The amount of GPU memory to The number of GPUs without warning or anything.

So, from DTD 1.12,

• gpu = The amount of GPU memory
  • For the backward compatibility. Cuebot will migrate it to gpu_memory internally.
• gpu_memory = The amount of GPU memory
• gpus = The number of GPUs

1. User has gpu in their Job PyOutline script.
2. They updates PyOutline, which means user starts to use DTD 1.12.
3. PyOutline warns gpu usage and generate XML with gpu_memory

Also I will rename gpu to gpu_mem or gpus in V10 migration and Cuebot codes. It will help catching unintentional errors earlier.

[splhack]

OpenCue uses 100 = 1 physical CPU core.
Probably we should use the same way for GPU.
It's not so common that multiple processes share one GPU core, but it's not impossible. Like NVIDIA MPS.
Apparently not good idea, fractional allocation is not supported.

OpenCue/rqd/rqd/rqmachine.py

Lines 637 to 639 in 58c81fb

	if reservedCores % 100:
	log.debug('Taskset: Can not reserveHT with fractional cores')
	return None

[splhack]

@larsbijl @bcipriano I think we can start review technical design and code since, at least all test passed.

[larsbijl]

self.min_gpu.setValue(service.data.min_gpu // 1024)
AttributeError: min_gpu

in the services menu of cuegui

[splhack]

will update (migration V10 -> V11, VERSION 0.9 -> 0.10) when #936 merged

[splhack]

@bcipriano @larsbijl
Is there anything I can do to help merging this pull request?

[bcipriano]

Did one last review, changes LGTM.