Properly set number of GPUs in dask-sidecar

[sanderegg]

Currently the dask-sidecar management of GPUs on the underlying machine relies on defaults.

• on start it counts the number of available GPUs, relying on running a container nvidia-smi --list-gpus,
• it reports the number of available GPUs to the dask-scheduler,
• when a service requiring GPUs is started, it has to be inheriting an nvidia-based container so that it actually requires GPUs and gets by default all of them assigned to the container

This works as is but generates issues:

• inconsistent with how CPUs/RAM is assigned to the containers,
• in case of sharing a mutli-GPU machine with multiple single GPU container it would fail as all the container would get all GPUs assigned,

This shall be modified by doing the following changes:

• DV-2/Clusters-keeper: the assigned GPU resources in case of using the full machine should be set correctly
• Dask-sidecar: assign the correct amount of GPUs to the container:

"HostConfig": {
            "DeviceRequests": [
                {
                    "Driver": "nvidia",
                    "count": int(num_GPUs),
                    "Capabilities": [["gpu"]],
                }
            ],
        }