Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature request: multi GPU support with gpu and vram? #549

Open
3blake7 opened this issue Nov 14, 2019 · 4 comments
Open

feature request: multi GPU support with gpu and vram? #549

3blake7 opened this issue Nov 14, 2019 · 4 comments

Comments

@3blake7
Copy link

3blake7 commented Nov 14, 2019

I was wondering if you'd be willing to add multi GPU support with gpu and vram?

@marcinbogdanski
Copy link

Both Multi-GPU and VRAM support would be super amazing and the most useful thing ever.

@menoos
Copy link

menoos commented Sep 16, 2021

I see not much interest in this feature, but I'd love to have it too. Adding comment just to refresh it :)

@dcerisano
Copy link

dcerisano commented Aug 31, 2023

There is support for multi-gpu system (sort of) on linux with nvidia at least:

Edit gpu.sh, the script that polls the GPU:

nano ~/.local/share/gnome-shell/extensions/[email protected]/gpu.sh

Where you see nvidia-smi -i 0 ... change to your desired GPU (GPU 0 is the default).

No restart necessary, the change starts to work immediately (awesome).

Note: Since it is already using nvidia-smi, all info about all GPUs is already available. Should not be too hard to graph all GPUs at the same time (different color scheme for each). That would lead to a pretty cluttered graph, but it is do-able. Maybe a separate chart for each would be cool.

@dcerisano
Copy link

dcerisano commented Sep 5, 2023

I realized that the current gpu_usage.sh uses GPU0 as default which on a multi-gpu system can easily be the wrong one, depending on where you have your monitors plugged in. nvidia-smi always sets GPU0 as the first one detected (CUDA sets GPU0 as the fastest).

The vast majority of systems will have a single GPU so no need to arbitrarily pick GPU0, since only one will be returned form nvidia-smi. In the case of multi-gpu, aggregation of all GPUs can be used (exactly like the CPU multi-core monitor)

I rewrote the polling for NVidia GPUs and created a pull request.

This is a multi (1 to any) GPU stat aggregator that works exactly the same as current polling in the case of 1. In the case of a multi-GPU set up, the stats returned from nvidia-smi are added (aggregated) for each GPU and the last aggregate is returned.

Fully tested, works for 1 to any number of GPUs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants