Add full process info.

[bethune-bryant]

Fixes #50

I added -f, --show-full-cmd that show the full process info as discussed in #50.

Right now it shows the percent of CPU usage and the percent of system memory in use, but that can be changed.

Let me know what you think.

Example:

server1  Wed Jun 26 15:24:33 2019  418.67
[0] Tesla V100-SXM2-16GB | 34'C,  23 % |  1097 / 16130 MB | user1(1087M)
 └─ 72041 ( 80%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[1] Tesla V100-SXM2-16GB | 36'C,  23 % |  1097 / 16130 MB | user1(1087M)
 └─ 72042 ( 80%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[2] Tesla V100-SXM2-16GB | 35'C, 100 % |  2130 / 16130 MB | user2(777M) user1(1343M)
 ├─ 95638 (100%,  0.16%): /mnt/home/user2/anaconda3/envs/env/bin/python test_c10d.py
 └─ 72043 (100%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py
[3] Tesla V100-SXM2-16GB | 34'C,  22 % |  1097 / 16130 MB | user1(1087M)
 └─ 72044 ( 40%,  0.24%): python /mnt/home/user1/git/horovod/examples/keras_mnist.py

[wookayin]

Thanks for your contribution. After all CI tests fixed, I'll have a look to merge this. BTW I am debating what would be the best command line flag (rather than -f): one option that comes across is -C (similar to -c), but not fully convinced myself.

[bethune-bryant]

Thanks @wookayin

After all CI tests fixed,

Yeah, the unicode characters aren't behaving nicely with Python 2.7. I'm trying to figure that out.

BTW I am debating what would be the best command line flag (rather than -f): one option that comes across is -C (similar to -c), but not fully convinced myself.

I don't have strong feelings one way or another on this -C makes sense.

[bethune-bryant]

@wookayin
All the tests are passing. I had to add a python 2 specific line.

Let me know what you decide about the flag and I can make that change.

[Stonesjtu]

Just FYI.
python2.7 is retiring in Jan 1st 2020, so probably the master branch can focus on only python 3.
Ref. https://pythonclock.org/

[bethune-bryant]

Just FYI.
python2.7 is retiring in Jan 1st 2020, so probably the master branch can focus on only python 3.
Ref. https://pythonclock.org/

That makes sense to me. That would definitely simplify these changes.

[Stonesjtu]

@wookayin
According to https://stackoverflow.com/questions/40114100/uploading-different-versions-python-2-7-vs-3-5-to-pypi, we can simply add python_requires='>=3.4' and bump the version.
Then if you install the package via pip2, the out-dated one instead of the newest version will be selected.

[bethune-bryant]

I opened an issue (#66) and can make the Python 2 changes separately before we merge these changes.

[wookayin]

The current master has dropped python2 -- would you want to rebase onto master and drop changesets for python2 support if necessary?

[bethune-bryant]

@wookayin
Yeah, I'll make another commit to remove the python2 specific changes.

[bethune-bryant]

@wookayin
I rebased this PR and removed the python 2 specific changes.

Let me know if there's anything else you'd like me to change.

[wookayin]

I love this feature, and thanks for your contribution.

Other than the line-by-line comments, there is one more thing we need to resolve: it's slow. cpu_percent() needs 0.1 seconds of blocking sampling per process; in a case where we have 64+ processes it will take many seconds. If we can cache and reuse the Process instances then it can be non-blocking via interval=0 (for gpustat -i cases), but not sure for one-time run without loops. Perhaps we can also parallelize using async/await features.

[bethune-bryant]

If we can cache and reuse the Process instances then it can be non-blocking via interval=0 (for gpustat -i cases), but not sure for one-time run without loops. Perhaps we can also parallelize using async/await features.

I will spend some time figuring out the best way to do this.

[bethune-bryant]

@wookayin
I made changes to improve the performance if the cpu usage checking, and it works really well on python 3.7. Apparently the asyncio stuff has changed alot throughout 3.4-3.7, so I'll need to look into it more.

[wookayin]

Looks cool. Two comments:

• In light of getting process information, the gpustat object now requires a context (e.g. pid and process) so it doesn't make any more sense to have a static initializer retrieve all the necessary information (e.g. you had to use static global variables which is not good). But I think this is a matter of refactoring I should take care of -- so let's merge it soon and let me work on this. It is also related to Improve speed by sparingly querying nvml #61.

• Can we make sure asyncio features work well throughout python 3.4-3.7? Or how can we test it? As it requires a bit complicated mocking so writing a test seems to be a bit tricky. But it'd be great to have a minimal test; later on I will refactor test cases using smarter mocking libraries (mockito).

[bethune-bryant]

@wookayin
What do you think? Is there anything else you'd like me to change before we merge?

[wookayin]

Looks good to me, thanks!

[wookayin]

Let's get this merged. I'll add a bit of follow-up commits and cleanups. Thanks a lot for your hard work!

[wookayin]

I made a small changes in a9c27f3.