[Windows] psutil.net_io_counters I/O is limited to 32bit unsigned integer because of MIB_IFROW

[jomann09]

Interesting problem I have seen for a while with using psutil. I'm by no means an expert with this but it is strange that when I approach 4 billion bytes it seems to rollover... I thought Python accounted for 32-bit integers and whatnot. Can someone explain if this can be accounted for?

What I'm using:
Python 2.7.11 (tried both 32 and 64 bit)
Windows 10 64 bit
psutil 4.1.0

Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

>>> import psutil
>>> psutil.net_io_counters(pernic=True)
>>> {'Ethernet 4': snetio(bytes_sent=378813079, bytes_recv=838898995, packets_sent=4236049, packets_recv=20927779, errin=0, errout=0, dropin=212, dropout=0)}

Is this a product of Windows? Are the counters only a certain size? Because in my windows ethernet status screen under 'Activity' it shows:

Bytes sent: 337,887,481
Byes recieved: 30,797,026,512

While the bytes_sent is off it doesn't surprise me that there may be small differences but the bytes_recv about a hour ago showed a value of around 3,400,000,000 bytes and now it is only showing in the 800 millions.

[jomann09]

Okay I did a little digging and I am wondering if we can work around this;

Here is the code in psutil I was looking at:
https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_windows.c#L2188

for windows the net IO counters are coming from a MIB, MIB_IFROW which uses dword (apparently 32-bit unsigned integer with a max of 4.3bil decimal) and since I haven't written anything in C in a long time I am not as familiar with all this. However, I do believe that there are other MIBs available with the values in larger types.

The structure of MIB_IF_ROW2 seems to support much larger numbers and in Microsofts documentation it does say to use this one if you have a NIC > x amount.

Would it be possible to check for the existence of the MIB_IF_ROW2 and use that, and if it doesn't exist fall back on the MIB_IFROW since it looks like the updated MIB_IF_ROW2 supports Windows Vista + but I do believe this would give the proper value for the output of octets/bytes for interfaces with > 4.3GB counters.

Edit: A little more digging, here is GetIfEntry2 which is basically equivalent to GetIfEntry but for MIB_IF_ROW2.

[jomann09]

Tried doing a little more with this but since I haven't done any major C programming I could not figure out how to get the MIB_IF_ROW2 struct to even be defined though I believe it's supposed to be in the same library as MIB_IFROW. However, I did take a look into past commits for this type of issue and it looks like in psutil 2.1.1 the issue was brought up but the fix in 50bd135 does not help since it's the value coming from the MIB that is wrapping, not what is getting sent to Python.

[giampaolo]

It occurred to me there was a PR for using MIB_IF_ROW2 that has been languishing for 2 years:
#554
What's missing is the proper ifdef condition to make sure that the old code keeps being executed in case the MIB_IF_ROW2 struct is not available (namely: on Windows XP).
If you have a Win XP available feel free to take over the PR and fix it if you can. That would be great actually. I have no Windows XP so that's why I never couldn't get around to it.
The best thing would be to use a reliable conditional to figure out whether MIB_IF_ROW2 is available but if you can't figure out how to do that also using a check based on the Windows version could be OK (not sure when exactly MIB_IF_ROW2 was introduced so some Googling is necessary).
This is how you can conditionally test the Windows version:

~/svn/psutil/psutil$ grep -r WIN32 .
./_psutil_windows.c:#define WIN32_LEAN_AND_MEAN
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0600)  // Windows Vista
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0600)  // Windows Vista
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0501)  // Windows XP with SP2
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0501)  // Windows XP with SP2
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0501)  // Windows XP with SP2
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0600)  // Windows Vista
./_psutil_windows.c:#if (_WIN32_WINNT >= 0x0600)  // Windows Vista
./arch/windows/services.c:        // SERVICE_WIN32)
./arch/windows/services.c:            SERVICE_WIN32,  // XXX - extend this to include drivers etc.?

[jomann09]

Ahh okay great, thanks! I actually somehow completely missed that pull request. Those are almost the exact changes I had made. I believe I have a Windows XP box, actually, so I will try to test this out too along with trying a workaround for XP.

[giampaolo]

Yeah sorry I couldn't get to signal you that earlier. It would have saved you a lot of troubles.

[jomann09]

No problem, I don't mind learning new things! Is there any particular reason to keep XP / Server 2003 support though? Windows XP has been officially EOL for over 2 years now, anyone running anything in it could just use psutil < 4.3.0 or whatever release it'd be in. I'll still try and test it, I have both Server 2003 and XP for testing but it may take a little while.

[giampaolo]

Yeah, I've been thinking about that, also because Win XP support was dropped in Python 3.3. Let's just say that if it's not too much effort (see psutil currently works on Win XP - and I don't know) I don't think it would be worth it to drop support for such a small change.
If on the other hand support is already broken I might think about that.

[jomann09]

I've got it working... Actually have to change the output to read with K not k (for Python unisgned long 64bit)

>>> ifs = psutil.net_io_counters(pernic=True)
>>> ifs['Local Area Connection']
snetio(bytes_sent=141287396L, bytes_recv=6234335854L, packets_sent=1847007L, packets_recv=4211775L, errin=0L, errout=0L, dropin=0L, dropout=0L)

Before I even continue I'd like to mention that the reason I couldn't get it working before was that #include <ws2tcpip.h> must come before #include <iphlpapi.h> and when you do that some of the mingw32 network connection fixes (specifically structs starting on lines 88, 99, 131, 138) cause duplicate errors. If you don't move the header though, MIB_IF_ROW2 is never defined. Can I get a confirmation that MinGW is not supported for building anymore? I know the build docs said something about no longer supporting it.

[jomann09]

On a fresh Windows XP (pro) trying to install 4.2.0 works although running it causes an issue since there's a DLL error. Should we fix this? I'm assuming it's because ws2tcpip.h got added along the way and/or one of the other headers have something that isn't available in XP.

One more edit: After testing a bit further it looks like 3.4.2 is the last version of psutil to work on XP.

[jomann09]

Closing this since you merged in #835