psutil fails to build on Solaris 10

[grzn]

Installed /root/python/lib64/python2.7/site-packages/z3c.recipe.scripts-1.0.1-py2.7.egg
Searching for psutil>=2.0
Reading http://pypi.infinidat.com/simple/psutil/
Reading http://pypi.infinidat.com/simple/psutil/None
Best match: psutil 3.0.1
Downloading http://pypi.infinidat.com/media/dists/psutil-3.0.1.tar.gz#md5=e6592b4e240a85a17a66636d849759d6
Processing psutil-3.0.1.tar.gz
Writing /tmp/easy_install-eZ6nM5/psutil-3.0.1/setup.cfg
Running psutil-3.0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-eZ6nM5/psutil-3.0.1/egg-dist-tmp-w2Z3dl
warning: no previously-included files matching '*' found under directory 'docs/_build'
psutil/_psutil_sunos.c: In function ‘psutil_net_if_stats’:
psutil/_psutil_sunos.c:1230:44: error: ‘struct ifreq’ has no member named ‘ifr_mtu’
                                         ifr.ifr_mtu);
                                            ^
error: Setup script exited with error: command 'gcc' failed with exit status 1
script failed!

[wiggin15]

According to /usr/include/net/if.h on both Solaris 10 and 11:

/*
 * OBSOLETE: Replaced by struct lifreq. Supported for compatibility.
...
 */
struct  ifreq {

the ifreq struct doesn't have the mtu member, while the new non-obsolete lifreq struct does. I committed a fix to switch to lifreq here: Infinidat@e8c06de
However this is not all: after fixing this, the build still fails on Solaris 10 due to:

psutil/_psutil_posix.c:15:21: fatal error: ifaddrs.h: No such file or directory
 #include <ifaddrs.h>

(this file doesn't exist on Solaris 10 and also on AIX so it doesn't entirely belong to the "posix" module...) I think the only way to fix this problem is by #ifdef-ing out and not supporting the function that uses ifaddrs.h (psutil_net_if_addrs). What do you think?

[giampaolo]

Can you make a PR for Infinidat/psutil@e8c06de?

(this file doesn't exist on Solaris 10 and also on AIX so it
doesn't entirely belong to the "posix" module...) I think the
only way to fix this problem is by #ifdef-ing out and not
supporting the function that uses ifaddrs.h (psutil_net_if_addrs).
What do you think?

I think it makes sense. Can you make a PR for this as well?

[wiggin15]

The problem with disabling this function is that there is no way to tell Solaris 10 and 11 apart according to the predefined macros (see https://gist.github.com/basman/587684). My current commit to fix this is here: Infinidat@006d277
but that disables the function in Solaris 11 too.
Maybe I can check if ifaddrs.h exists in setup.py and pass a macro to the extension if it doesn't exist, but that will complicate the code even further. Do you think the commit above is good enough?

[giampaolo]

If by default there's no way to distinguish Solaris 10 from 11+ in C (are we 100% sure? it's weird) then I'd look into how to do this as a macro defined in setup.py. Infinidat@006d277 doesn't look good enough as it disables the functionality on all solaris versions.

[mpp4manu]

I may be out of my depth here, so feel free to point and laugh, but there is a Solaris C system call for 10 and 11 in sys/utsname.h.
Solaris 10:
http://docs.oracle.com/cd/E26505_01/html/816-5167/uname-2.html#scrolltoc
Solaris 11:
http://docs.oracle.com/cd/E23824_01/html/821-1463/uname-2.html#scrolltoc
Also, a colleague of mine "fixed" issues he found with _psutil_sunos.c. He says he'll document and push them up here after testing.

We built our Python 2.7.8 on Sun Studio 12 cc rather than gcc because gcc wouldn't build the Python libpython shared object, although it did build a viable Python executable.

[wiggin15]

Thanks for pointing it out, but the system call is not good enough because we need to distinguish between the two operating systems in compile time, not run time ('#include ifaddrs.h' is parsed during preprocessing).
Also, make sure the fixes to _psutil_sunos.c aren't already in the pull requests, e.g. here: #610

[mpp4manu]

Again, feel free to laugh, but wouldn’t the simple execution of a “uname –r” tell you? I’m not sufficiently knowledgeable about the compilation process to know whether or not this is possible. Sorry.

I’m not sure if the modifications have been pushed by my colleague yet. We’re still testing internally. The reason we made those changes was we could not get the psutil Python module to work properly on our Solaris 10 systems. For some reason, it couldn’t find the Solaris 10 network interfaces.

Michael Peoples (mp4783)
AT&T Network Operations
Network Operations Centers Planning
Office: +1 614-886-0923
Mobile: +1 614-886-0923

Principal Applications Developer
[email protected]:[email protected]

From: wiggin15 [mailto:[email protected]]
Sent: Friday, May 08, 2015 9:22 AM
To: giampaolo/psutil
Cc: PEOPLES, MICHAEL P
Subject: Re: [psutil] psutil fails to build on Solaris 10 (#609)

Thanks for pointing it out, but the system call is not good enough because we need to distinguish between the two operating systems in compile time, not run time ('#include ifaddrs.h' is parsed during preprocessing).
Also, make sure the fixes to _psutil_sunos.c aren't already in the pull requests, e.g. here: #610#610

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/609#issuecomment-100230816.

[wiggin15]

The compiler can't run commands like 'uname' - it's not a shell.
What we can do is use Python to determine the platform (e.g. using the 'platform' module) and then define macros accordingly before compilation. I did that in pull request #610, so it's not a problem any more.

[giampaolo]

This is supposed to have been fixed by #610.

[dmurphy18]

I think this may not have been fixed fully since I am seeing the original error ifr_mtu not existing and from examining the code I do not see anything which address's this issue in _psutil_posix.c. Using release-5.0.0 tag from github https://github.com/giampaolo/psutil

I am building on a Solaris 10 update 8 on Intel in a VirtualBox, but see the same issue on Sparc with similar level of OS, with current OpenCSW tools installed on both.

Command to reproduce (relevant excerpts from bash script):

export CFLAGS='-I/opt/local/include -I/opt/csw/includei -I/opt/local/include/python2.7'
export LDFLAGS=
export PKG_CONFIG_PATH='/opt/csw/lib/pkgconfig:/opt/local/lib/pkgconfig'
ln -s /opt/csw/include/openssl /opt/local/include/python2.7
ln -s /opt/csw/bin/gcc /usr/local/bin/ccATH='/opt/csw/lib/pkgconfig:/opt/local/lib/pkgconfig'

cd psutil
python -m pip install psutil  2>&1 | tee -a $LOGGING || {
    _error "$0:$FUNCNAME failed to install psutil, retcode '${$?}'";
}

Log output:

165     building 'psutil._psutil_sunos' extension
166     creating build/temp.solaris-2.10-i86pc.32bit-2.7
167     creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil
168     gcc -m32 -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/csw/includei -I/opt/local/include/python2.7 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=500 -DPSUTI    L_SUNOS=1 -I/opt/local/include/python2.7 -c psutil/_psutil_sunos.c -o build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_sunos.o
169     psutil/_psutil_sunos.c: In function ‘psutil_proc_name_and_args’:
170     psutil/_psutil_sunos.c:123:15: warning: unused variable ‘py_args’ [-Wunused-variable]
171          PyObject *py_args;
172                    ^
173     psutil/_psutil_sunos.c:122:15: warning: unused variable ‘py_name’ [-Wunused-variable]
174          PyObject *py_name;
175                    ^
176     gcc -m32 -shared -L/opt/local/lib -L/opt/csw/lib -I/opt/local/include -I/opt/csw/includei -I/opt/local/include/python2.7 build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psuti    l_sunos.o -lkstat -lnsl -lsocket -o build/lib.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_sunos.so
177     building 'psutil._psutil_posix' extension
178     creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch
179     creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch/solaris
180     creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch/solaris/v10
181     gcc -m32 -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/csw/includei -I/opt/local/include/python2.7 -fPIC -DPSUTIL_SUNOS10=1 -I/opt/local/include/pytho    n2.7 -c psutil/_psutil_posix.c -o build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_posix.o
182     psutil/_psutil_posix.c: In function ‘psutil_net_if_mtu’:
183     psutil/_psutil_posix.c:272:14: error: ‘struct ifreq’ has no member named ‘ifr_mtu’
184          **mtu = ifr.ifr_mtu;**
185                   ^
186     error: command 'gcc' failed with exit status 1
187 
188     ----------------------------------------
189 Command "/opt/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-p4Okvc/psutil/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().repla    ce('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-fGaBTl-record/install-record.txt --single-version-externally-managed --compile" failed     with error code 1 in /tmp/pip-build-p4Okvc/psutil/
190 + _error './tools/build_sol32_2016_11_1:_install_psutil failed to install psutil, retcode '\''15283'\'''
191 + msg='ERROR: ./tools/build_sol32_2016_11_1:_install_psutil failed to install psutil, retcode '\''15283'\'''
192 + echo 'ERROR: ./tools/build_sol32_2016_11_1:_install_psutil failed to install psutil, retcode '\''15283'\'''
193 ERROR: ./tools/build_sol32_2016_11_1:_install_psutil failed to install psutil, retcode '15283'

Wondering if I missed something, like should posix even being getting built, my environment is set to en_US.UTF-8.

[dmurphy18]

problem still occurs with fixed include form above, had typo,extra 'i' in -I/opt/csw/includei

corrected below:

    gcc -m32 -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/csw/include -I/opt/local/include/python2.7 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=500 -DPSUTIL_SUNOS=1 -I/opt/local/include/python2.7 -c psutil/_psutil_sunos.c -o build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_sunos.o
    psutil/_psutil_sunos.c: In function ‘psutil_proc_name_and_args’:
    psutil/_psutil_sunos.c:123:15: warning: unused variable ‘py_args’ [-Wunused-variable]
         PyObject *py_args;
                   ^
    psutil/_psutil_sunos.c:122:15: warning: unused variable ‘py_name’ [-Wunused-variable]
         PyObject *py_name;
                   ^
    gcc -m32 -shared -L/opt/local/lib -L/opt/csw/lib -I/opt/local/include -I/opt/csw/include -I/opt/local/include/python2.7 build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_sunos.o -lkstat -lnsl -lsocket -o build/lib.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_sunos.so
    building 'psutil._psutil_posix' extension
    creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch
    creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch/solaris
    creating build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/arch/solaris/v10
    gcc -m32 -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/opt/local/include -I/opt/csw/include -I/opt/local/include/python2.7 -fPIC -DPSUTIL_SUNOS10=1 -I/opt/local/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.solaris-2.10-i86pc.32bit-2.7/psutil/_psutil_posix.o
    psutil/_psutil_posix.c: In function ‘psutil_net_if_mtu’:
    psutil/_psutil_posix.c:272:14: error: ‘struct ifreq’ has no member named ‘ifr_mtu’
         mtu = ifr.ifr_mtu;
                  ^
    error: command 'gcc' failed with exit status 1

[dmurphy18]

FYI - I checked #610 and recent code to make sure no over-writing of fix, but from what I ascertain the original fixes (610 et al.) did not fix the issue with mtu

[giampaolo]

What if you grep for "ifr_mtu" into /usr/include? Is it defined somewhere?

[dmurphy18]

@giampaolo no, in none of the /opt or /usr, for example:
root@unknown:/export/home/root# find / -name ".h" | xargs grep -w ifr_mtu
root@unknown:/export/home/root#
root@unknown:/export/home/root#
root@unknown:/export/home/root# find / -name ".h" | xargs grep -w lifr_mtu
/usr/include/net/if.h:#define lifr_mtu lifr_lifru.lifru_mtu /* mtu */
root@unknown:/export/home/root#

[wiggin15]

What's defined in Solaris is this:

 /usr/include/net/if.h:#define	lifr_mtu	lifr_lifru.lifru_mtu	/* mtu */

In Solaris it's lifr, not ifr, and that's what we use in the Solaris implementation of net_if_stats: https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_sunos.c#L1252 (this is the fix from #610)

For some reason the compilation tries to compile _psutil_posix.c with the wrong definition of ifr_mtu, but the line with this definition is wrapped with an ifdef:

#if defined(__FreeBSD__) || defined(__OpenBSD__) || defined(__APPLE__) || defined(__NetBSD__)

so the compiler shouldn't really get there....

I suspect you're not using the latest version from this repository: your compiler hits the error on psutil/_psutil_posix.c:272 but this is a blank line in the current implementation: https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_posix.c#L272

[giampaolo]

@wiggin15 AFAICT the ifr_mtu line is not wrapped into an ifdef:

psutil/psutil/_psutil_posix.c

Line 282 in a26d896

mtu = ifr.ifr_mtu;

It looks like in order to fix this problem we should be doing this:

#if defined(__sun)
    mtu = ifr.lifr_mtu;
#else
    mtu = ifr.ifr_mtu;
#endif

@dmurphy18 can you try that?

[wiggin15]

I see. I looked at psutil_net_if_stats and not at psutil_net_if_mtu, which is relatively new in _psutil_posix.c (after the fix in #610, so it's not fixed for Solaris 10). Your suggestion looks good.

[dmurphy18]

suggest using #ifdef PSUTIL_SUNOS10 as at the start of the file,
and current ln# 268 and 277 needs changing also since ifreq is not valid, really needs to be lifreq.

I can make the changes I feel needed, test it and let you know.
The issue with using a straight replacement is that I am unsure that the offset's into the structure agree between lifr.lifr_mtu and ifr.ifr_mtu

[dmurphy18]

The following compiles cleanly on Solaris, it could be optimized leveraging offsetof definitions, but preferred to keep it straight-forward. Changes ifdef'd by PSUTIL_SUNOS10.

252 static PyObject *
253 psutil_net_if_mtu(PyObject *self, PyObject *args) {
254     char *nic_name;
255     int sock = 0;
256     int ret;
257     int mtu;
258 
259 #ifdef PSUTIL_SUNOS10
260     struct lifreq lifr;
261 #else
262     struct ifreq ifr;
263 #endif
264 
265     if (! PyArg_ParseTuple(args, "s", &nic_name))
266         return NULL;
267 
268     sock = socket(AF_INET, SOCK_DGRAM, 0);
269     if (sock == -1)
270         goto error;
271 
272 #ifdef PSUTIL_SUNOS10
273     strncpy(lifr.lifr_name, nic_name, sizeof(lifr.lifr_name));
274     ret = ioctl(sock, SIOCGIFMTU, &lifr);
275     if (ret == -1)
276         goto error;
277     close(sock);
278     mtu = lifr.lifr_mtu;
279 #else
280     strncpy(ifr.ifr_name, nic_name, sizeof(ifr.ifr_name));
281     ret = ioctl(sock, SIOCGIFMTU, &ifr);
282     if (ret == -1)
283         goto error;
284     close(sock);
285     mtu = ifr.ifr_mtu;
286 #endif
287 
288     return Py_BuildValue("i", mtu);

[giampaolo]

By the way, I'm having issues compiling psutil on Solaris. Do any of you guys have a solution?
http://stackoverflow.com/questions/41233004/compile-c-python-extension-module-on-solaris

[giampaolo]

I should have fixed this in 405c5fe.
@dmurphy18 can you give it a try?

[dmurphy18]

@giampaolo wondering about ifr on a Solaris system, will throw compile errors since not defined

+#ifdef PSUTIL_SUNOS10

• strncpy(lifr.lifr_name, nic_name, sizeof(lifr.lifr_name));
  +#else
  strncpy(ifr.ifr_name, nic_name, sizeof(ifr.ifr_name));
  +#endif
  ret = ioctl(sock, SIOCGIFMTU, &ifr); <============

Have not compiled since error is plain.

[giampaolo]

Right. That's what happens when you're not able to test a change on the actual system. =)
Fixed now in c9a417a. Could you guys please verify it works?

[dmurphy18]

Taking another look and will try on Solaris box in VirtualBox

[dmurphy18]

@giampaolo it compiled fine and the code looks good. Thanks for fixing this so quickly.

[giampaolo]

Cool. I will make a release soon.