structure size mismatch

[billcchung]

Running psutil on solaris 10 (SunOS dhcp-10-140-112-88 5.10 Generic_142910-17 i86pc i386 i86pc Solaris),
while trying to get process cpu_percent I got:

Python 2.7.11 (default, Mar 14 2016, 17:31:55)
[GCC 5.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process(16311)
>>> p.cpu_percent(interval=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/csw/lib/python2.7/site-packages/psutil/__init__.py", line 928, in cpu_percent
    pt1 = self._proc.cpu_times()
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 323, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 429, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
RuntimeError: structure size mismatch

any solutions?

[giampaolo]

The error originates from here:
https://github.com/giampaolo/psutil/blob/master/psutil/_psutil_sunos.c#L74
Does this happen for all processes or 1 process only?

[billcchung]

Hi @giampaolo , looks like only some processes

   eserv 15951 15947   0 21:12:54 ?           0:01 /export/home/perf/dogfood2007/splunk/bin/python -O /export/home/perf/dogfood200
   eserv 15946     1   0 21:12:53 ?           0:02 splunkd -p 8089 start
   eserv 15880 15879   0 21:09:00 ?           0:00 /usr/lib/ssh/sshd
   eserv 15948 15947   0 21:12:54 ?           0:00 mongod --dbpath=/export/home/eserv/indicesPrebuilt/dogfood2007/6.2.11/kvstore/m
   eserv 15886 15880   0 21:09:01 pts/2       0:00 -bash
   eserv 15947 15946   0 21:12:53 ?           0:00 splunkd -p 8089 start

the processes of bash and sshd are fine:

>>> p = psutil.Process(15886)
>>> p.cpu_percent(interval=1)
0.0
>>> p = psutil.Process(15880)
>>> p.cpu_percent(interval=1)
0.0

but others are not:

>>> p = psutil.Process(15951)
>>> p.cpu_percent(interval=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/csw/lib/python2.7/site-packages/psutil/__init__.py", line 928, in cpu_percent
    pt1 = self._proc.cpu_times()
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 323, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 429, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
RuntimeError: structure size mismatch
>>> p = psutil.Process(15946)
>>> p.cpu_percent(interval=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/csw/lib/python2.7/site-packages/psutil/__init__.py", line 928, in cpu_percent
    pt1 = self._proc.cpu_times()
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 323, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 429, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
RuntimeError: structure size mismatch
>>> p = psutil.Process(15948)
>>> p.cpu_percent(interval=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/csw/lib/python2.7/site-packages/psutil/__init__.py", line 928, in cpu_percent
    pt1 = self._proc.cpu_times()
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 323, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 429, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
RuntimeError: structure size mismatch
>>> p = psutil.Process(15947)
>>> p.cpu_percent(interval=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/csw/lib/python2.7/site-packages/psutil/__init__.py", line 928, in cpu_percent
    pt1 = self._proc.cpu_times()
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 323, in wrapper
    return fun(self, *args, **kwargs)
  File "/opt/csw/lib/python2.7/site-packages/psutil/_pssunos.py", line 429, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
RuntimeError: structure size mismatch

so it's related to the process, what would you suggest me to investigate?
Thanks!

[myaut]

@billcchung
Have you tried to truss your python while running cpu_percent? I'm getting this on Solaris 11.3:

open("/proc/25076/lwp/1/lwpstatus", O_RDONLY)   = 3
read(3, 0xF6619540, 800)                        Err#79 EOVERFLOW

I think this explains it all:

2019    /*
2020     * A 32-bit process cannot get the status of a 64-bit process.
2021     * The fields for the 64-bit quantities are not large enough.
2022     */
2023    if (PROCESS_NOT_32BIT(p)) {
2024        prunlock(pnp);
2025        return (EOVERFLOW);
2026    }

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/fs/proc/prvnops.c#1839

Python binary is 32bit while some processes could be 64bit.

[giampaolo]

Apparently this happens only with cpu_times() / cpu_percent().

[giampaolo]

I made a change in 05defa0. Error still occurs but now it's at least the correct one:

Traceback (most recent call last):
  File "scripts/ps.py", line 103, in <module>
    main()
  File "scripts/ps.py", line 53, in main
    pinfo = p.as_dict(attrs, ad_value='')
  File "/export/home/user/psutil/psutil/__init__.py", line 526, in as_dict
    ret = meth()
  File "/export/home/user/psutil/psutil/__init__.py", line 1034, in cpu_times
    return self._proc.cpu_times()
  File "/export/home/user/psutil/psutil/_pssunos.py", line 324, in wrapper
    return fun(self, *args, **kwargs)
  File "/export/home/user/psutil/psutil/_pssunos.py", line 449, in cpu_times
    times = cext.proc_cpu_times(self.pid, self._procfs_path)
OSError: [Errno 79] Value too large for defined data type

[giampaolo]

Problem occurs even with open().read(), meaning it's not a psutil bug per se, but rather something weird with the OS. read() syscall can't simply read that file:

user@solaris~/psutil$ cat /proc/1460/status 
cat: input error on /proc/1460/status: Value too large for defined data type
user@solaris~/psutil$
user@solaris~/psutil$ python -c "print(open('/proc/1460/status').read())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 79] Value too large for defined data type

This is very weird.

[giampaolo]

Also occurs as root, just to rule out any permission-related problem. The process above (1460) was owned by a regular user.

[giampaolo]

Yes, the problem appears to be we cannot read CPU info of a 64bit process with a 32bit Python.
It turns out also threads() and memory_map() methods have the same issue.
Since also cat / read() fail in the same way I committed c01e005 and 07b6faa: "null" values are now returned as a fallback (which sucks, but really can't see another choice).