_psutil_linux.get_process_cpu_affinity always throws OSError: [Errno 22] Invalid argument #522

mlowicki · 2014-07-18T13:39:36Z

nginx     73376  0.2  0.0 166900  3840 ?        S<   06:26   1:05 nginx: worker process
root      90978  0.0  0.0 164992  2356 ?        Ss   Jun26   0:00 nginx: master process /usr/sbin/nginx
root     120554  0.0  0.0   7832   880 pts/0    S+   13:36   0:00 grep nginx
root@front1:/var/diamond# ./bin/python
Python 2.7.3 (default, Jan  2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import _psutil_linux
>>> _psutil_linux.get_process_cpu_affinity(90987)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 22] Invalid argument
>>>

root      90978  0.0  0.0 164992  2356 ?        Ss   Jun26   0:00 nginx: master process /usr/sbin/nginx
root     120995  0.0  0.0   7832   876 pts/0    S+   13:42   0:00 grep nginx
root@front1:/var/diamond# ./bin/python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process(90978)
>>> p.get_cpu_affinity()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/__init__.py", line 544, in get_cpu_affinity
    return self._platform_impl.get_process_cpu_affinity()
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 463, in wrapper
    return fun(self, *args, **kwargs)
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 821, in get_process_cpu_affinity
    bitmask = _psutil_linux.get_process_cpu_affinity(self.pid)
OSError: [Errno 22] Invalid argument
>>>

I'm running Debian Wheezy and psutil 1.2.1

The text was updated successfully, but these errors were encountered:

mlowicki · 2014-07-18T13:49:28Z

The same happens with version 2.1.1:

root      90978  0.0  0.0 164992  2356 ?        Ss   Jun26   0:00 nginx: master process /usr/sbin/nginx
root     122629  0.0  0.0   7832   876 pts/0    S+   13:47   0:00 grep nginx
root@front1:/var/diamond# ./bin/python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'2.1.1'
>>> p = psutil.Process(90978)
>>> dir(p)
['__class__', '__delattr__', '__dict__', '__doc__', '__eq__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_create_time', '_exe', '_gone', '_hash', '_ident', '_init', '_last_proc_cpu_times', '_last_sys_cpu_times', '_name', '_pid', '_ppid', '_proc', '_send_signal', 'as_dict', 'children', 'cmdline', 'connections', 'cpu_affinity', 'cpu_percent', 'cpu_times', 'create_time', 'cwd', 'exe', 'get_children', 'get_connections', 'get_cpu_affinity', 'get_cpu_percent', 'get_cpu_times', 'get_ext_memory_info', 'get_io_counters', 'get_ionice', 'get_memory_info', 'get_memory_maps', 'get_memory_percent', 'get_nice', 'get_num_ctx_switches', 'get_num_fds', 'get_num_threads', 'get_open_files', 'get_rlimit', 'get_threads', 'getcwd', 'gids', 'io_counters', 'ionice', 'is_running', 'kill', 'memory_info', 'memory_info_ex', 'memory_maps', 'memory_percent', 'name', 'nice', 'num_ctx_switches', 'num_fds', 'num_threads', 'open_files', 'parent', 'pid', 'ppid', 'resume', 'rlimit', 'send_signal', 'set_cpu_affinity', 'set_ionice', 'set_nice', 'set_rlimit', 'status', 'suspend', 'terminal', 'terminate', 'threads', 'uids', 'username', 'wait']
>>> p.get_cpu_affinity()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/_common.py", line 144, in inner
    return getattr(self, replacement)(*args, **kwargs)
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/__init__.py", line 679, in cpu_affinity
    return self._proc.cpu_affinity_get()
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 694, in wrapper
    return fun(self, *args, **kwargs)
  File "/var/diamond/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 1065, in cpu_affinity_get
    bitmask = cext.proc_cpu_affinity_get(self.pid)
OSError: [Errno 22] Invalid argument
>>>

mlowicki · 2014-08-01T13:23:11Z

Ping.

giampaolo · 2014-08-01T13:31:37Z

Sorry, this totally slipped under my radar.
I don't know why sched_getaffinity() (the C function used internally)
returns EINVAL.
What I know though is that this exact same function made its appearance
into Python 3.3 so you could take a Python 3.3+ interpret and see if it
works.
On my Linux laptop with Python 3.4 I get this:

import os
os.sched_getaffinity(os.getpid())
{0, 1, 2, 3}

If that works for you as well it means psutil implementation is broken and
needs to be fixed.

On Fri, Aug 1, 2014 at 3:23 PM, Michał Łowicki [email protected]
wrote:

Ping.

—
Reply to this email directly or view it on GitHub
#522 (comment).

Giampaolo - http://grodola.blogspot.com

mlowicki · 2014-08-01T13:49:13Z

Python 3.4.1 (default, Aug  1 2014, 13:40:44) 
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> p = psutil.Process(121892)
>>> p.get_cpu_affinity()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_common.py", line 144, in inner
    return getattr(self, replacement)(*args, **kwargs)
  File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/__init__.py", line 679, in cpu_affinity
    return self._proc.cpu_affinity_get()
  File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_pslinux.py", line 694, in wrapper
    return fun(self, *args, **kwargs)
  File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_pslinux.py", line 1065, in cpu_affinity_get
    bitmask = cext.proc_cpu_affinity_get(self.pid)
OSError: [Errno 22] Invalid argument
>>>

mongodb  121892  3.8  3.7 20718192 2462168 ?    Sl   Jul15 943:54 /usr/bin/mongod --config /etc/mongod.conf

The same issue.

giampaolo · 2014-08-01T13:56:22Z

OK, in that case I would recommend filing a bug on the Python bug tracker:
http://bugs.python.org/
From there we can see if someone pops up and suggests a solution.
Please provide the link to the actual bug once you have submitted it.
Thanks.

On Fri, Aug 1, 2014 at 3:49 PM, Michał Łowicki [email protected]
wrote:

Python 3.4.1 (default, Aug 1 2014, 13:40:44)
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import psutil
p = psutil.Process(121892)
p.get_cpu_affinity()
Traceback (most recent call last):
File "", line 1, in
File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_common.py", line 144, in inner
return getattr(self, replacement)(_args, *_kwargs)
File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/init.py", line 679, in cpu_affinity
return self._proc.cpu_affinity_get()
File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_pslinux.py", line 694, in wrapper
return fun(self, _args, *_kwargs)
File "/var/diamond/python341/Python-3.4.1/psutil-2.1.1/psutil/_pslinux.py", line 1065, in cpu_affinity_get
bitmask = cext.proc_cpu_affinity_get(self.pid)
OSError: [Errno 22] Invalid argument

The same issue.

—
Reply to this email directly or view it on GitHub
#522 (comment).

Giampaolo - http://grodola.blogspot.com

mlowicki · 2014-08-07T11:24:06Z

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>


int main() {
    cpu_set_t my_set;
    pid_t p = 104208;
    int ret;
    CPU_ZERO(&my_set);

    ret = sched_getaffinity(p, sizeof(cpu_set_t), &my_set);

    if (ret == -1) {
        perror("sched_getaffinity");
    }

    printf("sched_getaffinity = %d\n", ret);

    return EXIT_SUCCESS;
}

sched_getaffinity seems to work fine:

root@front1:~# ./a.out 
sched_getaffinity = 0

Do you control in psutil size of cpuset? Maybe it's related to it:

EINVAL

(sched_getaffinity() and, in kernels before 2.6.9, sched_setaffinity()) cpusetsize is smaller than the size of the affinity mask used by the kernel.

mlowicki · 2014-08-07T11:28:33Z

When I do it as in psutil_proc_cpu_affinity_get:

    unsigned long mask;
    unsigned int len = sizeof(mask);
    pid_t p = 104208;
    int ret;

    ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

    if (ret == -1) {
        perror("sched_getaffinity");
    }

I get:

root@front1:~# ./a.out 
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:~#

I've replaced:

unsigned long mask;

with

cpu_set_t mask;

and seems to work fine.

ddaeschler · 2014-08-07T14:23:35Z

That does appear to be the issue.

psutil_proc_cpu_affinity_get() is passing a long when the
sched_getaffinity function
expects a cpu_set_t. Seems like a quick fix unless the linux API had
changed for this between versions.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 7:28 AM, Michał Łowicki [email protected]
wrote:

When I do it as in psutil_proc_cpu_affinity_get:
unsigned long mask;
unsigned int len = sizeof(mask);
pid_t p = 104208;
int ret;

ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

if (ret == -1) {
    perror("sched_getaffinity");
}
I get:

root@front1:# ./a.out
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:#

—
Reply to this email directly or view it on GitHub
#522 (comment).

ddaeschler · 2014-08-07T14:33:42Z

I just took a look at sched.h (
http://stuff.mit.edu/afs/sipb/user/mkgray/bar/usr/include/bits/sched.h ).
It appears that the cpu_set_t is 1024 bits that form a large bitmask for
big processor systems. A long won't have enough bits to hold the return
values from linux.

So maybe not an easy fix since it is a change in the return value from the
C function. Unless the return value is already being specially processed
for the linux case.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 7:28 AM, Michał Łowicki [email protected]
wrote:

When I do it as in psutil_proc_cpu_affinity_get:
unsigned long mask;
unsigned int len = sizeof(mask);
pid_t p = 104208;
int ret;

ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

if (ret == -1) {
    perror("sched_getaffinity");
}
I get:

root@front1:# ./a.out
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:#

—
Reply to this email directly or view it on GitHub
#522 (comment).

giampaolo · 2014-08-07T14:39:14Z

Can you provide a patch?

On Thu, Aug 7, 2014 at 4:33 PM, ddaeschler [email protected] wrote:

I just took a look at sched.h (
http://stuff.mit.edu/afs/sipb/user/mkgray/bar/usr/include/bits/sched.h ).
It appears that the cpu_set_t is 1024 bits that form a large bitmask for
big processor systems. A long won't have enough bits to hold the return
values from linux.

So maybe not an easy fix since it is a change in the return value from the
C function. Unless the return value is already being specially processed
for the linux case.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 7:28 AM, Michał Łowicki [email protected]
wrote:

When I do it as in psutil_proc_cpu_affinity_get:

unsigned long mask;
unsigned int len = sizeof(mask);
pid_t p = 104208;
int ret;

ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

if (ret == -1) {
perror("sched_getaffinity");
}

I get:

root@front1:# ./a.out
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:#

—
Reply to this email directly or view it on GitHub
#522 (comment).

—
Reply to this email directly or view it on GitHub
#522 (comment).

Giampaolo - http://grodola.blogspot.com

ddaeschler · 2014-08-07T14:42:09Z

Sure. I can take a look at this one over the weekend.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 10:39 AM, giampaolo [email protected] wrote:

Can you provide a patch?

On Thu, Aug 7, 2014 at 4:33 PM, ddaeschler [email protected]
wrote:

I just took a look at sched.h (
http://stuff.mit.edu/afs/sipb/user/mkgray/bar/usr/include/bits/sched.h
).
It appears that the cpu_set_t is 1024 bits that form a large bitmask for
big processor systems. A long won't have enough bits to hold the return
values from linux.

So maybe not an easy fix since it is a change in the return value from
the
C function. Unless the return value is already being specially processed
for the linux case.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 7:28 AM, Michał Łowicki [email protected]

wrote:

When I do it as in psutil_proc_cpu_affinity_get:

unsigned long mask;
unsigned int len = sizeof(mask);
pid_t p = 104208;
int ret;

ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

if (ret == -1) {
perror("sched_getaffinity");
}

I get:

root@front1:# ./a.out
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:#

—
Reply to this email directly or view it on GitHub
#522 (comment).

—
Reply to this email directly or view it on GitHub
#522 (comment).

Giampaolo - http://grodola.blogspot.com

—

Reply to this email directly or view it on GitHub
#522 (comment).

ddaeschler · 2014-08-09T17:38:01Z

Patch attached. Tested and works on updated ubuntu VM.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 10:39 AM, giampaolo [email protected] wrote:

Can you provide a patch?

On Thu, Aug 7, 2014 at 4:33 PM, ddaeschler [email protected]
wrote:

I just took a look at sched.h (
http://stuff.mit.edu/afs/sipb/user/mkgray/bar/usr/include/bits/sched.h
).
It appears that the cpu_set_t is 1024 bits that form a large bitmask for
big processor systems. A long won't have enough bits to hold the return
values from linux.

So maybe not an easy fix since it is a change in the return value from
the
C function. Unless the return value is already being specially processed
for the linux case.

David Daeschler
Partner,
InWorldz, LLC

On Thu, Aug 7, 2014 at 7:28 AM, Michał Łowicki [email protected]

wrote:

When I do it as in psutil_proc_cpu_affinity_get:

unsigned long mask;
unsigned int len = sizeof(mask);
pid_t p = 104208;
int ret;

ret = sched_getaffinity(p, len, (cpu_set_t *)&mask);

if (ret == -1) {
perror("sched_getaffinity");
}

I get:

root@front1:# ./a.out
sched_getaffinity: Invalid argument
sched_getaffinity = -1
root@front1:#

—
Reply to this email directly or view it on GitHub
#522 (comment).

—
Reply to this email directly or view it on GitHub
#522 (comment).

Giampaolo - http://grodola.blogspot.com

—

Reply to this email directly or view it on GitHub
#522 (comment).

mlowicki · 2014-08-10T07:53:10Z

@ddaeschler where I can find a patch?

ddaeschler · 2014-08-10T14:41:05Z

I forked and created a pull request which I think makes it easier to merge.

I did attach the patch file to the email but it doesn't look like it makes
it to github. If need be I can also put up the patch file somewhere.

David Daeschler
Partner,
InWorldz, LLC

On Sun, Aug 10, 2014 at 3:53 AM, Michał Łowicki [email protected]
wrote:

@ddaeschler https://github.com/ddaeschler where I can find a patch?

—
Reply to this email directly or view it on GitHub
#522 (comment).

ddaeschler · 2014-08-10T19:07:57Z

Just committed the last change to the pull request that makes all CI tests pass.

giampaolo · 2014-08-13T13:01:45Z

I'll take a look at this when I find some time (I'm going to Japan soon so I'm very short on time) but AFAICT it looks ok to merge. I'll just make sure this will work on python 2.4 as well.

ddaeschler · 2014-08-14T01:17:26Z

OK! The PR will be here when you get back. Enjoy your time in Japan.

giampaolo · 2014-08-14T02:25:15Z

Thanks man.
Il 18/lug/2014 15:39 "Michał Łowicki" [email protected] ha scritto:

nginx 73376 0.2 0.0 166900 3840 ? S< 06:26 1:05 nginx: worker process
root 90978 0.0 0.0 164992 2356 ? Ss Jun26 0:00 nginx: master process /usr/sbin/nginx
root 120554 0.0 0.0 7832 880 pts/0 S+ 13:36 0:00 grep nginx
root@front1:/var/diamond# ./bin/python
Python 2.7.3 (default, Jan 2 2013, 13:56:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import _psutil_linux
_psutil_linux.get_process_cpu_affinity(90987)
Traceback (most recent call last):
File "", line 1, in
OSError: [Errno 22] Invalid argument

I'm running Debian Wheezy and psutil 1.2.1

—
Reply to this email directly or view it on GitHub
#522.

mlowicki · 2014-09-10T08:48:49Z

Ping.

giampaolo · 2014-09-15T15:31:51Z

Merged (finally). Sorry for waiting this long.

…). Since OpenStack uses this version of psutil, we cannot use the latest version that solves this problem. Instead, we use other library, cpu-affinity, to get the CPU affinity

* Bugs in python-psutil Need to upgrade to package from jessie-backports in order to address the following issues: giampaolo/psutil#522 giampaolo/psutil#572 * Race when checking for processes It seems that is possible to query for a process that no longer exists. Handle that situation by ignoring it. * Wrong message when no processes are matches Not a problem at all, but handle that situation by printing a different message, just to be clear. * Filter out qemu-system-x86_64 processes that do not have Ceph disks For some reason, it seems that QEMU maps librados and librbd libraries even when not using them (ie NFS, DRBD). Add a function that parses the cmdline of each QEMU process, looks for disk drives, and by looking at the path, tells if we should check that process or not. Only check VMs with rbd and tapdev (Archipelago) disks. Also, split out process filtering stuff to a separate function.

kormoc mentioned this issue Jul 23, 2014

OSError while running processresources.py BrightcoveOS/Diamond#682

Closed

giampaolo mentioned this issue Aug 10, 2014

Fix get_affinity for linux #527

Merged

giampaolo added linux enhancement labels Sep 15, 2014

giampaolo closed this as completed Sep 15, 2014

giampaolo mentioned this issue Sep 24, 2014

undefined symbol: CPU_ALLOC #536

Closed

ddaeschler mentioned this issue Sep 24, 2014

Fix psutil_proc_cpu_affinity_get to work without CPU_ALLOC #537

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

_psutil_linux.get_process_cpu_affinity always throws OSError: [Errno 22] Invalid argument #522

_psutil_linux.get_process_cpu_affinity always throws OSError: [Errno 22] Invalid argument #522

mlowicki commented Jul 18, 2014

mlowicki commented Jul 18, 2014

mlowicki commented Aug 1, 2014

giampaolo commented Aug 1, 2014

mlowicki commented Aug 1, 2014

giampaolo commented Aug 1, 2014

mlowicki commented Aug 7, 2014

mlowicki commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

giampaolo commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

ddaeschler commented Aug 9, 2014

mlowicki commented Aug 10, 2014

ddaeschler commented Aug 10, 2014

ddaeschler commented Aug 10, 2014

giampaolo commented Aug 13, 2014

ddaeschler commented Aug 14, 2014

giampaolo commented Aug 14, 2014

mlowicki commented Sep 10, 2014

giampaolo commented Sep 15, 2014

_psutil_linux.get_process_cpu_affinity always throws OSError: [Errno 22] Invalid argument #522

_psutil_linux.get_process_cpu_affinity always throws OSError: [Errno 22] Invalid argument #522

Comments

mlowicki commented Jul 18, 2014

mlowicki commented Jul 18, 2014

mlowicki commented Aug 1, 2014

giampaolo commented Aug 1, 2014

mlowicki commented Aug 1, 2014

giampaolo commented Aug 1, 2014

mlowicki commented Aug 7, 2014

mlowicki commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

giampaolo commented Aug 7, 2014

ddaeschler commented Aug 7, 2014

ddaeschler commented Aug 9, 2014

mlowicki commented Aug 10, 2014

ddaeschler commented Aug 10, 2014

ddaeschler commented Aug 10, 2014

giampaolo commented Aug 13, 2014

ddaeschler commented Aug 14, 2014

giampaolo commented Aug 14, 2014

mlowicki commented Sep 10, 2014

giampaolo commented Sep 15, 2014