Resolve AccessDenied exception problem occurring most of the times on OSX

[giampaolo]

From g.rodola on June 30, 2012 04:16:11

This problem has always been affecting OSX since the first release of psutil 
and it's been raised different times ( issue 106 , issue 108 , issue 173 , 
issue 207 , issue 215 and probably others I can't find right now).

Basically, every time we use task_for_pid() call for every process != 
os.getpid() we get an AccessDenied exception.
This makes psutil basically unusable as a limited user, even for determining 
basic process information such as CPU percent or memory usage.
The only solution proposed so far has been either to run the python process as 
root or use setuid bit against the python executable.

The functions affected by this are:

get_memory_info
get_memory_percent
get_cpu_times
get_cpu_percent
get_num_threads
get_nice

After some research I discovered that such process information can also be 
determined by using proc_pidinfo(), which uses a less strict security model 
than task_for_pid(). 
Judging from the tests I've conducted so far it seems we are able to use such 
functions against all PIDs except #0, without incurring into any permission error.

Original issue: http://code.google.com/p/psutil/issues/detail?id=297

[giampaolo]

From [email protected] on June 29, 2012 19:40:29

Very interesting, this sounds like a huge improvement for OS X users. What's 
the compatibility for proc_pidinfo() - is it available on all versions of OS X, 
or only on specific versions? And which functions will we be able to replace? 
Is it interchangeable with everywhere we're using task_for_pid() currently or 
only for certain data?

[giampaolo]

From g.rodola on June 29, 2012 19:53:19

I'm not sure as there's no documentation whatsoever.
The only reference I've found is: 
http://www.opensource.apple.com/source/Libc/Libc-594.9.4/darwin/libproc.c What 
you end up doing is something like this:

static PyObject*
get_cpu_times(PyObject* self, PyObject* args)
{
    long pid;
    int ret;
    struct proc_taskinfo pti;
    if (! PyArg_ParseTuple(args, "l", &pid)) {
        return NULL;
    }
    proc_pidinfo(pid, PROC_PIDTASKINFO, 0, &pti, sizeof(pti));
    if (ret == 0) {
        if (! pid_exists(pid)) {
            return NoSuchProcess();
        }
        else {
            return AccessDenied();
        }
    }
    return Py_BuildValue("(dd)",
                         (float)pti.pti_total_user / 1000000000.0,
                         (float)pti.pti_total_system / 1000000000.0);
}

Apparently it works but I'm still not sure about the level of reliability.
In fact here's what I've just found and it doesn't look promising: 
http://vinceyuan.blogspot.it/2011/12/wrong-info-from-procpidinfo.html I'll take 
a deeper look tomorrow.

[giampaolo]

From [email protected] on June 29, 2012 20:15:03

Hmm, yes that doesn't look good. It looks like there's still a security 
constraint in place. We'll have to do some testing and figure out if we can get 
more information with this method; according to that blog post it works for 
processes owned by the user. If so that's still an improvement over the 
task_for_pid() method that only works for the calling process and not for any 
other processes even if owned by the same user.

[giampaolo]

From g.rodola on June 30, 2012 05:41:43

Indeed it works for a lot more processes, basically all the ones owned by user 
plus some other ones owned by root.
We also have a considerable speed up which is around +2.5x!
Before the patch we were able to get information *for the current process* only.
Now here's what we get:

for p in sorted(psutil.process_iter(), key=lambda x: x.pid):
    print p.pid, p.username,
    try:
        print p.get_cpu_times()
    except psutil.AccessDenied:
        print "AD"

0 root AD
1 root AD
10 root AD
11 root AD
12 root AD
13 root AD
14 root AD
15 daemon AD
34 _mdnsresponder AD
39 root AD
47 root AD
51 root AD
52 root cputimes(user=5.984751616, system=7.740481536)
53 root AD
55 root AD
56 root AD
58 root AD
61 root AD
65 root AD
66 root AD
74 root AD
77 root AD
94 user cputimes(user=4.031159552, system=9.16877312)
99 user cputimes(user=0.935546304, system=1.066071104)
103 user cputimes(user=1.283517568, system=1.664189824)
105 user cputimes(user=2.164055296, system=2.992535808)
107 user cputimes(user=0.048872692, system=0.029297902)
109 user cputimes(user=3.386659584, system=2.997433344)
111 user cputimes(user=40.16314368, system=26.4152064)
112 root AD
1280 user cputimes(user=3.944569088, system=19.325812736)
2890 user cputimes(user=0.980440576, system=0.725596544)
2891 user AD
2892 user cputimes(user=0.034143492, system=0.03271414)
2904 root AD
2911 user cputimes(user=5.010681344, system=8.001439232)
2912 user cputimes(user=0.04282536, system=0.055716432)
2923 user cputimes(user=0.434029024, system=0.5263992)
2924 user cputimes(user=3.201548032, system=4.336747008)
2925 user AD
2926 user cputimes(user=2.1825984, system=3.235505664)
2927 user AD
2928 user cputimes(user=0.330242016, system=0.475155328)
2996 root AD
3001 user cputimes(user=2.496256256, system=3.513113856)
3002 user cputimes(user=1.285510016, system=2.346267136)
5298 user cputimes(user=1.565955712, system=1.213062784)
5310 user AD
5311 user cputimes(user=0.133296064, system=0.156760896)
5324 root AD
6368 user cputimes(user=1.404700928, system=1.178013952)
7361 user cputimes(user=0.094803128, system=0.076498376)
7429 user cputimes(user=0.073027336, system=0.04521082)
7539 user cputimes(user=0.098621352, system=0.088993512)
7618 user cputimes(user=0.18574752, system=0.136978896)


This is now committed in r1404 .
We can probably do the same thing for get_memory_maps() and get_threads(), 
which are still using task_for_pid().

[giampaolo]

From [email protected] on June 30, 2012 07:44:34

That's great news, I'm sure all the OS X users will be thrilled! Nice find on that one.

[giampaolo]

From g.rodola on June 30, 2012 10:37:32

Labels: -Priority-Medium Priority-High

[giampaolo]

From g.rodola on July 04, 2012 07:04:21

Status: FixedInSVN

[giampaolo]

From g.rodola on August 13, 2012 09:25:13

Fixed in version 0.6.0, released just now.

Status: Fixed
Labels: -Milestone-1.0.0 Milestone-0.6.0

[giampaolo]

From g.rodola on August 17, 2012 11:14:08

Issue 173 has been merged into this issue.

[giampaolo]

From g.rodola on March 02, 2013 04:10:09

Updated csets after the SVN -> Mercurial migration: r1404 == revision 
09f17ac17db2