Process() and get_process_list() do not close handles after capturing a process snapshot

[giampaolo]

From [email protected] on July 06, 2010 18:40:20

What steps will reproduce the problem?  
1. Call psutil.get_process_list() or psutil.Process(pid) on a continuous timer.
2. Open Task Manager and watch the handles climb continuously (I've 
tested to over 2 million open handles). 

What is the expected output?  


What do you see instead?  
Handles need to be closed when they are no longer needed. What 
version of psutil are you using? On what operating system? psutil 0.1.3
Windows XP Pro Please provide any additional information below.

Original issue: http://code.google.com/p/psutil/issues/detail?id=102

[giampaolo]

From [email protected] on July 06, 2010 10:01:40

Hi Derek, 

Can you confirm if this happens if you just create a new Process 
object without accessing any of the properties etc.? If so then that 
indicates it is probably in get_process_cpu_times() that the issue is 
occurring. Is it possible the process you're testing against does not exist? 

Thanks, 

-Jay

Labels: OpSys-Windows Performance

[giampaolo]

From [email protected] on July 06, 2010 10:04:23

Added some changes as r584 to get_process_cpu_times() to make sure 
we're closing the handle in every return path... if it is indeed 
get_process_cpu_times that's at fault then this is likely only 
occurring in cases where we are getting an access error or a nonexisting PID.

[giampaolo]

From [email protected] on July 06, 2010 10:18:20

Jay,

The following code snippet (run on a timer, but otherwise stand 
alone) creates the problem for me:

currentPIDs = psutil.get_pid_list()
for pid in currentPIDs:
    proc = psutil.Process(pid)

It also happens when I replace the code above with:

currentProcs = psutil.get_process_list()

So this does happen without accessing any properties.  The processes 
also almost always exist (the exception being when a process dies 
between when the process list snapshot was taken and when the 
individual process is being opened).

Thanks!

- Derek

[giampaolo]

From g.rodola on July 06, 2010 10:19:19

Tried r584 and it seems it didn't fix the problem.
I'm using this script:

import psutil, os

for x in xrange(1000):
    p = psutil.Process(os.getpid())

raw_input()

for x in xrange(1000):
    p = psutil.Process(os.getpid())

raw_input()


...while I keep looking at the task manager.
After the first raw_input() the number of open handlers bump up from 1113 to 2113.

[giampaolo]

From [email protected] on July 06, 2010 10:33:05

Here's a little test program to demonstrate the issue.

Attachment: psutil102.py

[giampaolo]

From [email protected] on July 06, 2010 10:39:33

I think I found the culprit this time, pid_is_running() is being 
called as the internal part of pid_exists() every time we create a 
Process object, and pid_is_running was not closing the handle it was 
creating, on any of the return paths. I've committed r585 with 
changes to correct that, Giampaolo if you can test it for me and let 
me know if it resolves the problem at your end that would be great.

Thanks,

-Jay

[giampaolo]

From [email protected] on July 06, 2010 10:49:33

Looks like it's resolved at least on my end, testing does not show 
any increased handle count using r585

[giampaolo]

From g.rodola on July 06, 2010 13:22:56

Verified and it works fine here as well (XP sp3).

I think we should provide a test for this, maybe by exposing a 
private C call to retrieve the number of open handles per-process, 
and check the result after calling every Process method.

We should also verify whether we have this same issue on OSX and BSD.

Labels: Milestone-0.2.0

[giampaolo]

From g.rodola on July 06, 2010 16:33:07

Windows appears to be the only implementation which deals with "handles".
After some research it seems that retrieving the number of open 
handles for writing a test is not as easy as I though at first.
I'll see whether this can be done through WMI in Python though.
As for now I'm closing this out as fixed.
Thanks a lot for the report.

Status: Fixed

[giampaolo]

From g.rodola on March 02, 2013 03:53:38

Updated csets after the SVN -> Mercurial migration: r584 == revision 
4dce9ff396b6 r585 == revision 905174598325