Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process.status() returns 'running' for a ZombieProcess #783

Closed
the-gigi opened this issue Feb 28, 2016 · 32 comments · Fixed by #908
Closed

Process.status() returns 'running' for a ZombieProcess #783

the-gigi opened this issue Feb 28, 2016 · 32 comments · Fixed by #908
Labels

Comments

@the-gigi
Copy link

This is on a Mac OSX El Capitan Here is a quick session to demonstrate. Process 70 on my machine right now is indeed a zombie process.

Python 2.7.10 (default, Jun 10 2015, 19:43:32)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> psutil.__version__
'4.0.0'
>>> p = psutil.Process(pid=70)
>>> p.status()
'running'
>>> p.name()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/__init__.py", line 564, in name
    cmdline = self.cmdline()
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/__init__.py", line 617, in cmdline
    return self._proc.cmdline()
  File "/Users/gigi/.virtualenvs/work/lib/python2.7/site-packages/psutil/_psosx.py", line 207, in wrapper
    raise ZombieProcess(self.pid, self._name, self._ppid)
psutil.ZombieProcess: psutil.ZombieProcess process still exists but it's a zombie (pid=70)

I followed the code to the status() method in _psxosx.py to see why it returns 'running'

    @wrap_exceptions
    def status(self):
        code = cext.proc_status(self.pid)
        # XXX is '?' legit? (we're not supposed to return it anyway)
        return PROC_STATUSES.get(code, '?')

The cext.proc_status(self.pid) returns 2, which is the code for 'running'.

@the-gigi the-gigi changed the title psutil Process.status() returns 'running' for a ZombieProcess Process.status() returns 'running' for a ZombieProcess Feb 28, 2016
@giampaolo
Copy link
Owner

Mmmm if that is true it's the kernel which is returning the bad info.
There's a test for this:

@unittest.skipUnless(POSIX, 'posix only')

....which explicitly tests the case assert p.status() == ZOMBIE:
call_until(lambda: zproc.status(),

Can you run tests (make test) and see if that fails?

@giampaolo
Copy link
Owner

Bump?

@zachriggle
Copy link

zachriggle commented May 7, 2016

Many tests currently fail immediately OS X 10.11 El Capitain.

I'd recommend adding a Travis CI target for OSX.

$ uname -a
Darwin riggle-macbookpro2 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
$ make test
...
======================================================================
FAIL: test__all__ (test_misc.TestMisc)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_misc.py", line 156, in test__all__
    self.fail('%r not in psutil.__all__' % name)
AssertionError: 'x' not in psutil.__all__

======================================================================
FAIL: test_vmem_free (test_osx.TestSystemAPIs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 68771840L != 27615232 within 512000 delta

======================================================================
FAIL: test_fetch_all (test_process.TestFetchAllProcesses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1594, in test_fetch_all
    self.fail(''.join(failures))
AssertionError:
    ======================================================================
    FAIL: test_exe (proc=psutil.Process(pid=82273, name='cupsd'), ret='/usr/sbin/cupsd'))
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1579, in test_fetch_all
        meth(ret, p)
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1614, in exe
        self.assertTrue(os.access(ret, os.X_OK))
      File "/Users/riggle/.pyenv/versions/2.7.9/lib/python2.7/unittest/case.py", line 422, in assertTrue
        raise self.failureException(msg)
    AssertionError: False is not true


======================================================================
FAIL: test_proc_cwd (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2021, in test_proc_cwd
    self.assertEqual(encode_path(p.cwd()), funny_directory)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/%C0%80' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/\xc0\x80'

======================================================================
FAIL: test_proc_exe (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1983, in test_proc_exe
    self.assertEqual(encode_path(os.path.basename(p.exe())), b"\xc0\x80")
AssertionError: '%C0%80' != '\xc0\x80'

======================================================================
FAIL: test_proc_open_files (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2036, in test_proc_open_files
    self.assertIn(funny_file, encode_path(path))
AssertionError: '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/\xc0\x80' not found in '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmptYaTwJ/%C0%80'

======================================================================
FAIL: test_prog_w_funky_name (test_process.TestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 744, in test_prog_w_funky_name
    os.path.normcase(funky_path))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu1GIghwfoo bar )' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu1GIghwfoo bar )'

======================================================================
FAIL: test_proc_cwd (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1905, in test_proc_cwd
    self.assertEqual(p.cwd(), tdir)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-e\xcc\x80-RaOftt' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-\xc3\xa8-RaOftt'

======================================================================
FAIL: test_proc_open_files (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1919, in test_proc_open_files
    self.assertEqual(os.path.normcase(path), os.path.normcase(self.uexe))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psuExQYoPe\xcc\x80' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psuExQYoP\xc3\xa8'

----------------------------------------------------------------------
Ran 359 tests in 4.451s

FAILED (failures=9, skipped=195)
make: *** [test] Error 1

@giampaolo
Copy link
Owner

There's already a Travis CI although I'm not sure what OSX version is running.
As for the issue at hand, your test output tells status() correctly signals a zombie process so I'm not sure how to proceed (and I don't have an OSX El Captain to tests against unfortunately).

@zachriggle
Copy link

zachriggle commented May 7, 2016

Unfortunately, the issue is valid.

>>> import psutil
>>> psutil.__version__
'4.2.0'
>>> init = psutil.Process(1)
>>> init
<psutil.Process(pid=1, name='launchd') at 4422598736>
>>> init.status()
'running'
>>> init.is_running()
True
>>> init.cmdline()
---------------------------------------------------------------------------
ZombieProcess                             Traceback (most recent call last)
<ipython-input-10-b51ce1ac1fc9> in <module>()
----> 1 init.cmdline()

/Users/riggle/psutil/psutil/__init__.pyc in cmdline(self)
    615     def cmdline(self):
    616         """The command line this process has been called with."""
--> 617         return self._proc.cmdline()
    618
    619     def status(self):

/Users/riggle/psutil/psutil/_psosx.pyc in wrapper(self, *args, **kwargs)
    212                     raise NoSuchProcess(self.pid, self._name)
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):
    216                 raise AccessDenied(self.pid, self._name)

ZombieProcess: psutil.ZombieProcess process still exists but it's a zombie (pid=1, name='launchd')
>>> %debug
> /Users/riggle/psutil/psutil/_psosx.py(214)wrapper()
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):

ipdb> list
    209         except OSError as err:
    210             if err.errno == errno.ESRCH:
    211                 if not pid_exists(self.pid):
    212                     raise NoSuchProcess(self.pid, self._name)
    213                 else:
--> 214                     raise ZombieProcess(self.pid, self._name, self._ppid)
    215             if err.errno in (errno.EPERM, errno.EACCES):
    216                 raise AccessDenied(self.pid, self._name)
    217             raise
    218     return wrapper
    219
ipdb> p err
OSError(3, 'No such process')

@giampaolo
Copy link
Owner

giampaolo commented May 7, 2016

This was Python 2 I suppose? Can you try Python 3? I wonder whether those apparently unicode related issues are 2.X only.

@zachriggle
Copy link

Nope, the test failures are still present during make test, and the Zombie process issue also repros.

$ python --version
Python 3.5.1
$ python -c 'import psutil; print(psutil.__version__)'
4.2.0
$ uname -a
Darwin riggle-macbookpro2 15.4.0 Darwin Kernel Version 15.4.0: Fri Feb 26 22:08:05 PST 2016; root:xnu-3248.40.184~3/RELEASE_X86_64 x86_64
$ make test
======================================================================
ERROR: test_halfway_terminated_process (test_process.TestProcess)
----------------------------------------------------------------------
RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1277, in test_halfway_terminated_process
    ret = meth()
  File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
    return self._proc.open_files()
  File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
    rawlist = cext.proc_open_files(self.pid)
SystemError: <class 'OSError'> returned a result with an error set

======================================================================
ERROR: test_zombie_process (test_process.TestProcess)
----------------------------------------------------------------------
RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1342, in test_zombie_process
    zproc.as_dict()
  File "/Users/riggle/psutil/psutil/__init__.py", line 473, in as_dict
    ret = attr()
  File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
    return self._proc.open_files()
  File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
    rawlist = cext.proc_open_files(self.pid)
SystemError: <class 'OSError'> returned a result with an error set

======================================================================
FAIL: test_vmem_free (test_osx.TestSystemAPIs)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
  File "/Users/riggle/psutil/psutil/tests/__init__.py", line 491, in wrapper
    return fun(*args, **kwargs)
  File "/Users/riggle/psutil/psutil/tests/test_osx.py", line 156, in test_vmem_free
    delta=MEMORY_TOLERANCE)
AssertionError: 3845955584 != 1574113280 within 512000 delta

======================================================================
FAIL: test_fetch_all (test_process.TestFetchAllProcesses)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1594, in test_fetch_all
    self.fail(''.join(failures))
AssertionError:
    ======================================================================
    FAIL: test_open_files (proc=psutil.Process(pid=1, name='launchd'))
    ----------------------------------------------------------------------
    RuntimeError: proc_pidinfo(PROC_PIDLISTFDS) failed

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1559, in test_fetch_all
        ret = attr(*args)
      File "/Users/riggle/psutil/psutil/__init__.py", line 1081, in open_files
        return self._proc.open_files()
      File "/Users/riggle/psutil/psutil/_psosx.py", line 208, in wrapper
        return fun(self, *args, **kwargs)
      File "/Users/riggle/psutil/psutil/_psosx.py", line 313, in open_files
        rawlist = cext.proc_open_files(self.pid)
    SystemError: <class 'OSError'> returned a result with an error set


======================================================================
FAIL: test_proc_cwd (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2021, in test_proc_cwd
    self.assertEqual(encode_path(p.cwd()), funny_directory)
AssertionError: b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/%C0%80' != b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/\xc0\x80'

======================================================================
FAIL: test_proc_exe (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1983, in test_proc_exe
    self.assertEqual(encode_path(os.path.basename(p.exe())), b"\xc0\x80")
AssertionError: b'%C0%80' != b'\xc0\x80'

======================================================================
FAIL: test_proc_open_files (test_process.TestNonUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 2036, in test_proc_open_files
    self.assertIn(funny_file, encode_path(path))
AssertionError: b'/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/\xc0\x80' not found in b'/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/tmp5gyeiu5j/%C0%80'

======================================================================
FAIL: test_prog_w_funky_name (test_process.TestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 744, in test_prog_w_funky_name
    os.path.normcase(funky_path))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )
? --------
+ /var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psu8reo_y0wfoo bar )


======================================================================
FAIL: test_proc_cwd (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1905, in test_proc_cwd
    self.assertEqual(p.cwd(), tdir)
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_' != '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_
?                                                                 ^^
+ /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psutil-è-x40mmfk_
?                                                                 ^


======================================================================
FAIL: test_proc_open_files (test_process.TestUnicode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/riggle/psutil/psutil/tests/test_process.py", line 1919, in test_proc_open_files
    self.assertEqual(os.path.normcase(path), os.path.normcase(self.uexe))
AssertionError: '/private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè' != '/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè'
- /private/var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè
? --------                                                            ^^
+ /var/folders/hv/wnvzfp2d73b8cdxj7w_98tv0008rd2/T/psux5cme4_xè
?                                                             ^


----------------------------------------------------------------------
Ran 359 tests in 3.912s

FAILED (failures=8, errors=2, skipped=195)
...

@jomann09
Copy link
Contributor

jomann09 commented Jun 17, 2016

Hello Giampaolo, I am actually getting this error/issue with psutil 4.2.0 on Mac OS X 10.11 and running Python 2.7.10 which is the default for OS X 10.11. I have not had a chance to look at this closer but I will try to soon.

@giampaolo
Copy link
Owner

Hello. I should have fixed the zombie issue (status() erroneously returning "running" for zombies) in here: #908
Can somebody please confirm the issue is gone?

@zachriggle
Copy link

Why not add a test and have it checked via macOS continuous integration / Travis?

@giampaolo
Copy link
Owner

That would definitively be cool but I couldn't be able to reproduce the issue thus far. I just corrected what was clearly wrong with pid_exists()

@giampaolo
Copy link
Owner

Actually I realize I stated earlier in this thread that there's a test for this:

@unittest.skipUnless(POSIX, 'posix only')

....which explicitly tests the case assert p.status() == ZOMBIE:
call_until(lambda: zproc.status(),

Now that I look at the original OP's description more accurately I think that #908 won't fix this specific issue though (status() not returning "zombie"). :(
It really seems to come from the kernel as @the-gigi said:

The cext.proc_status(self.pid) returns 2, which is the code for 'running'.

@giampaolo
Copy link
Owner

Point is on OSX 10.5 (where I'm at) it does return the right value.

@zachriggle
Copy link

As in OSX 10.5 that was released ten years ago? Like Ubuntu 7.04?

@giampaolo
Copy link
Owner

Exactly :(. I develop on Linux and virtualize OSX on VirtualBox by using an hacked OSX version called iDeneb.

@josteink
Copy link

josteink commented Oct 12, 2016

As in OSX 10.5 that was released ten years ago? Like Ubuntu 7.04?

If Apple provided up to date OS images which developers could use in Azure or AWS (like Microsoft does!), maybe this wouldn't be such an issue.

Most developers are not going to bother paying $1000s of bucks to verify if the free & gratis code they are shipping (at no cost) works on a platform they don't use.

This is entirely Apple's failing.

@zachriggle
Copy link

Most developers are not going to bother paying $1000s of bucks to verify if the free & gratis code they are shipping (at no cost) works on a platform they don't use.

Travis CI offers Mac OS X environments at no cost. The psutil project already uses this: https://github.com/giampaolo/psutil/blob/master/.travis.yml#L16-L18

@giampaolo
Copy link
Owner

giampaolo commented Oct 12, 2016

OK, I got access to an OSX 10.10.4 Darwin and applied a lots of improvements to the C code.
As of now I am still unable to reproduce this issue.
Can somebody please git pull the latest code and run make test?

@zachriggle
Copy link

Can do tomorrow, thanks Giampaolo! <3
On Wed, Oct 12, 2016 at 1:53 PM giampaolo [email protected] wrote:

OK, I got access to an OSX 10.10.4 Darwin and applied a lots of
improvements to the C code.
As of now I am still unable to replicate this issue.
Can somebody please git pull the latest code and run make test?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#783 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAG0GJ0t3qx3gG4IrkBTcBPxKC8_LCx0ks5qzUjFgaJpZM4Hk5kN
.

@rp-tanium
Copy link

rp-tanium commented Oct 17, 2016

FWIW - at commit bdcde7e with OSX El Capitan, I get a test failure for

psutil.tests.test_process.TestProcess.test_zombie_process
test_zombie_process
    self.assertIn(zpid, psutil.pids())

AssertionError: 1619 not found in [1618, ...]

I get a few other test failures on el capitan as well, but less than I think I was getting in prior builds - here are all failures. This is with python 2.7, after doing make install and manually running tests.

======================================================================
FAIL: psutil.tests.test_osx.TestSystemAPIs.test_vmem_free
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/__init__.py", line 415, in wrapper
    return fun(*args, **kwargs)
  File "/build/psutil/psutil/tests/test_osx.py", line 161, in test_vmem_free
    delta=MEMORY_TOLERANCE)
AssertionError: 462008320L != 393748480 within 512000 delta

======================================================================
FAIL: psutil.tests.test_posix.TestSystemAPIs.test_pids
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/__init__.py", line 415, in wrapper
    return fun(*args, **kwargs)
  File "/build/psutil/psutil/tests/test_posix.py", line 249, in test_pids
    self.fail("difference: " + str(difference))
AssertionError: difference: [0]

======================================================================
FAIL: psutil.tests.test_process.TestNonUnicode.test_proc_cwd
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 2036, in test_proc_cwd
    self.assertEqual(encode_path(p.cwd()), funny_directory)
AssertionError: '/private/tmp/tmp3gYnjq/%C0%80' != '/private/tmp/tmp3gYnjq/\xc0\x80'

======================================================================
FAIL: psutil.tests.test_process.TestNonUnicode.test_proc_exe
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1998, in test_proc_exe
    self.assertEqual(encode_path(os.path.basename(p.exe())), b"\xc0\x80")
AssertionError: '%C0%80' != '\xc0\x80'

======================================================================
FAIL: psutil.tests.test_process.TestNonUnicode.test_proc_open_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 2052, in test_proc_open_files
    self.assertIn(funny_file, encode_path(path))
AssertionError: '/tmp/tmp3gYnjq/\xc0\x80' not found in '/private/tmp/tmp3gYnjq/%C0%80'

======================================================================
FAIL: psutil.tests.test_process.TestProcess.test_connections
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/__init__.py", line 575, in wrapper
    return fun(*args, **kwargs)
  File "/build/psutil/psutil/tests/test_process.py", line 1034, in test_connections
    ("all", "inet", "inet4", "tcp", "tcp4"))
  File "/build/psutil/psutil/tests/test_process.py", line 977, in check_conn
    self.compare_proc_sys_cons(proc.pid, [conn])
  File "/build/psutil/psutil/tests/test_process.py", line 954, in compare_proc_sys_cons
    self.assertEqual(sorted(proc_cons), sorted(sys_cons))
AssertionError: Lists differ: [pconn(fd=3, family=2, type=1,... != [pconn(fd=3, family=2, type=1,...

Second list contains 1 additional elements.
First extra element 1:
pconn(fd=8, family=1, type=1, laddr='', raddr='/var/run/mDNSResponder', status='NONE')

- [pconn(fd=3, family=2, type=1, laddr=('127.0.0.1', 60968), raddr=(), status='LISTEN')]
?                                                                                      ^

+ [pconn(fd=3, family=2, type=1, laddr=('127.0.0.1', 60968), raddr=(), status='LISTEN'),
?                                                                                      ^

+  pconn(fd=8, family=1, type=1, laddr='', raddr='/var/run/mDNSResponder', status='NONE')]

======================================================================
FAIL: psutil.tests.test_process.TestProcess.test_pid_0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1406, in test_pid_0
    self.assertRaises(psutil.NoSuchProcess, psutil.Process, 0)
AssertionError: NoSuchProcess not raised

======================================================================
FAIL: psutil.tests.test_process.TestProcess.test_zombie_process
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1396, in test_zombie_process
    self.assertIn(zpid, psutil.pids())
AssertionError: 8381 not found in [8380, 8338, 8329, 8264, 8263, 8205, 8168, 99978, 97899, 91428, 91427, 91426, 90163, 89309, 89308, 89307, 89306, 89083, 89050, 83868, 76490, 76482, 76432, 76431, 73985, 73835, 73834, 73833, 66976, 61629, 95310, 93676, 91952, 91944, 91921, 91879, 91878, 91877, 91876, 91874, 91872, 91867, 91856, 91712, 91687, 91657, 90277, 55249, 37461, 37378, 37359, 37281, 22933, 38721, 94838, 94818, 41674, 29841, 28293, 56551, 45694, 45693, 45692, 45689, 45686, 45685, 45684, 45682, 45680, 45678, 45666, 45664, 45662, 45660, 45657, 45655, 45652, 45650, 45647, 45605, 45602, 45563, 36900, 33652, 16828, 7819, 7818, 7817, 7791, 7771, 6056, 99420, 92057, 92056, 92052, 92049, 99075, 99073, 99068, 27927, 27926, 27925, 70523, 70522, 66857, 66812, 25154, 11554, 11552, 11551, 9767, 9766, 9765, 9763, 9726, 9676, 5539, 4698, 92128, 61235, 61227, 19008, 19003, 18998, 18984, 1822, 1655, 73139, 73134, 73131, 73128, 73127, 73126, 73125, 59233, 58453, 56656, 56031, 55975, 55972, 55971, 54377, 48273, 47949, 17935, 15280, 80825, 78041, 78040, 78039, 78038, 88006, 77202, 77201, 16205, 16105, 16096, 65897, 64073, 64072, 60837, 60822, 59798, 59797, 59796, 59795, 59794, 59792, 59791, 57201, 57167, 57166, 56885, 56883, 56882, 56694, 55417, 55377, 55369, 70137, 94742, 88898, 47547, 34081, 15893, 11527, 11525, 11476, 11436, 9188, 7268, 1266, 1248, 1178, 1175, 1174, 1149, 1148, 1142, 1125, 976, 975, 963, 962, 957, 956, 955, 949, 934, 933, 930, 902, 882, 870, 869, 868, 867, 863, 862, 860, 859, 858, 855, 852, 851, 849, 848, 844, 842, 840, 837, 808, 807, 806, 805, 803, 802, 801, 797, 517, 516, 513, 509, 508, 466, 463, 462, 432, 431, 426, 421, 412, 410, 407, 406, 401, 400, 397, 395, 390, 389, 380, 378, 371, 369, 368, 366, 365, 364, 363, 362, 358, 357, 355, 350, 349, 348, 345, 343, 341, 337, 335, 333, 328, 326, 320, 319, 318, 315, 312, 311, 308, 307, 306, 305, 302, 301, 299, 298, 296, 294, 293, 292, 291, 290, 289, 287, 286, 283, 277, 274, 259, 256, 253, 248, 239, 238, 236, 235, 234, 233, 232, 231, 229, 222, 219, 216, 212, 204, 199, 188, 187, 175, 170, 169, 166, 150, 144, 134, 123, 119, 118, 117, 116, 113, 112, 111, 110, 109, 105, 104, 103, 102, 100, 98, 97, 96, 95, 94, 92, 91, 89, 85, 83, 75, 74, 72, 69, 67, 63, 62, 61, 60, 56, 54, 53, 52, 50, 49, 1, 0]

======================================================================
FAIL: psutil.tests.test_process.TestUnicode.test_proc_cwd
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1919, in test_proc_cwd
    self.assertEqual(p.cwd(), tdir)
AssertionError: '/private/tmp/psutil-e\xcc\x80-cYkAGp' != '/private/tmp/psutil-\xc3\xa8-cYkAGp'

======================================================================
FAIL: psutil.tests.test_process.TestUnicode.test_proc_open_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1933, in test_proc_open_files
    self.assertEqual(os.path.normcase(path), os.path.normcase(self.uexe))
AssertionError: '/private/tmp/psutil-unittest-0uiLJue\xcc\x80' != '/private/tmp/psutil-unittest-0uiLJu\xc3\xa8'

@giampaolo
Copy link
Owner

OK, this looks better. My interpretation of the test output is that for some reason zombie PIDs do not show up in psutil.pids() list. That info comes directly from the kernel so it kind of looks like el captain filters zombies although it's weird. I don't have this problem on 14.04 / yosemite.
Also, it looks like Process.status() correctly returns "zombie".

As for the other failures:

  • I fixed "free" memory failure
  • can't reproduce the unicode related issues on yosemite; I'm currently setting up an el captain box so let's see how that goes
  • it looks like pid 0 is not returned by psutil.pids() but psutil.Process(0) doesn't fail so either psutil.pids() it's wrong ir psutil.Process(0) is.

@rp-tanium
Copy link

I forgot that I am no longer running el cap, but am now on sierra, though my build machines are el capitan so I should be able to verify both. My theory is that the unicode test failures have something to do with the way /private/tmp directories are handled on el capitan/sierra (system integrity protection?), and I think the tests themselves are otherwise fine, but I can't imagine what the actual interaction is. That's a wild theory, and I've been too lazy to reboot into safe mode and disable system integrity protection. I'm happy to help verify tests.

@giampaolo
Copy link
Owner

OK, I have fixed a tons of stuff on OSX. The only failing test which I get on el captain right now is the one related to connections() but it's a test issue.
Can you please try to re-run tests and post results?

@rp-tanium
Copy link

rp-tanium commented Oct 19, 2016

I get this after a pull from master (now commit b184c48) and a tox run:

py27 installed: configparser==3.5.0,enum34==1.1.6,flake8==3.0.4,funcsigs==1.0.2,ipaddress==1.0.17,mccabe==0.5.2,mock==2.0.0,pbr==1.10.0,-e [email protected]:giampaolo/psutil.git@b184c486bf9f9af150c69103c86d2bf10c7bdf02#egg=psutil,pycodestyle==2.0.0,pyflakes==1.2.3,six==1.10.0
py27 runtests: PYTHONHASHSEED='1662262215'
py27 runtests: commands[0] | python psutil/tests/runner.py
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss...........................sssssssssssssssssss...................ss..................F..F.F.F........sFs..s........ss..........s.......sssss......s.........F.F.Fs.............s....................ssssssssssssssssssssssssssssssssssssss
======================================================================
FAIL: psutil.tests.test_process.TestFetchAllProcesses.test_fetch_all
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1588, in test_fetch_all
    self.fail(''.join(failures))
AssertionError:
    ======================================================================
    FAIL: test_exe (proc=psutil.Process(pid=16828, name='com.adobe.ARMDC.'), ret='/Library/PrivilegedHelperTools/com.adobe.ARMDC.Communicator'))
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/build/psutil/psutil/tests/test_process.py", line 1573, in test_fetch_all
        meth(ret, p)
      File "/build/psutil/psutil/tests/test_process.py", line 1608, in exe
        self.assertTrue(os.access(ret, os.X_OK))
      File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 422, in assertTrue
        raise self.failureException(msg)
    AssertionError: False is not true


======================================================================
FAIL: psutil.tests.test_process.TestInvalidUnicode.test_proc_cwd
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1910, in test_proc_cwd
    self.assertEqual(p.cwd(), self.udir)
AssertionError: '/build/psutil/$psutild%C0%80' != '/build/psutil/$psutild\xc0\x80'

======================================================================
FAIL: psutil.tests.test_process.TestInvalidUnicode.test_proc_exe
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1888, in test_proc_exe
    self.assertEqual(p.exe(), self.uexe)
AssertionError: '/build/psutil/$psutilf%C0%80' != '/build/psutil/$psutilf\xc0\x80'

======================================================================
FAIL: psutil.tests.test_process.TestInvalidUnicode.test_proc_open_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1925, in test_proc_open_files
    os.path.normcase(self.uexe))
AssertionError: '/build/psutil/$psutilf%C0%80' != '/build/psutil/$psutilf\xc0\x80'

======================================================================
FAIL: psutil.tests.test_process.TestProcess.test_connections_unix
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/__init__.py", line 568, in wrapper
    return fun(*args, **kwargs)
  File "/build/psutil/psutil/tests/test_process.py", line 1068, in test_connections_unix
    check(SOCK_STREAM)
  File "/build/psutil/psutil/tests/test_process.py", line 1057, in check
    check_connection_ntuple(conn)
  File "/build/psutil/psutil/tests/__init__.py", line 666, in check_connection_ntuple
    assert not conn.raddr, repr(conn.raddr)
AssertionError: '/var/run/mDNSResponder'

======================================================================
FAIL: psutil.tests.test_process.TestUnicode.test_proc_cwd
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1910, in test_proc_cwd
    self.assertEqual(p.cwd(), self.udir)
AssertionError: '/build/psutil/$psutile\xcc\x80dir' != '/build/psutil/$psutil\xc3\xa8dir'

======================================================================
FAIL: psutil.tests.test_process.TestUnicode.test_proc_exe
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1888, in test_proc_exe
    self.assertEqual(p.exe(), self.uexe)
AssertionError: '/build/psutil/$psutile\xcc\x80file' != '/build/psutil/$psutil\xc3\xa8file'

======================================================================
FAIL: psutil.tests.test_process.TestUnicode.test_proc_open_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/psutil/psutil/tests/test_process.py", line 1925, in test_proc_open_files
    os.path.normcase(self.uexe))
AssertionError: '/build/psutil/$psutile\xcc\x80file' != '/build/psutil/$psutil\xc3\xa8file'

----------------------------------------------------------------------
Ran 326 tests in 4.165s

FAILED (failures=8, skipped=165)

So these are now passing (yay!)
psutil.tests.test_osx.TestSystemAPIs.test_vmem_free
psutil.tests.test_posix.TestSystemAPIs.test_pids

But I am still getting some failures. I am wondering if - and i don't know how this could be - it's not testing the current build somehow. But I've tried to block against that as much as possible.

@giampaolo
Copy link
Owner

OK that looks so much better! I believe all of these failures are related to tests, and do not indicate something wrong with psutil. I'm just not sure how to fix them as I can't reproduce them on my laptop.
Most importantly, it looks like the zombie related issues are gone.
I should have addressed those in bf15b4c and b940247.

@giampaolo
Copy link
Owner

I am keen to consider this one fixed. @the-gigi can you please confirm either by running the test suite or by verifying manually?

@the-gigi
Copy link
Author

I'm sorry. I don't have access to a Mac anymore. I'm Windows + Linux only these days.

@giampaolo
Copy link
Owner

OK, I'm gonna consider this fixed. We have a test for it which is passing (also on Travis).

@the-gigi
Copy link
Author

the-gigi commented Jan 10, 2017 via email

@tarunlalwani
Copy link

I am facing the same issue here. For a exited Zombie process the status is still running on MacOSX EL Capitan

>>> psutil.__version__
'5.3.1'

>>> proc.is_running()
True
>>> proc.memory_info()
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/__init__.py", line 1119, in memory_info
    return self._proc.memory_info()
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_psosx.py", line 295, in wrapper
    return fun(self, *args, **kwargs)
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_psosx.py", line 421, in memory_info
    rawtuple = self._get_pidtaskinfo()
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_common.py", line 336, in wrapper
    return fun(self)
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_psosx.py", line 352, in _get_pidtaskinfo
    ret = cext.proc_pidtaskinfo_oneshot(self.pid)
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tarun.lalwani/.virtualenvs/project/lib/python3.6/site-packages/psutil/_psosx.py", line 324, in catch_zombie
    raise ZombieProcess(proc.pid, proc._name, proc._ppid)
psutil.ZombieProcess: psutil.ZombieProcess process still exists but it's a zombie (pid=3555, name='firefox-bin')

@giampaolo
Copy link
Owner

What you see is correct. What does proc.status() say?

@tarunlalwani
Copy link

tarunlalwani commented Oct 7, 2017

that says zombie. Thanks will update my script

nlevitt added a commit to nlevitt/psutil that referenced this issue Apr 9, 2019
* master: (375 commits)
  update psutil
  fix procsmem script which was not printing processes
  try to fix tests on travis
  try to fix tests on travis
  OSX: fix compilation warning
  giampaolo#936: give credits to Max Bélanger
  giampaolo#811: move DLL check logic in _pswindows.py
  winmake: use the right win slashes
  winmake: do not try to install GIT commit hook if this is not a GIT cloned dir
  giampaolo#811: on Win XP let the possibility to install psutil from sources as it still (kind of) works)
  giampaolo#811: add a Q&A section in the doc; tell what Win versions are supported
  giampaolo#811: raise a meaningful error message if on Windows XP
  update doc; bump up version
  giampaolo#939: update MANIFEST to include only src files and not much else
  update HISTORY
  travis: execute mem leaks and flake8 tests only on py 2.7 and 3.5; no need to test all python versions
  bump up version
  update version in doc
  add simple test case for oneshot() ctx manager
  add simple test case for oneshot() ctx manager
  speedup fetch all process test by using oneshot
  giampaolo#799 - oneshot / linux: speedup memory_full_info and memory_maps
  fix flake8
  first pass
  giampaolo#943: better error message in case of version conflict on import.
  update doc
  799 onshot / win: no longer store the handle in python; I am now sure this is slower than using OpenProcess/CloseHandle in C
  update doc
  (win) add memleak test for proc_info()
  move stuff around
  memleak: fix false positive on windows
  giampaolo#933 (win) fix memory leak in WindowsService.description()
  giampaolo#933 (win) fix memory leak in cpu_stats() (missing free())
  refactoring
  giampaolo#799 / win: pass handle also to memory_maps() and username() functions
  fix numbers
  mem leak script: provide better error output in case of failure
  refactor memleak script
  refactor memleak script
  refactor memleak script
  refactor memleak script
  refactor memleak script: get rid of no longer used logic to deal with Process properties
  memleak script refactoring
  doc styling
  giampaolo#799 / win: use oneshot() around num_threads() and num_ctx_switches(); speedup from 1.2x to 1.8x
  refactor windows tests
  win: enable dueal process impl tests
  win / C: refactor memory_info_2 code() and return it along side other proc_info() metrics
  windows c refactor proc_info() code
  update windmake script
  winmake clean: make it an order of magnitude faster; also update Makefile
  update doc
  bench script: add psutil ver
  winmake: more aggressive logic to uninstall psutil
  adjust bench2 script to new perf API
  try to adjust perf
  upgrade perf code
  memory leak script: humanize memory difference in case of failure
  style changes
  fix giampaolo#932 / netbsd: check connections return value and raise exception
  netbsd / connections: refactoring
  netbsd / connections: refactoring
  netbsd / connections: refactoring
  netbsd / connections: refactoring
  netbsd / connections: refactoring
  testing make clean with unittests was a bad idea after all
  make 'make clean' 4x faster!
  add test for make clean
  adjust winmake script
  fix netbsd/openvsd compilation failure
  bsd: fix mem leak
  osx: fix memory leak
  pre-release
  refactoring
  update IDEAS
  add mtu test for osx and bsd
  osx: separate IFFLAGS function
  osx/bsd: separate IFFLAGS function
  linux: separate IFFLAGS function
  share C function to retrieve MTU across all UNIXes
  HISTORY: make anchors more easily referenceable
  fix giampaolo#927: Popen.__del__ may cause maximum recursion depth error.
  fix Popen test which is occasionally failing
  more releases timeline from README to doc
  ignore failing tests on OSX + TRAVIS
  update INSTALL instructions
  update print_announce.py script
  update HISTORY
  HISTORY: provide links to issues on the bug tracker
  update IDEAS
  giampaolo#910: [OSX / BSD] in case of error, psutil.pids() raised RuntimeError instead of the original OSError exception.
  fix unicode tests on windows / py3
  small test refactoring
  fix giampaolo#926: [OSX] Process.environ() on Python 3 can crash interpreter if process environ has an invalid unicode string.
  osx: fix compiler warnings
  refactor unicode tests
  fix unicode test
  giampaolo#783: fix some unicode related test failures on osx
  test refactoring
  test refactroring
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants
@zachriggle @josteink @giampaolo @the-gigi @tarunlalwani @jomann09 @rp-tanium and others