coverage==4.0 hangs indefinitely on python2.7 + windows

[nedbat]

Originally reported by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

The reproduction here is a bit complicated because I haven't quite narrowed it down yet but here goes.

I noticed this initially when my build was running indefinitely (and eventually failing) here : https://ci.appveyor.com/project/asottile/pre-commit/branch/master/job/r57jhonel6dlgd9d

My "minimal" reproduction is as follows:

1. git clone git://github.com/pre-commit/pre-commit
2. cd pre-commit
3. virtualenv venv
4. . venv/Scripts/activate
5. pip install -r requirements-dev.txt
6. pip install 'coverage<4'
7. coverage run -m pytest tests -k test_parse_merge # Runs and succeeds
8. pip install 'coverage>=4'
9. coverage run -m pytest tests -k test_parse_merge # Hangs forever

I've attached a screenshot (the contrast sucks because the window froze on my VM)

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/420
• This issue had attachments: Screen Shot 2015-10-02 at 6.32.03 PM.png. See the original issue for details.

[nedbat]

For some reason, pytest is stopping at this line: https://github.com/pytest-dev/pytest/blob/master/_pytest/capture.py#L340 I don't know why it only happens under coverage.py, or why only in this repo.

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

It would appear this is the minimal reproduce instructions (for now):

Put this in a conftest.py:

import subprocess

def cmd_output(*cmd, **kwargs):
    proc = subprocess.Popen(
        cmd,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
    )
    stdout, stderr = proc.communicate()
    returncode = proc.poll()

    if returncode is not None and returncode:
        raise CalledProcessError(
            returncode, cmd, retcode, output=(stdout, stderr),
        )
    return returncode, stdout, stderr

print(cmd_output('python', '-c', 'print(123)'))

And then run:

virtualenv ve
ve\scripts\pip install pytest coverage
ve\scripts\coverage run -m pytest --help #gets stuck

It would appear that the conftest gets run somewhere in the middle of pytest initializing all the plugins (including the capture plugin).

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

Some WinDbg session for that here http://imgur.com/oEj06YD&MofncLw&QjMdaSz

Tho the stack look unbelievable. Maybe WinDbg failed to get the right symbols coverage's C extension.

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

So I went to bisect this and I can't seem to reproduce when checking out the tagged coverage-4.0.

I'm using the git mirror because I understand git (for bisect) way better than I do mercurial.

# C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> cat .\asottile\asottile.py
import subprocess
import traceback
import threading

SUCCESS = 0
TIMEOUT = 1
FAILED = 2


def call_with_timeout(*cmd, **kwargs):
    timeout = kwargs.pop('timeout', 10)

    class out:
        pass

    def execute_command():
        out.proc = subprocess.Popen(cmd, **kwargs)
        out.proc.communicate()
        if out.proc.returncode:
            out.reason = FAILED
        else:
            out.reason = SUCCESS

    thread = threading.Thread(target=execute_command)
    thread.start()

    thread.join(timeout)

    if thread.is_alive():
        out.proc.terminate()
        thread.join()
        out.reason = TIMEOUT

    return out.reason


def main():
    try:
        subprocess.check_call(('rm', '-rf', 'venv'))
        subprocess.check_call(('virtualenv', 'venv'))
        subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
    except Exception:
        print('*' * 79)
        print('Assuming not installable')
        print('*' * 79)
        traceback.print_exc()
        return 0

    return call_with_timeout('venv/Scripts/coverage.exe', 'run', '-m', 'pytest', '--help', timeout=2)

if __name__ == '__main__':
    exit(main())

With this (checked out at https://github.com/nedbat/coveragepy/releases/tag/coverage-4.0), I get the following:

New python executable in venv\Scripts\python.exe
Installing setuptools, pip...done.
Processing c:\users\anthony\desktop\git\coveragepy
Collecting pytest
  Using cached pytest-2.8.1-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.30-py2.py3-none-any.whl
Collecting colorama (from pytest)
  Using cached colorama-0.3.3.tar.gz
Installing collected packages: py, colorama, pytest, coverage
  Running setup.py install for colorama
  Running setup.py install for coverage
Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
usage: pytest.py [options] [file_or_dir] [file_or_dir] [...]

# The rest of pytests's help

C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> echo $LASTEXITCODE
0

However if I modify my script to just be:

C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> C:\\Users\\Anthony\\AppData\\Local\\GitHub\\PortableGit_c2ba306e5
36fdf878271f7fe636a147ff37326ad\\bin\diff.exe -u .\asottile\asottile.py .\asottile\asottile2.py
--- .\asottile\asottile.py      Mon Oct  5 19:19:09 2015
+++ .\asottile\asottile2.py     Mon Oct  5 19:19:28 2015
@@ -38,7 +38,7 @@
     try:
         subprocess.check_call(('rm', '-rf', 'venv'))
         subprocess.check_call(('virtualenv', 'venv'))
-        subprocess.check_call(('venv/Scripts/pip.exe', 'install', '.', 'pytest'))
+        subprocess.check_call(('venv/Scripts/pip.exe', 'install', 'coverage', 'pytest'))
     except Exception:
         print('*' * 79)
         print('Assuming not installable')

I get:

C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +5 ~0 -0 !]> python .\asottile\asottile2.py
New python executable in venv\Scripts\python.exe
Installing setuptools, pip...done.
←[33mYou are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.←[0m
Collecting coverage
  Using cached coverage-4.0-cp27-none-win32.whl
Collecting pytest
  Using cached pytest-2.8.1-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.30-py2.py3-none-any.whl
Collecting colorama (from pytest)
  Using cached colorama-0.3.3.tar.gz
Installing collected packages: coverage, py, colorama, pytest
  Running setup.py install for colorama
Successfully installed colorama-0.3.3 coverage-4.0 py-1.4.30 pytest-2.8.1
C:\Users\Anthony\Desktop\git\coveragepy [(coverage-4.0) +4 ~0 -0 !]> echo $LASTEXITCODE
1

Maybe a bad wheel?

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

In fact, changing it to pip install --no-use-wheel coverage pytest, my little test also succeeds.

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

I'm getting somewhere, turns out the wheels / sdists that were "working" weren't building the C bits, I can reliably get both sdist and wheel to fail now -> to git bisect!

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

Bisect points at this commit:

81f56

   70  git bisect start
   71  git bisect bad `git rev-parse HEAD`
   72  git bisect good `git rev-parse coverage-3.7.1`
   73  git bisect run python asottile/asottile.py

...

81f5697e7cb2f5a064fd891617c0c8caf5ab21d9 is the first bad commit
commit [81f5697e7cb2f5a064fd891617c0c8caf5ab21d9 (bb)](https://bitbucket.org/ned/coveragepy/commits/81f5697e7cb2f5a064fd891617c0c8caf5ab21d9)
Author: Ned Batchelder <[email protected]>
Date:   Sat Nov 8 18:42:32 2014 -0500

    Use a WeakKeyDictionary to track coroutine objects to prevent leaks. Fixes #330.

:040000 [040000 (bb)](https://bitbucket.org/ned/coveragepy/commits/040000) [4d1c3da10c0447aca06560a908d48a5f726d1803 (bb)](https://bitbucket.org/ned/coveragepy/commits/4d1c3da10c0447aca06560a908d48a5f726d1803) [517dc4f3cc1ce573673e1937b7250613cd1f02b3 (bb)](https://bitbucket.org/ned/coveragepy/commits/517dc4f3cc1ce573673e1937b7250613cd1f02b3) M      coverage
:040000 [040000 (bb)](https://bitbucket.org/ned/coveragepy/commits/040000) [34a2a5d7944a390cada7f2b71f16fee9e1ca68fc (bb)](https://bitbucket.org/ned/coveragepy/commits/34a2a5d7944a390cada7f2b71f16fee9e1ca68fc) [02892b0265447b6e85f0c2485ae73eb3978ce03c (bb)](https://bitbucket.org/ned/coveragepy/commits/02892b0265447b6e85f0c2485ae73eb3978ce03c) M      tests
bisect run success

[nedbat]

I've confirmed that @ionelmc's conftest.py demonstrates the problem, and that the commit @asottile found is the first showing it. Its parent does not have the problem. Any theories about what's wrong with that commit?

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

For what it's worth here's a simpler reproduction. Removing any line here causes it to pass.

# test.py
import test2

# test2.py
import subprocess
subprocess.Popen(
    ('echo',),
    stdin=subprocess.PIPE,
    stdout=subprocess.PIPE,
).communicate()

And coverage run -m test

[nedbat]

The troublesome commit is 2ff50c9162ce (bb), and it also is the apparent cause of #445....

[nedbat]

Fixed in 231bad38c241 (bb)

[nedbat]

Original comment by Anthony Sottile (Bitbucket: asottile, GitHub: asottile)

---

Yay, seems fixed:

(ignore the sha, that's from the git repo I mercurial cloned in a subdir (sanity is not always my forte))

IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ pip wheel .
Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy
Building wheels for collected packages: coverage
  Running setup.py bdist_wheel for coverage
  Stored in directory: c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse
Successfully built coverage
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ ls wheelhouse/
coverage-4.0.3-cp27-none-win32.whl
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ pip install wheelhouse/coverage-4.0.3-cp27-none-win32.whl
Processing c:\users\ieuser\desktop\coveragepy\hg\coveragepy\wheelhouse\coverage-4.0.3-cp27-none-win32.whl
Installing collected packages: coverage
Successfully installed coverage-4.0.3
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ cp ../../test
test.py    test2.py   test2.pyc  tests/
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ cp ../../test
test.py    test2.py   test2.pyc  tests/
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ cp ../../test*.py ./
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ coverage run -m test
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$ echo $?
0
(venv)
IEUser@IE11Win7 MINGW32 ~/Desktop/coveragepy/hg/coveragepy ((2ddd73a...))
$