Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process hang with Coverage 6.3 #1310

Closed
tunetheweb opened this issue Jan 25, 2022 · 74 comments · Fixed by cleanlab/cleanlab#262
Closed

Process hang with Coverage 6.3 #1310

tunetheweb opened this issue Jan 25, 2022 · 74 comments · Fixed by cleanlab/cleanlab#262
Labels
bug Something isn't working concurrency fixed

Comments

@tunetheweb
Copy link

Describe the bug
A clear and concise description of the bug.

We've been having issues with our CI in GitHub Actions for the last few hours, and think it might be because of Coverage 6.3 - it's the one thing that's changed, and freezing it at 6.2 seems to allow runs to complete successfully.

To Reproduce
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:

  1. What version of Python are you using? 3.7 seems to be a bit buggy with this. We also run 3.8 - 3.10 in CI and no issues seen with that.
  2. What version of coverage.py shows the problem? The output of coverage debug sys is helpful. 6.2.3
  3. What versions of what packages do you have installed? The output of pip freeze is helpful. See https://gist.github.com/tunetheweb/4d288ea4467ba74a66b3a0e2e8d5e4ea
  4. What code shows the problem? Give us a specific commit of a specific repo that we can check out. If you've already worked around the problem, please provide a commit before that fix. This is tricky. We run a lot of commands in CI, but checking out https://github.com/sqlfluff/sqlfluff/ and running tox -e py37 -- -n 2 test should reproduce it. Having problems setting up a 3.7 environment but will try to get a better test case. We do use a multithreaded process and noticed some changes to that.
  5. What commands did you run?

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Will try to get a better repo.

@vEpiphyte
Copy link

vEpiphyte commented Jan 25, 2022

We've experienced a similar issue with CI timeouts in CircleCI when using Python 3.8.
This occurs when using pytest as our test runner with coverage.
The hang occurs after tests pass and the coverage is reported to the console; it's almost like the python process fails to exit.

(venv) root@af00fef01e49:/home/circleci/repo# python -V
Python 3.8.12

(venv) root@af00fef01e49:/home/circleci/repo# coverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 6.3
                coverage_module: /home/circleci/repo/venv/lib/python3.8/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: /home/circleci/repo/.coveragerc
                    config_file: /home/circleci/repo/.coveragerc
                config_contents: b'[report]\nomit =\n    */synapse/tests/test_*\n'
                      data_file: -none-
                         python: 3.8.12 (default, Aug 31 2021, 16:30:39) [GCC 9.3.0]
                       platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.29
                 implementation: CPython
                     executable: /home/circleci/repo/venv/bin/python3
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 322
                            cwd: /home/circleci/repo
                           path: /home/circleci/repo/venv/bin
                                 /home/circleci/.pyenv/versions/3.8.12/lib/python38.zip
                                 /home/circleci/.pyenv/versions/3.8.12/lib/python3.8
                                 /home/circleci/.pyenv/versions/3.8.12/lib/python3.8/lib-dynload
                                 /home/circleci/repo/venv/lib/python3.8/site-packages
                                 /home/circleci/repo
                    environment: HOME = /root
                   command_line: /home/circleci/repo/venv/bin/coverage debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.31.1
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-9.3.0; ENABLE_COLUMN_METADATA; ENABLE_DBSTAT_VTAB
                                 ENABLE_FTS3; ENABLE_FTS3_PARENTHESIS; ENABLE_FTS3_TOKENIZER
                                 ENABLE_FTS4; ENABLE_FTS5; ENABLE_JSON1
                                 ENABLE_LOAD_EXTENSION; ENABLE_PREUPDATE_HOOK; ENABLE_RTREE
                                 ENABLE_SESSION; ENABLE_STMTVTAB; ENABLE_UNLOCK_NOTIFY
                                 ENABLE_UPDATE_DELETE_LIMIT; HAVE_ISNAN; LIKE_DOESNT_MATCH_BLOBS
                                 MAX_SCHEMA_RETRY=25; MAX_VARIABLE_NUMBER=250000; OMIT_LOOKASIDE
                                 SECURE_DELETE; SOUNDEX; TEMP_STORE=1
                                 THREADSAFE=1; USE_URI

Pip freeze output

(venv) root@af00fef01e49:/home/circleci/repo# pip freeze
aiohttp==3.8.1
aiohttp-socks==0.6.1
aioimaplib==0.9.0
aiosignal==1.2.0
aiosmtplib==1.1.6
antlr4-python3-runtime==4.8
appdirs==1.4.4
arrow==1.2.1
async-timeout==4.0.2
attrs==21.4.0
base58==2.1.1
bech32==1.2.0
cattrs==1.10.0
cbor2==5.4.2
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.10
colorama==0.4.4
coverage==6.3
cpe==1.2.1
cryptography==36.0.1
fastjsonschema==2.14.5
fqdn==1.5.1
frozenlist==1.3.0
idna==3.3
importlib-resources==5.4.0
iniconfig==1.1.1
isoduration==20.11.0
jsonpointer==2.2
jsonschema==4.4.0
lark-parser==0.11.2
lmdb==1.3.0
more-itertools==8.12.0
msgpack==1.0.3
multidict==6.0.2
oauthlib==3.1.1
packaging==20.9
pluggy==1.0.0
prompt-toolkit==3.0.24
py==1.11.0
pycodestyle==2.8.0
pycparser==2.21
pycryptodome==3.12.0
Pygments==2.7.4
pyOpenSSL==21.0.0
pyparsing==3.0.7
pyrsistent==0.18.1
pytest==6.2.5
pytest-cov==3.0.0
python-bitcoinlib==0.11.0
python-dateutil==2.8.2
python-socks==1.2.4
PyYAML==6.0
regex==2022.1.18
requests==2.27.1
requests-cache==0.9.1
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
scalecodec==1.0.2
simplejson==3.17.6
six==1.16.0
stix2-patterns==1.3.2
stix2-validator==3.0.2
-e git+ssh://[email protected]/vertexproject/synapse.git@c8568da02165fe0c8131b8ffcc1e48e20dc2ad07#egg=synapse
toml==0.10.2
tomli==2.0.0
tornado==6.1
typing_extensions==4.0.1
uri-template==1.1.0
url-normalize==1.4.3
urllib3==1.26.8
vcrpy==4.1.1
wcwidth==0.2.5
webcolors==1.11.1
wrapt==1.13.3
xxhash==2.0.2
yarl==1.7.2
zipp==3.7.0

This happened to our master commit here
vertexproject/synapse@1b2d327

This was fixed in https://github.com/vertexproject/synapse/pull/2534/files#diff-c9c796259f3852b51b531b79cbf07820145088d4f108fb006ab44b15f0b15934R8

We did find the following line in a CI log for one of the failing tests which may or may not be related - it occurred prior to the pytest warnings summary and the console coverage output.

/home/circleci/repo/venv/lib/python3.8/site-packages/coverage/data.py:129: 
CoverageWarning: Data file '/home/circleci/repo/.coverage.803d4f93f49b.429.890315' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

@tunetheweb
Copy link
Author

Interesting that you saw this on 3.8. We experienced some intermittent hanging in other Python versions but repeated hanging (every time) in 3.7, so wasn’t sure if the intermittent issues were related.

I also forgot to include my error message. It happened near the end of the test, but not quite at the end as you experienced (not sqlfluff is our project, but the last line might be useful):

[gw1] [ 90%] PASSED test/rules/yaml_test_cases_test.py::test__rule_test_global_config /home/runner/work/sqlfluff/sqlfluff/.tox/py37/lib/python3.7/site-packages/coverage/inorout.py:535: CoverageWarning: Module sqlfluff was previously imported, but not measured (module-not-measured)
5583
  self.warn(msg, slug="module-not-measured")

@nedbat
Copy link
Owner

nedbat commented Jan 26, 2022

I'm not clear what problems you are reporting. @tunetheweb the original issue description doesn't mention what the actual problem is. @vEpiphyte says they've seen similar problems, and mentions hanging. Then @tunetheweb shows a module-not-measured warning. This sounds like two different issues.

Anything you can do to give me a reproducible scenario would be great.

@tunetheweb
Copy link
Author

Yes I appreciate this is a very poor issue to raise. Sorry about that, but thought I'd give you a heads up as saw it right after the release, and it stopped happening right after I pinned to the old version.

Will try to give a more meaningful reproducible use case if I can narrow it down.

The issue is jobs hanging (I cancelled them after an hour, when they normally take 5 mins. One job I left running and it took 3 hours and counting). That was for Python 3.7.
I also saw a lot of slow down in jobs (5 min jobs were more often than not taking 20mins when they did complete). That was for Python 3.8+.

Initially I thought GitHub Actions were on the blink but at soon as I pinned the old version of coverage it all worked. Unpinning it again breaks it again. Nothing else changes between the runs.

But as to narrowing it down why, at the moment I don't have more info to help explain that. But will keep digging...

Think the error message is a red-herring. It was the last thing printed before it froze. But do see that at the end of the run elsewhere (though interestingly I only see it at the end of good runs, whereas that appeared after our CI job was only at 90% and then hung).

Sorry, again for such a poor report. Can close it if you want until I can get you more info, as appreciate it's difficult to do anything with what I've written here but as I say this was more a heads up, and a vague hope someone would have an idea what it might be 😞

@nedbat
Copy link
Owner

nedbat commented Jan 26, 2022

@tunetheweb @vEpiphyte @sir-sigurd Can you try installing this branch of coverage.py to see if it fixes the problem?

pip install git+https://github.com/nedbat/coveragepy@nedbat/fix-1312

@tunetheweb
Copy link
Author

Doesn't look any better with that to be honest.

BTW happy to move this discussion to #1312 as that issue would appear to be a duplicate of this one, but with much better detail!

@tunetheweb
Copy link
Author

Yeah definitely seeing same issue with that unfortunately :-(

But it is very reproducible so, while I haven't been able to help with a minimal reproducible test case, I'm very happy to test any other test branches you have. Just takes about 10mins to run.

@sir-sigurd
Copy link

@tunetheweb @vEpiphyte @sir-sigurd Can you try installing this branch of coverage.py to see if it fixes the problem?

pip install git+https://github.com/nedbat/coveragepy@nedbat/fix-1312

Nope, it doesn't help.

vEpiphyte added a commit to vertexproject/synapse that referenced this issue Jan 26, 2022
@vEpiphyte
Copy link

I did reproduce the error locally with a set of our unit tests. Recording this big blob for posterity - I haven't had the time to whittle it down yet but will do shortly.

$ python -m pytest  -v -s -rs --durations 6 --maxfail 6 -p no:logging --junitxml=test-reports/junit.xml --cov synapse --no-cov-on-fail synapse/tests/test_cmds_trigger.py synapse/tests/test_lib_cell.py synapse/tests/test_lib_crypto_ecc.py synapse/tests/test_lib_hashset.py synapse/tests/test_lib_layer.py synapse/tests/test_lib_node.py synapse/tests/test_lib_scope.py synapse/tests/test_lib_stormhttp.py synapse/tests/test_lib_stormlib_model.py synapse/tests/test_lib_stormlib_yaml.py synapse/tests/test_lib_trigger.py synapse/tests/test_model_auth.py synapse/tests/test_model_geospace.py synapse/tests/test_model_media.py synapse/tests/test_servers_axon.py synapse/tests/test_tools_backup.py synapse/tests/test_tools_guid.py synapse/tests/test_tools_storm.py

This produces the same hang locally using Python 3.8.10 on linux (ubuntu 21.04) and coverage 6.3.0.

I then installed ned's patch with the following

pip install git+https://github.com/nedbat/coveragepy@nedbat/fix-1312

And got the same hang. Sending sigint (ctrl+c) to the process yielded the following traceback

^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
Process SpawnProcess-22:
Process SpawnProcess-18:
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1096, in execute
    return self.con.execute(sql, parameters)
Traceback (most recent call last):
sqlite3.OperationalError: no such table: coverage_schema

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1101, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: coverage_schema

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 318, in _bootstrap
    util._exit_function()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 270, in _read_db
    schema_version, = db.execute_one("select version from coverage_schema")
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 334, in _exit_function
    _run_finalizers(0)
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1129, in execute_one
    rows = list(self.execute(sql, parameters))
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 99, in cleanup
    _cleanup(_active_cov)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1118, in execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 84, in _cleanup
    cov.stop()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 700, in save
    data = self.get_data()
coverage.exceptions.DataError: Couldn't use data file '/home/user/PycharmProjects/synapse/.coverage.vertex005c.35596.883395': no such table: coverage_schema
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 769, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))

During handling of the above exception, another exception occurred:

  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 236, in _wrapped
    return method(self, *args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 466, in add_lines
    file_id = self._file_id(filename, add=True)
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 388, in _file_id
    cur = con.execute("insert or replace into file (path) values (?)", (filename,))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1096, in execute
    return self.con.execute(sql, parameters)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 608, in _on_sigterm
    self._atexit("sigterm")
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 604, in _atexit
    self.save()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 700, in save
    data = self.get_data()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 769, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 234, in _wrapped
    with self._lock:
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 318, in _bootstrap
    util._exit_function()
KeyboardInterrupt
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 334, in _exit_function
    _run_finalizers(0)
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 99, in cleanup
    _cleanup(_active_cov)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 84, in _cleanup
    cov.stop()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 700, in save
    data = self.get_data()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 769, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 236, in _wrapped
    return method(self, *args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 459, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 518, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 313, in _connect
    self._open_db()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 264, in _open_db
    self._read_db()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 273, in _read_db
    self._init_db(db)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 299, in _init_db
    db.executescript(SCHEMA)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1156, in executescript
    self.con.executescript(script)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 608, in _on_sigterm
    self._atexit("sigterm")
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 604, in _atexit
    self.save()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 700, in save
    data = self.get_data()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 769, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 234, in _wrapped
    with self._lock:
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1096, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: context

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1101, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: context

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 236, in _wrapped
    return method(self, *args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 463, in add_lines
    self._set_context_id()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 421, in _set_context_id
    context_id = self._context_id(context)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 397, in _context_id
    row = con.execute_one("select id from context where context = ?", (context,))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1129, in execute_one
    rows = list(self.execute(sql, parameters))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1118, in execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/home/user/PycharmProjects/synapse/.coverage.vertex005c.35541.378922': no such table: context

@nedbat
Copy link
Owner

nedbat commented Jan 26, 2022

@vEpiphyte thanks for the steps. I can see the hang on my Mac. If I remove pytest-cov from the command, then the hang goes away. That is, I remove --cov synapse --no-cov-on-fail from the command line, and change from:

$ python -m pytest ...

to:

$ python -m coverage run -m pytest ...

@tunetheweb It looks like you are also using pytest-cov. If you remove it, and run under coverage, do you still see the problem?

@vEpiphyte
Copy link

vEpiphyte commented Jan 26, 2022

@nedbat Thanks for that tip about pytest-cov vs raw coverage. That worked as expected - there was no lock when I ran directly with coverage. I've also reduced the tests down to a single test and pushed that test file up to a branch here
https://github.com/vertexproject/synapse/blob/5a80408d1e8f75c3ab13df48a8845da7c8ab2051/synapse/tests/test_cov_fail.py

The following command hangs like we've seen already.

python -m pytest -v -s -rs --durations 6 --maxfail 6 -p no:logging --junitxml=test-reports/junit.xml --cov synapse --no-cov-on-fail synapse/tests/test_cov_fail.py 

The following command works as you demonstrated.

python -m coverage run -m pytest -v -s -rs --durations 6 --maxfail 6 -p no:logging --junitxml=test-reports/junit.xml synapse/tests/test_cov_fail.py 

This particular test does have the effect of testing lots of multiprocessing targets to test a bunch of error handling and bad-path logic.

@tunetheweb
Copy link
Author

@tunetheweb It looks like you are also using pytest-cov. If you remove it, and run under coverage, do you still see the problem?

I can confirm for me the problem does not arise for me when running 6.3 outside of pytest.

@vEpiphyte
Copy link

Using my test file with pytest-cov and git bisect on coveragepy repo points to the problem starting to appear with this commit dd575ee

Patching the sigterm handler

    def _on_sigterm(self, signum_unused, frame_unused):
        """A handler for signal.SIGTERM."""
        pid = os.getpid()
        print(f'IN on_sigterm {pid=}')
        
        self._atexit("sigterm")
        # Statements after here won't be seen by metacov because we just wrote
        # the data, and are about to kill the process.
        signal.signal(signal.SIGTERM, self._old_sigterm)    # pragma: not covered
        print(f'FINI on_sigterm {pid=}')
        os.kill(pid, signal.SIGTERM)                # pragma: not covered

seems to stop at the signal.signal() call in the forkserver process. We never print the FINI ... message :(

user    9428  4.7  0.2 141008 88796 pts/2    S+   17:09   0:01 /home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python -c from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=7, pipe_handle=61) --multiprocessing-fork

@nedbat
Copy link
Owner

nedbat commented Jan 26, 2022

@vEpiphyte Thanks for the bisect and the debugging! Would you mind wrapping a try/except around that signal.signal line and seeing what's getting raised?
Also, if you could run coverage with a COVERAGE_DEBUG=process,pid environment variable, there will be more debugging output. You might need -s on the pytest line to get it all.

@vEpiphyte
Copy link

The signal.signal didn't raise an exception (or at least not one caught with a bare try/except block). Here is my output:

$ COVERAGE_DEBUG=process,pid python -m pytest -v -s -rs --durations 6 --maxfail 6 -p no:logging --junitxml=test-reports/junit.xml --cov synapse --no-cov-on-fail synapse/tests/test_cov_fail.py 
10949.6f92: cwd is now '/home/user/PycharmProjects/synapse'
10949.6f92: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10949.6f92: New process: cmd: ['/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest/__main__.py', '-v', '-s', '-rs', '--durations', '6', '--maxfail', '6', '-p', 'no:logging', '--junitxml=test-reports/junit.xml', '--cov', 'synapse', '--no-cov-on-fail', 'synapse/tests/test_cov_fail.py']
10949.6f92: New process: pid: 10949, parent pid: 4785
========================================================================================= test session starts =========================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-0.13.1 -- /home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python
cachedir: .pytest_cache
rootdir: /home/user/PycharmProjects/synapse, configfile: pytest.ini
plugins: forked-1.3.0, anyio-3.3.1, xdist-2.5.0, cov-3.0.0
collecting ... 10951.dea1: cwd is now '/home/user/PycharmProjects/synapse'
10951.dea1: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10951.dea1: New process: cmd: ['-c']
10951.dea1: New process: pid: 10951, parent pid: 10949
collected 1 item                                                                                                                                                                                      

synapse/tests/test_cov_fail.py::CovFailTest::test_cell_stream_backup 10955.ed98: cwd is now '/home/user/PycharmProjects/synapse'
10955.ed98: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10955.ed98: New process: cmd: ['-c']
10955.ed98: New process: pid: 10955, parent pid: 10949
error during task iterBackupArchive
Traceback (most recent call last):
  File "/home/user/PycharmProjects/synapse/synapse/daemon.py", line 138, in t2call
    async for item in valu:
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 675, in iterBackupArchive
    await self.cell.iterBackupArchive(name, user=self.user)
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 1572, in iterBackupArchive
    raise s_exc.BadArg(mesg=mesg, arg='path', valu=path)
synapse.exc.BadArg: BadArg: arg='path' mesg='Specified backup path has no cell.guid file.' valu='/tmp/tmpcxvtnpu0/backups/nope'
10962.2919: cwd is now '/home/user/PycharmProjects/synapse'
10962.2919: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10962.2919: New process: cmd: ['-c', '--multiprocessing-fork']
10962.2919: New process: pid: 10962, parent pid: 10949
10980.e8f4: cwd is now '/home/user/PycharmProjects/synapse'
10980.e8f4: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10980.e8f4: New process: cmd: ['-c', '--multiprocessing-fork']
10980.e8f4: New process: pid: 10980, parent pid: 10949
IN on_sigterm pid=10980
10980.e8f4: sigterm: pid: 10980, instance: <coverage.control.Coverage object at 0x7f060047e610>
FINI on_sigterm pid=10980
IN on_sigterm pid=10980
10980.e8f4: sigterm: pid: 10980, instance: <coverage.control.Coverage object at 0x7f060047e700>
FINI on_sigterm pid=10980
10982.e52e: cwd is now '/home/user/PycharmProjects/synapse'
10982.e52e: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10982.e52e: New process: cmd: ['-c', '--multiprocessing-fork']
10982.e52e: New process: pid: 10982, parent pid: 10949
IN on_sigterm pid=10982
10982.e52e: sigterm: pid: 10982, instance: <coverage.control.Coverage object at 0x7f1bbd0a7610>
FINI on_sigterm pid=10982
IN on_sigterm pid=10982
10982.e52e: sigterm: pid: 10982, instance: <coverage.control.Coverage object at 0x7f1bbd0a7700>
FINI on_sigterm pid=10982
error during task iterNewBackupArchive
Traceback (most recent call last):
  File "/home/user/PycharmProjects/synapse/synapse/daemon.py", line 138, in t2call
    async for item in valu:
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 692, in iterNewBackupArchive
    await self.cell.iterNewBackupArchive(user=self.user, name=name, remove=remove)
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 1625, in iterNewBackupArchive
    raise s_exc.BadArg(mesg=mesg)
synapse.exc.BadArg: BadArg: mesg='Backup with name already exists'
10983.0649: cwd is now '/home/user/PycharmProjects/synapse'
10983.0649: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10983.0649: New process: cmd: ['-c', '--multiprocessing-fork']
10983.0649: New process: pid: 10983, parent pid: 10949
10987.51f1: cwd is now '/home/user/PycharmProjects/synapse'
10987.51f1: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
10987.51f1: New process: cmd: ['-c', '--multiprocessing-fork']
10987.51f1: New process: pid: 10987, parent pid: 10949
11005.18da: cwd is now '/home/user/PycharmProjects/synapse'
11005.18da: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11005.18da: New process: cmd: ['-c', '--multiprocessing-fork']
11005.18da: New process: pid: 11005, parent pid: 10949
11009.15a2: cwd is now '/home/user/PycharmProjects/synapse'
11009.15a2: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11009.15a2: New process: cmd: ['-c', '--multiprocessing-fork']
11009.15a2: New process: pid: 11009, parent pid: 10949
11027.82f1: cwd is now '/home/user/PycharmProjects/synapse'
11027.82f1: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11027.82f1: New process: cmd: ['-c', '--multiprocessing-fork']
11027.82f1: New process: pid: 11027, parent pid: 10949
11031.4f81: cwd is now '/home/user/PycharmProjects/synapse'
11031.4f81: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11031.4f81: New process: cmd: ['-c', '--multiprocessing-fork']
11031.4f81: New process: pid: 11031, parent pid: 10949
11049.61f4: cwd is now '/home/user/PycharmProjects/synapse'
11049.61f4: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11049.61f4: New process: cmd: ['-c', '--multiprocessing-fork']
11049.61f4: New process: pid: 11049, parent pid: 10949
error during task iterNewBackupArchive
Traceback (most recent call last):
  File "/home/user/PycharmProjects/synapse/synapse/daemon.py", line 138, in t2call
    async for item in valu:
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 692, in iterNewBackupArchive
    await self.cell.iterNewBackupArchive(user=self.user, name=name, remove=remove)
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 1632, in iterNewBackupArchive
    await self.runBackup(name)
  File "/home/user/PycharmProjects/synapse/synapse/lib/cell.py", line 1361, in runBackup
    raise s_exc.BackupAlreadyRunning(mesg='Another backup is already running')
synapse.exc.BackupAlreadyRunning: BackupAlreadyRunning: mesg='Another backup is already running'
11053.881e: cwd is now '/home/user/PycharmProjects/synapse'
11053.881e: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11053.881e: New process: cmd: ['-c', '--multiprocessing-fork']
11053.881e: New process: pid: 11053, parent pid: 10949
11072.aa01: cwd is now '/home/user/PycharmProjects/synapse'
11072.aa01: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11072.aa01: New process: cmd: ['-c', '--multiprocessing-fork']
11072.aa01: New process: pid: 11072, parent pid: 10949
IN on_sigterm pid=11072
11072.aa01: sigterm: pid: 11072, instance: <coverage.control.Coverage object at 0x7f1d1da5dc40>
FINI on_sigterm pid=11072
IN on_sigterm pid=11072
11072.aa01: sigterm: pid: 11072, instance: <coverage.control.Coverage object at 0x7f1d21f24610>
FINI on_sigterm pid=11072
IN on_sigterm pid=11072
11072.aa01: sigterm: pid: 11072, instance: <coverage.control.Coverage object at 0x7f1d21f24700>
FINI on_sigterm pid=11072
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup/axon/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup2/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup2/axon/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup3/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/bkup3/axon/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/20220126172444/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
Migrating Nexus log v1->v2 for /tmp/tmpcxvtnpu0/20220126172444/axon/slabs/nexus.lmdb
Nothing in the nexuslog sequence to migrate.
...Nexus log migration complete
11076.8294: cwd is now '/home/user/PycharmProjects/synapse'
11076.8294: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11076.8294: New process: cmd: ['-c', '--multiprocessing-fork']
11076.8294: New process: pid: 11076, parent pid: 10949
11094.4f88: cwd is now '/home/user/PycharmProjects/synapse'
11094.4f88: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11094.4f88: New process: cmd: ['-c', '--multiprocessing-fork']
11094.4f88: New process: pid: 11094, parent pid: 10949
11098.e216: cwd is now '/home/user/PycharmProjects/synapse'
11098.e216: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11098.e216: New process: cmd: ['-c', '--multiprocessing-fork']
11098.e216: New process: pid: 11098, parent pid: 10949
11122.50a7: cwd is now '/home/user/PycharmProjects/synapse'
11122.50a7: New process: executable: '/home/user/.pyenv/versions/3.8.10/envs/syn3810/bin/python'
11122.50a7: New process: cmd: ['-c', '--multiprocessing-fork']
11122.50a7: New process: pid: 11122, parent pid: 10949
IN on_sigterm pid=11122
11122.50a7: sigterm: pid: 11122, instance: <coverage.control.Coverage object at 0x7fe6c0ebad90>
PASSED

========================================================================================== warnings summary ===========================================================================================
../../.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
  /home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: log_cli
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

../../.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
  /home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: log_cli_date_format
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

../../.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
  /home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: log_cli_format
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

../../.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233
  /home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/_pytest/config/__init__.py:1233: PytestConfigWarning: Unknown config option: log_cli_level
  
    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
---------------------------------------------------------- generated xml file: /home/user/PycharmProjects/synapse/test-reports/junit.xml ----------------------------------------------------------

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name                                                   Stmts   Miss  Cover
--------------------------------------------------------------------------
synapse/__init__.py                                        8      3    62%

..... snip ...

--------------------------------------------------------------------------
TOTAL                                                  37535  25690    32%

========================================================================================= slowest 6 durations =========================================================================================
13.52s call     synapse/tests/test_cov_fail.py::CovFailTest::test_cell_stream_backup
0.00s setup    synapse/tests/test_cov_fail.py::CovFailTest::test_cell_stream_backup
0.00s teardown synapse/tests/test_cov_fail.py::CovFailTest::test_cell_stream_backup
=================================================================================== 1 passed, 4 warnings in 17.61s ====================================================================================
10949.6f92: atexit: pid: 10949, instance: <coverage.control.Coverage object at 0x7f07160a3910>



^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
Process SpawnProcess-17:
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/process.py", line 318, in _bootstrap
    util._exit_function()
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 334, in _exit_function
    _run_finalizers(0)
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/home/user/.pyenv/versions/3.8.10/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 99, in cleanup
    _cleanup(_active_cov)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/pytest_cov/embed.py", line 84, in _cleanup
    cov.stop()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 708, in save
    data = self.get_data()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 777, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 236, in _wrapped
    return method(self, *args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 459, in add_lines
    self._choose_lines_or_arcs(lines=True)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 518, in _choose_lines_or_arcs
    with self._connect() as con:
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 313, in _connect
    self._open_db()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 264, in _open_db
    self._read_db()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 268, in _read_db
    with self._dbs[threading.get_ident()] as db:
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1073, in __enter__
    self._connect()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1052, in _connect
    self.con = sqlite3.connect(self.filename, check_same_thread=False)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 611, in _on_sigterm
    self._atexit("sigterm")
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 604, in _atexit
    self.save()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 708, in save
    data = self.get_data()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/control.py", line 777, in get_data
    if self._collector and self._collector.flush_data():
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/collector.py", line 473, in flush_data
    self.covdata.add_lines(self.mapped_file_dict(self.data))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 234, in _wrapped
    with self._lock:
KeyboardInterrupt
11122.50a7: atexit: pid: 11122, instance: <coverage.control.Coverage object at 0x7fe6c0ebad90>
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1096, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: context

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1101, in execute
    return self.con.execute(sql, parameters)
sqlite3.OperationalError: no such table: context

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 236, in _wrapped
    return method(self, *args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 463, in add_lines
    self._set_context_id()
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 421, in _set_context_id
    context_id = self._context_id(context)
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 397, in _context_id
    row = con.execute_one("select id from context where context = ?", (context,))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1129, in execute_one
    rows = list(self.execute(sql, parameters))
  File "/home/user/.pyenv/versions/3.8.10/envs/syn3810/lib/python3.8/site-packages/coverage/sqldata.py", line 1118, in execute
    raise DataError(f"Couldn't use data file {self.filename!r}: {msg}") from exc
coverage.exceptions.DataError: Couldn't use data file '/home/user/PycharmProjects/synapse/.coverage.vertex005c.11122.270781': no such table: context
(syn3810) user@vertex005c:~/PycharmProjects/synapse$ 10951.dea1: atexit: pid: 10951, instance: <coverage.control.Coverage object at 0x7fab05dff610>
10955.ed98: atexit: pid: 10955, instance: <coverage.control.Coverage object at 0x7fce74ac3610>

@nedbat nedbat changed the title Possible issues with Coverage 6.3 instability Process hang with Coverage 6.3 Jan 26, 2022
@nmdefries
Copy link

nmdefries commented Jan 27, 2022

This is caused by running code coverage on a function that uses multiprocessing.Pool to fork threads.

The test suite hangs on this line of coverage, specifically:

".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 234 in _wrapped

Child threads are unable to acquire an available mutex; when they are created, they are given a copy of an (unavailable) mutex that is thus never updated (some info about why). Probably related to this change "Feature: coverage measurement data will now be written when a SIGTERM signal is received by the process."

Fixed in my case by any of: using multiprocessing to "spawn" instead of forking, reverting to version 6.2.0, or turning off code coverage.

Edit:
Full stack trace of stalled thread

Current thread 0x00007f5675b14740 (most recent call first):
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 234 in _wrapped
  File ".../env/lib/python3.8/site-packages/coverage/collector.py", line 473 in flush_data
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 766 in get_data
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 697 in save
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 601 in _atexit
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 605 in _on_sigterm
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 1156 in executescript
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 299 in _init_db
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 273 in _read_db
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 264 in _open_db
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 313 in _connect
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 518 in _choose_lines_or_arcs
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 459 in add_lines
  File ".../env/lib/python3.8/site-packages/coverage/sqldata.py", line 236 in _wrapped
  File ".../env/lib/python3.8/site-packages/coverage/collector.py", line 473 in flush_data
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 766 in get_data
  File ".../env/lib/python3.8/site-packages/coverage/control.py", line 697 in save
  File ".../env/lib/python3.8/site-packages/pytest_cov/embed.py", line 85 in _cleanup
  File ".../env/lib/python3.8/site-packages/pytest_cov/embed.py", line 99 in cleanup
  File "/usr/lib/python3.8/multiprocessing/util.py", line 224 in __call__
  File "/usr/lib/python3.8/multiprocessing/util.py", line 300 in _run_finalizers
  File "/usr/lib/python3.8/multiprocessing/util.py", line 334 in _exit_function
  File "/usr/lib/python3.8/multiprocessing/process.py", line 318 in _bootstrap
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen
  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 326 in _repopulate_pool_static
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 303 in _repopulate_pool
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 212 in __init__
  File "/usr/lib/python3.8/multiprocessing/context.py", line 119 in Pool
  File ".../delphi_safegraph_patterns/run.py", line 108 in run_module
  File ".../tests/test_run.py", line 36 in test_output_files
  File "/usr/lib/python3.8/unittest/mock.py", line 1325 in patched
  File ".../env/lib/python3.8/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File ".../env/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File ".../env/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File ".../env/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File ".../env/lib/python3.8/site-packages/_pytest/python.py", line 1641 in runtest
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File ".../env/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File ".../env/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File ".../env/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 255 in <lambda>
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 215 in call_and_report
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File ".../env/lib/python3.8/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File ".../env/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File ".../env/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File ".../env/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File ".../env/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File ".../env/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File ".../env/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File ".../env/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File ".../env/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File ".../env/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File ".../env/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File ".../env/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall
  File ".../env/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File ".../env/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__
  File ".../env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File ".../env/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "../env/bin/pytest", line 8 in <module>

@nedbat
Copy link
Owner

nedbat commented May 18, 2022

I've made the SIGTERM handler opt-in, so these issues should now be fixed. Commited in 803a549

@pllim
Copy link

pllim commented May 18, 2022

Thanks for the fix! Which release will this be in? 🙏

@nedbat
Copy link
Owner

nedbat commented May 18, 2022

This will become a 6.4 release, though I'm not sure when. It would be great if people could do a test with the commit from GitHub:

pip install git+https://github.com/nedbat/coveragepy.git@803a5494ef23#egg=coverage

(this will claim a version of 6.3.4a0, which is fine.)
If something still seems amiss, please open a new issue.

@asottile
Copy link
Contributor

can confirm it works with my minimal reproduction above 🎉 (and again fails with [run] sigterm = true as expected!)

@tunetheweb
Copy link
Author

Can confirm I can see no issues with my pipeline with this branch either.

Great to get this resolve so can unpin the version number on the next release. Thanks for your efforts!

anishathalye added a commit to anishathalye/cleanlab that referenced this issue May 19, 2022
The issue with coverage==6.3 has been fixed:
nedbat/coveragepy#1310 (comment).
@nedbat
Copy link
Owner

nedbat commented May 22, 2022

This is now released as part of coverage 6.4.

@johnthagen
Copy link

@nedbat I can confirm that Coverage 6.4 solved our multiprocessing.Pool deadlocks in CI.

Thanks! ❤️

anishathalye added a commit to anishathalye/cleanlab that referenced this issue May 23, 2022
The issue that was introduced in coverage 6.3 has been fixed:
nedbat/coveragepy#1310 (comment).
We can't just upgrade to `coverage` or `coverage>=6.4` because the
former could install bad versions of coverage (e.g. 6.3), and the latter
is unsupported on Python 3.6, which we want to continue supporting.

This patch just bans coverage 6.3 / 6.3.x, so with Python 3.7+, we'll
use the latest version of coverage, and with Python 3.6, we'll use the
latest supported version of coverage that's not a 6.3 release.
@pllim
Copy link

pllim commented May 23, 2022

v6.4 works for me. Thanks!

jwmueller pushed a commit to cleanlab/cleanlab that referenced this issue May 25, 2022
The issue that was introduced in coverage 6.3 has been fixed:
nedbat/coveragepy#1310 (comment).
We can't just upgrade to `coverage` or `coverage>=6.4` because the
former could install bad versions of coverage (e.g. 6.3), and the latter
is unsupported on Python 3.6, which we want to continue supporting.

This patch just bans coverage 6.3 / 6.3.x, so with Python 3.7+, we'll
use the latest version of coverage, and with Python 3.6, we'll use the
latest supported version of coverage that's not a 6.3 release.
QuLogic added a commit to QuLogic/matplotlib that referenced this issue May 25, 2022
The upstream issue [1] has been fixed and release in 6.4.

[1] nedbat/coveragepy#1310
andrew-fennell pushed a commit to andrew-fennell/matplotlib that referenced this issue Jun 14, 2022
The upstream issue [1] has been fixed and release in 6.4.

[1] nedbat/coveragepy#1310
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this issue Aug 24, 2022
Version 6.4.4 — 2022-08-16
--------------------------

- Wheels are now provided for Python 3.11.


.. _changes_6-4-3:

Version 6.4.3 — 2022-08-06
--------------------------

- Fix a failure when combining data files if the file names contained
  glob-like patterns (`pull 1405`_).  Thanks, Michael Krebs and Benjamin
  Schubert.

- Fix a messaging failure when combining Windows data files on a different
  drive than the current directory. (`pull 1430`_, fixing `issue 1428`_).
  Thanks, Lorenzo Micò.

- Fix path calculations when running in the root directory, as you might do in
  a Docker container: `pull 1403`_, thanks Arthur Rio.

- Filtering in the HTML report wouldn't work when reloading the index page.
  This is now fixed (`pull 1413`_).  Thanks, Marc Legendre.

- Fix a problem with Cython code measurement (`pull 1347`_, fixing `issue
  972`_).  Thanks, Matus Valo.

.. _issue 972: nedbat/coveragepy#972
.. _pull 1347: nedbat/coveragepy#1347
.. _pull 1403: nedbat/coveragepy#1403
.. _pull 1405: nedbat/coveragepy#1405
.. _pull 1413: nedbat/coveragepy#1413
.. _issue 1428: nedbat/coveragepy#1428
.. _pull 1430: nedbat/coveragepy#1430


.. _changes_6-4-2:

Version 6.4.2 — 2022-07-12
--------------------------

- Updated for a small change in Python 3.11.0 beta 4: modules now start with a
  line with line number 0, which is ignored.  This line cannnot be executed, so
  coverage totals were thrown off.  This line is now ignored by coverage.py,
  but this also means that truly empty modules (like ``__init__.py``) have no
  lines in them, rather than one phantom line.  Fixes `issue 1419`_.

- Internal debugging data added to sys.modules is now an actual module, to
  avoid confusing code that examines everything in sys.modules.  Thanks,
  Yilei Yang (`pull 1399`_).

.. _pull 1399: nedbat/coveragepy#1399
.. _issue 1419: nedbat/coveragepy#1419


.. _changes_6-4-1:

Version 6.4.1 — 2022-06-02
--------------------------

- Greatly improved performance on PyPy, and other environments that need the
  pure Python trace function.  Thanks, Carl Friedrich Bolz-Tereick (`pull
  1381`_ and `pull 1388`_).  Slightly improved performance when using the C
  trace function, as most environments do.  Closes `issue 1339`_.

- The conditions for using tomllib from the standard library have been made
  more precise, so that 3.11 alphas will continue to work. Closes `issue
  1390`_.

.. _issue 1339: nedbat/coveragepy#1339
.. _pull 1381: nedbat/coveragepy#1381
.. _pull 1388: nedbat/coveragepy#1388
.. _issue 1390: nedbat/coveragepy#1390


.. _changes_64:

Version 6.4 — 2022-05-22
------------------------

- A new setting, :ref:`config_run_sigterm`, controls whether a SIGTERM signal
  handler is used.  In 6.3, the signal handler was always installed, to capture
  data at unusual process ends.  Unfortunately, this introduced other problems
  (see `issue 1310`_).  Now the signal handler is only used if you opt-in by
  setting ``[run] sigterm = true``.

- Small changes to the HTML report:

  - Added links to next and previous file, and more keyboard shortcuts: ``[``
    and ``]`` for next file and previous file; ``u`` for up to the index; and
    ``?`` to open/close the help panel.  Thanks, `J. M. F. Tsang
    <pull 1364_>`_.

  - The timestamp and version are displayed at the top of the report.  Thanks,
    `Ammar Askar <pull 1354_>`_. Closes `issue 1351`_.

- A new debug option ``debug=sqldata`` adds more detail to ``debug=sql``,
  logging all the data being written to the database.

- Previously, running ``coverage report`` (or any of the reporting commands) in
  an empty directory would create a .coverage data file.  Now they do not,
  fixing `issue 1328`_.

- On Python 3.11, the ``[toml]`` extra no longer installs tomli, instead using
  tomllib from the standard library.  Thanks `Shantanu <pull 1359_>`_.

- In-memory CoverageData objects now properly update(), closing `issue 1323`_.

.. _issue 1310: nedbat/coveragepy#1310
.. _issue 1323: nedbat/coveragepy#1323
.. _issue 1328: nedbat/coveragepy#1328
.. _issue 1351: nedbat/coveragepy#1351
.. _pull 1354: nedbat/coveragepy#1354
.. _pull 1359: nedbat/coveragepy#1359
.. _pull 1364: nedbat/coveragepy#1364


.. _changes_633:

Version 6.3.3 — 2022-05-12
--------------------------

- Fix: Coverage.py now builds successfully on CPython 3.11 (3.11.0b1) again.
  Closes `issue 1367`_.  Some results for generators may have changed.

.. _issue 1367: nedbat/coveragepy#1367


.. _changes_632:

Version 6.3.2 — 2022-02-20
--------------------------

- Fix: adapt to pypy3.9's decorator tracing behavior.  It now traces function
  decorators like CPython 3.8: both the @-line and the def-line are traced.
  Fixes `issue 1326`_.

- Debug: added ``pybehave`` to the list of :ref:`coverage debug <cmd_debug>`
  and :ref:`cmd_run_debug` options.

- Fix: show an intelligible error message if ``--concurrency=multiprocessing``
  is used without a configuration file.  Closes `issue 1320`_.

.. _issue 1320: nedbat/coveragepy#1320
.. _issue 1326: nedbat/coveragepy#1326


.. _changes_631:

Version 6.3.1 — 2022-02-01
--------------------------

- Fix: deadlocks could occur when terminating processes.  Some of these
  deadlocks (described in `issue 1310`_) are now fixed.

- Fix: a signal handler was being set from multiple threads, causing an error:
  "ValueError: signal only works in main thread".  This is now fixed, closing
  `issue 1312`_.

- Fix: ``--precision`` on the command-line was being ignored while considering
  ``--fail-under``.  This is now fixed, thanks to
  `Marcelo Trylesinski <pull 1317_>`_.

- Fix: releases no longer provide 3.11.0-alpha wheels. Coverage.py uses CPython
  internal fields which are moving during the alpha phase. Fixes `issue 1316`_.

.. _issue 1310: nedbat/coveragepy#1310
.. _issue 1312: nedbat/coveragepy#1312
.. _issue 1316: nedbat/coveragepy#1316
.. _pull 1317: nedbat/coveragepy#1317


.. _changes_63:

Version 6.3 — 2022-01-25
------------------------

- Feature: Added the ``lcov`` command to generate reports in LCOV format.
  Thanks, `Bradley Burns <pull 1289_>`_. Closes issues `587 <issue 587_>`_
  and `626 <issue 626_>`_.

- Feature: the coverage data file can now be specified on the command line with
  the ``--data-file`` option in any command that reads or writes data.  This is
  in addition to the existing ``COVERAGE_FILE`` environment variable.  Closes
  `issue 624`_. Thanks, `Nikita Bloshchanevich <pull 1304_>`_.

- Feature: coverage measurement data will now be written when a SIGTERM signal
  is received by the process.  This includes
  :meth:`Process.terminate <python:multiprocessing.Process.terminate>`,
  and other ways to terminate a process.  Currently this is only on Linux and
  Mac; Windows is not supported.  Fixes `issue 1307`_.

- Dropped support for Python 3.6, which reached end-of-life on 2021-12-23.

- Updated Python 3.11 support to 3.11.0a4, fixing `issue 1294`_.

- Fix: the coverage data file is now created in a more robust way, to avoid
  problems when multiple processes are trying to write data at once. Fixes
  issues `1303 <issue 1303_>`_ and `883 <issue 883_>`_.

- Fix: a .gitignore file will only be written into the HTML report output
  directory if the directory is empty.  This should prevent certain unfortunate
  accidents of writing the file where it is not wanted.

- Releases now have MacOS arm64 wheels for Apple Silicon, fixing `issue 1288`_.

.. _issue 587: nedbat/coveragepy#587
.. _issue 624: nedbat/coveragepy#624
.. _issue 626: nedbat/coveragepy#626
.. _issue 883: nedbat/coveragepy#883
.. _issue 1288: nedbat/coveragepy#1288
.. _issue 1294: nedbat/coveragepy#1294
.. _issue 1303: nedbat/coveragepy#1303
.. _issue 1307: nedbat/coveragepy#1307
.. _pull 1289: nedbat/coveragepy#1289
.. _pull 1304: nedbat/coveragepy#1304


.. _changes_62:

Version 6.2 — 2021-11-26
------------------------

- Feature: Now the ``--concurrency`` setting can now have a list of values, so
  that threads and another lightweight threading package can be measured
  together, such as ``--concurrency=gevent,thread``.  Closes `issue 1012`_ and
  `issue 1082`_.

- Fix: A module specified as the ``source`` setting is imported during startup,
  before the user program imports it.  This could cause problems if the rest of
  the program isn't ready yet.  For example, `issue 1203`_ describes a Django
  setting that is accessed before settings have been configured.  Now the early
  import is wrapped in a try/except so errors then don't stop execution.

- Fix: A colon in a decorator expression would cause an exclusion to end too
  early, preventing the exclusion of the decorated function. This is now fixed.

- Fix: The HTML report now will not overwrite a .gitignore file that already
  exists in the HTML output directory (follow-on for `issue 1244`_).

- API: The exceptions raised by Coverage.py have been specialized, to provide
  finer-grained catching of exceptions by third-party code.

- API: Using ``suffix=False`` when constructing a Coverage object with
  multiprocessing wouldn't suppress the data file suffix (`issue 989`_).  This
  is now fixed.

- Debug: The ``coverage debug data`` command will now sniff out combinable data
  files, and report on all of them.

- Debug: The ``coverage debug`` command used to accept a number of topics at a
  time, and show all of them, though this was never documented.  This no longer
  works, to allow for command-line options in the future.

.. _issue 989: nedbat/coveragepy#989
.. _issue 1012: nedbat/coveragepy#1012
.. _issue 1082: nedbat/coveragepy#1082
.. _issue 1203: nedbat/coveragepy#1203


.. _changes_612:

Version 6.1.2 — 2021-11-10
--------------------------

- Python 3.11 is supported (tested with 3.11.0a2).  One still-open issue has to
  do with `exits through with-statements <issue 1270_>`_.

- Fix: When remapping file paths through the ``[paths]`` setting while
  combining, the ``[run] relative_files`` setting was ignored, resulting in
  absolute paths for remapped file names (`issue 1147`_).  This is now fixed.

- Fix: Complex conditionals over excluded lines could have incorrectly reported
  a missing branch (`issue 1271`_). This is now fixed.

- Fix: More exceptions are now handled when trying to parse source files for
  reporting.  Problems that used to terminate coverage.py can now be handled
  with ``[report] ignore_errors``.  This helps with plugins failing to read
  files (`django_coverage_plugin issue 78`_).

- Fix: Removed another vestige of jQuery from the source tarball
  (`issue 840`_).

- Fix: Added a default value for a new-to-6.x argument of an internal class.
  This unsupported class is being used by coveralls (`issue 1273`_). Although
  I'd rather not "fix" unsupported interfaces, it's actually nicer with a
  default value.

.. _django_coverage_plugin issue 78: nedbat/django_coverage_plugin#78
.. _issue 1147: nedbat/coveragepy#1147
.. _issue 1270: nedbat/coveragepy#1270
.. _issue 1271: nedbat/coveragepy#1271
.. _issue 1273: nedbat/coveragepy#1273


.. _changes_611:

Version 6.1.1 — 2021-10-31
--------------------------

- Fix: The sticky header on the HTML report didn't work unless you had branch
  coverage enabled. This is now fixed: the sticky header works for everyone.
  (Do people still use coverage without branch measurement!? j/k)

- Fix: When using explicitly declared namespace packages, the "already imported
  a file that will be measured" warning would be issued (`issue 888`_).  This
  is now fixed.

.. _issue 888: nedbat/coveragepy#888


.. _changes_61:

Version 6.1 — 2021-10-30
------------------------

- Deprecated: The ``annotate`` command and the ``Coverage.annotate`` function
  will be removed in a future version, unless people let me know that they are
  using it.  Instead, the ``html`` command gives better-looking (and more
  accurate) output, and the ``report -m`` command will tell you line numbers of
  missing lines.  Please get in touch if you have a reason to use ``annotate``
  over those better options: [email protected].

- Feature: Coverage now sets an environment variable, ``COVERAGE_RUN`` when
  running your code with the ``coverage run`` command.  The value is not
  important, and may change in the future.  Closes `issue 553`_.

- Feature: The HTML report pages for Python source files now have a sticky
  header so the file name and controls are always visible.

- Feature: The ``xml`` and ``json`` commands now describe what they wrote
  where.

- Feature: The ``html``, ``combine``, ``xml``, and ``json`` commands all accept
  a ``-q/--quiet`` option to suppress the messages they write to stdout about
  what they are doing (`issue 1254`_).

- Feature: The ``html`` command writes a ``.gitignore`` file into the HTML
  output directory, to prevent the report from being committed to git.  If you
  want to commit it, you will need to delete that file.  Closes `issue 1244`_.

- Feature: Added support for PyPy 3.8.

- Fix: More generated code is now excluded from measurement.  Code such as
  `attrs`_ boilerplate, or doctest code, was being measured though the
  synthetic line numbers meant they were never reported.  Once Cython was
  involved though, the generated .so files were parsed as Python, raising
  syntax errors, as reported in `issue 1160`_.  This is now fixed.

- Fix: When sorting human-readable names, numeric components are sorted
  correctly: file10.py will appear after file9.py.  This applies to file names,
  module names, environment variables, and test contexts.

- Performance: Branch coverage measurement is faster, though you might only
  notice on code that is executed many times, such as long-running loops.

- Build: jQuery is no longer used or vendored (`issue 840`_ and `issue 1118`_).
  Huge thanks to Nils Kattenbeck (septatrix) for the conversion to vanilla
  JavaScript in `pull request 1248`_.

.. _issue 553: nedbat/coveragepy#553
.. _issue 840: nedbat/coveragepy#840
.. _issue 1118: nedbat/coveragepy#1118
.. _issue 1160: nedbat/coveragepy#1160
.. _issue 1244: nedbat/coveragepy#1244
.. _pull request 1248: nedbat/coveragepy#1248
.. _issue 1254: nedbat/coveragepy#1254
.. _attrs: https://www.attrs.org/


.. _changes_602:

Version 6.0.2 — 2021-10-11
--------------------------

- Namespace packages being measured weren't properly handled by the new code
  that ignores third-party packages. If the namespace package was installed, it
  was ignored as a third-party package.  That problem (`issue 1231`_) is now
  fixed.

- Packages named as "source packages" (with ``source``, or ``source_pkgs``, or
  pytest-cov's ``--cov``) might have been only partially measured.  Their
  top-level statements could be marked as unexecuted, because they were
  imported by coverage.py before measurement began (`issue 1232`_).  This is
  now fixed, but the package will be imported twice, once by coverage.py, then
  again by your test suite.  This could cause problems if importing the package
  has side effects.

- The :meth:`.CoverageData.contexts_by_lineno` method was documented to return
  a dict, but was returning a defaultdict.  Now it returns a plain dict.  It
  also no longer returns negative numbered keys.

.. _issue 1231: nedbat/coveragepy#1231
.. _issue 1232: nedbat/coveragepy#1232


.. _changes_601:

Version 6.0.1 — 2021-10-06
--------------------------

- In 6.0, the coverage.py exceptions moved from coverage.misc to
  coverage.exceptions. These exceptions are not part of the public supported
  API, CoverageException is. But a number of other third-party packages were
  importing the exceptions from coverage.misc, so they are now available from
  there again (`issue 1226`_).

- Changed an internal detail of how tomli is imported, so that tomli can use
  coverage.py for their own test suite (`issue 1228`_).

- Defend against an obscure possibility under code obfuscation, where a
  function can have an argument called "self", but no local named "self"
  (`pull request 1210`_).  Thanks, Ben Carlsson.

.. _pull request 1210: nedbat/coveragepy#1210
.. _issue 1226: nedbat/coveragepy#1226
.. _issue 1228: nedbat/coveragepy#1228


.. _changes_60:

Version 6.0 — 2021-10-03
------------------------

- The ``coverage html`` command now prints a message indicating where the HTML
  report was written.  Fixes `issue 1195`_.

- The ``coverage combine`` command now prints messages indicating each data
  file being combined.  Fixes `issue 1105`_.

- The HTML report now includes a sentence about skipped files due to
  ``skip_covered`` or ``skip_empty`` settings.  Fixes `issue 1163`_.

- Unrecognized options in the configuration file are no longer errors. They are
  now warnings, to ease the use of coverage across versions.  Fixes `issue
  1035`_.

- Fix handling of exceptions through context managers in Python 3.10. A missing
  exception is no longer considered a missing branch from the with statement.
  Fixes `issue 1205`_.

- Fix another rarer instance of "Error binding parameter 0 - probably
  unsupported type." (`issue 1010`_).

- Creating a directory for the coverage data file now is safer against
  conflicts when two coverage runs happen simultaneously (`pull 1220`_).
  Thanks, Clément Pit-Claudel.

.. _issue 1035: nedbat/coveragepy#1035
.. _issue 1105: nedbat/coveragepy#1105
.. _issue 1163: nedbat/coveragepy#1163
.. _issue 1195: nedbat/coveragepy#1195
.. _issue 1205: nedbat/coveragepy#1205
.. _pull 1220: nedbat/coveragepy#1220


.. _changes_60b1:

Version 6.0b1 — 2021-07-18
--------------------------

- Dropped support for Python 2.7, PyPy 2, and Python 3.5.

- Added support for the Python 3.10 ``match/case`` syntax.

- Data collection is now thread-safe.  There may have been rare instances of
  exceptions raised in multi-threaded programs.

- Plugins (like the `Django coverage plugin`_) were generating "Already
  imported a file that will be measured" warnings about Django itself.  These
  have been fixed, closing `issue 1150`_.

- Warnings generated by coverage.py are now real Python warnings.

- Using ``--fail-under=100`` with coverage near 100% could result in the
  self-contradictory message :code:`total of 100 is less than fail-under=100`.
  This bug (`issue 1168`_) is now fixed.

- The ``COVERAGE_DEBUG_FILE`` environment variable now accepts ``stdout`` and
  ``stderr`` to write to those destinations.

- TOML parsing now uses the `tomli`_ library.

- Some minor changes to usually invisible details of the HTML report:

  - Use a modern hash algorithm when fingerprinting, for high-security
    environments (`issue 1189`_).  When generating the HTML report, we save the
    hash of the data, to avoid regenerating an unchanged HTML page. We used to
    use MD5 to generate the hash, and now use SHA-3-256.  This was never a
    security concern, but security scanners would notice the MD5 algorithm and
    raise a false alarm.

  - Change how report file names are generated, to avoid leading underscores
    (`issue 1167`_), to avoid rare file name collisions (`issue 584`_), and to
    avoid file names becoming too long (`issue 580`_).

.. _Django coverage plugin: https://pypi.org/project/django-coverage-plugin/
.. _issue 580: nedbat/coveragepy#580
.. _issue 584: nedbat/coveragepy#584
.. _issue 1150: nedbat/coveragepy#1150
.. _issue 1167: nedbat/coveragepy#1167
.. _issue 1168: nedbat/coveragepy#1168
.. _issue 1189: nedbat/coveragepy#1189
.. _tomli: https://pypi.org/project/tomli/


.. _changes_56b1:

Version 5.6b1 — 2021-04-13
--------------------------

Note: 5.6 final was never released. These changes are part of 6.0.

- Third-party packages are now ignored in coverage reporting.  This solves a
  few problems:

  - Coverage will no longer report about other people's code (`issue 876`_).
    This is true even when using ``--source=.`` with a venv in the current
    directory.

  - Coverage will no longer generate "Already imported a file that will be
    measured" warnings about coverage itself (`issue 905`_).

- The HTML report uses j/k to move up and down among the highlighted chunks of
  code.  They used to highlight the current chunk, but 5.0 broke that behavior.
  Now the highlighting is working again.

- The JSON report now includes ``percent_covered_display``, a string with the
  total percentage, rounded to the same number of decimal places as the other
  reports' totals.

.. _issue 876: nedbat/coveragepy#876
.. _issue 905: nedbat/coveragepy#905
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working concurrency fixed
Projects
None yet
Development

Successfully merging a pull request may close this issue.