pytest and multiprocessing: CoverageException: Can't combine line data with arc dat

[nedbat]

Originally reported by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

I'm getting CoverageException: Can't combine line data with arc dat when using coverage with the pytest-cov pytest extension. I think this is similar to #399, but that issue is closed.

Example failure on travis: https://travis-ci.org/samuelcolvin/arq/builds/148541711

The test causing problems is using click's CliRunner, code here. The CLI command that's testing starts another process (using multiprocessing.Process) to run the worker which is what's causing the problem, if you prevent the worker process being started the exception doesn't occur.

I tried changing the concurrency mode but it made no difference.

To reproduce: clone, pip install -e . && pip install -r tests/requirements.txt, py.test --cov=arq.

relevant bits of pip freeze:

click==6.6
coverage==4.2
pytest==2.9.2
pytest-cov==2.3.0

Let me know if you need anymore information.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/512

[nedbat]

Original comment by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

Any update on this? I'm currently having to choose between coverage and running some tests.

[nedbat]

@samuelcolvin Sorry, I haven't started digging into this. It does sounds a lot like the fix I put in 4.2, so I will have to look at it soon. Have you tried it with the tip of coverage.py?

[nedbat]

Original comment by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

Same error with hg clone ... setup.py install I'm afraid:

...
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/pytest_cov/engine.py", line 150, in finish
INTERNALERROR>     self.cov.combine()
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/control.py", line 822, in combine
INTERNALERROR>     self.data_files.combine_parallel_data(self.data, aliases=aliases, data_paths=data_paths)
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/data.py", line 716, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "/home/samuel/code/arq/env/lib/python3.5/site-packages/coverage-4.2.1a0-py3.5-linux-x86_64.egg/coverage/data.py", line 481, in update
INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data

[nedbat]

I tried running your scenario, and did not get the error you describe. The full output is here: https://gist.github.com/nedbat/a482ea69fa509de5e0a7c3da8f378e4b

The test run ends with:

 tests/test_worker.py ✓✓✓✓✓✓✓✓✓✓✓                                                                                                                100% ██████████

---------- coverage: platform darwin, python 3.5.2-final-0 -----------
Name              Stmts   Miss Branch BrPart  Cover
---------------------------------------------------
arq/__init__.py       4      0      0      0   100%
arq/cli.py           15      0      0      0   100%
arq/logs.py          18      1      2      1    90%
arq/main.py          85      1     22      1    98%
arq/testing.py      164      6     34      0    97%
arq/utils.py         53      2     12      1    95%
arq/version.py        2      0      0      0   100%
arq/worker.py       245     31     54      4    88%
---------------------------------------------------
TOTAL               586     41    124      7    93%


Results (4.23s):
      29 passed
      11 failed
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /Users/ned/foo/arq/tests/test_main.py:137: assert None
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)
         - /usr/local/pythonz/pythons/CPython-3.5.2/lib/python3.5/asyncio/base_events.py:704: OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 6379), [Errno 61] Connect call failed ('::1', 6379), [Errno 61] Connect call failed ('fe80::1%lo0', 6379, 0, 1)

I'm not sure what caused my test failures, but I wouldn't think they are significant in this case.

[nedbat]

Original comment by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

looks like you don't have redis installed.

[nedbat]

@samuelcolvin I'm willing to try reproducing this, but I need explicit instructions, or I'll wander in the redis-wilderness for a while. What should I do on my Mac to get things to pass? Do you think the test failures are preventing the coverage failure?

[nedbat]

Original comment by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

Sorry about that, on ubuntu redis runs as a service so is always there. I've dusted off my mac and tried to run the code however brew seems to be broken so I can't install python3.5.

As far as I know it should be as simple:

• brew install redis
• redis-server, leave the terminal with redis running
• in another terminal in the arq directory git fetch && git checkout coverage-broken (I've added a test on that branch which breaks coverage), see here.
• run py.test all tests should pass
• run py.test --cov=arq you should get CoverageException as on travis.

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

I've taken a look at this as well, my opinion is that a combination of configuration, subprocess use and cwd changes cause the problem. That's why you get the data file saved without arcs (branch=False).

There are thee ways to deal with this:

• Specify --cov-config=setup.cfg (pytest-cov will absolutize it before tmpworkdir is used). Alternatively, you could use a .coveragerc (pytest-cov absolutize that if it exists).
• Fix the test to not use subprocesses. Afaik the test spawns a suprocess that sigterms the parent. This makes no sense to me - you can do that with no subprocess at all. Eg: signal.kill(os.getpid(), signal.SIGTERM) or even a thread.
• Stop changing current working directory (tmpworkdir fixture).

There is no bug in either coverage or pytest-cov here.

Also, note that concurrency = multiprocessing is completely un-necessary if you use pytest-cov (it completely manages coverage measurements in subprocesses for you).

[nedbat]

I've come to some different conclusions that Ionel did, though he was very helpful in getting me going with debugging with his aspectlib.

The problem here is the Process spawned by test_repeat_worker_close. It doesn't find the same configuration file that the rest of the tests do. Specifying --cov-config=setup.cfg doesn't help, because coverage.py only reads that file properly when it is read implicitly, not explicitly (that could be the subject of another bug report).

I think this is a combination of the direct use of Process, which perhaps coverage.py hasn't patched enough, and having settings in the implicitly-read setup.cfg file.

One simple solution is to move your coverage.py settings into a .coveragerc file. If you do that, things work.

[nedbat]

Original comment by Samuel Colvin (Bitbucket: samuelcolvin, GitHub: samuelcolvin)

---

Ned, thanks so much for the help. I've switched to using .coveragerc and that seems to have fixed it.

Perhaps a comment in the docs saying that using setup.cfg is not always the same as using .coveragerc would be useful?

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

pytest-cov could have an improvement to use setup.cfg if .coveragerc don't exist (same as coverage, but with pytest-cov's correct handling of relatiev paths). Open a bug report if you want it.

[nedbat]

@ionelmc it isn't enough to tell coverage.py to use setup.cfg. It won't be read properly unless it is discovered implicitly.

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

What do you mean, "discovered implicitly"?

[nedbat]

The sections are named differently ("[run]" vs "[coverage:run]"). If you specify setup.cfg as the file, coverage.py will try to read the "[run]" section and won't find it. It only looks for "[coverage:run]" when it is reading setup.cfg because it couldn't find .coveragerc.

Of course, this behavior could change, but that's the way the code is now.

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

I don't see how it's relevant. pytest-cov doesn't read the file, it will only pass the absolute path (if it exists).

[nedbat]

Original comment by Loic Dachary (Bitbucket: dachary, GitHub: dachary)

---

The following patch retries with the coverage: prefix if a non default file was specified and nothing was read from it. There could be a cleaner way to do the same and I'm willing to work on it if you think it is worth implementing. To be fully backward compatible it should only retry with the coverage: prefix in the case where a file was provided and nothing was read from it because it serves no useful purpose. Unless there is a valid use case for --rcfile=/dev/null ?

#!diff

diff -r [24aff3d7bfd5 (bb)](https://bitbucket.org/ned/coveragepy/commits/24aff3d7bfd5) coverage/config.py
--- a/coverage/config.py	Mon Dec 12 08:26:18 2016 +0100
+++ b/coverage/config.py	Thu Dec 15 10:35:14 2016 +0100
@@ -151,6 +151,7 @@
         # Metadata about the config.
         self.attempted_config_files = []
         self.config_files = []
+        self.any_set = False
 
         # Defaults for [run]
         self.branch = False
@@ -223,12 +224,11 @@
 
         self.config_files.extend(files_read)
 
-        any_set = False
         try:
             for option_spec in self.CONFIG_FILE_OPTIONS:
                 was_set = self._set_attr_from_config_option(cp, *option_spec)
                 if was_set:
-                    any_set = True
+                    self.any_set = True
         except ValueError as err:
             raise CoverageException("Couldn't read config file %s: %s" % (filename, err))
 
@@ -253,18 +253,18 @@
         if cp.has_section('paths'):
             for option in cp.options('paths'):
                 self.paths[option] = cp.getlist('paths', option)
-                any_set = True
+                self.any_set = True
 
         # plugins can have options
         for plugin in self.plugins:
             if cp.has_section(plugin):
                 self.plugin_options[plugin] = cp.get_section(plugin)
-                any_set = True
+                self.any_set = True
 
         # Was this file used as a config file? If no prefix, then it was used.
         # If a prefix, then it was only used if we found some settings in it.
         if section_prefix:
-            return any_set
+            return self.any_set
         else:
             return True
 
@@ -422,9 +422,12 @@
             config_read = config.from_file(fname, section_prefix=prefix)
             is_config_file = fname == config_file
 
-            if not config_read and is_config_file and specified_file:
-                raise CoverageException("Couldn't read '%s' as a config file" % fname)
-
+            if is_config_file and specified_file:
+                if not config_read:
+                    raise CoverageException("Couldn't read '%s' as a config file" % fname)
+                if not config.any_set:
+                    config_read = config.from_file(fname, section_prefix="coverage:")
+                    
             if config_read:
                 break
 

[nedbat]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque)

---

I think I may have hit this issue too.

Builds used to pass without any problems, but when moving the configuration from .coveragerc to setup.cfg (Peque/osbrain@f9eb8c6), it fails (https://travis-ci.org/Peque/osbrain/builds/237146025?utm_source=github_status&utm_medium=notification).

[nedbat]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

---

Maybe you need to tell pytest-cov what is the coverage config file (eg --cov-config=setup.cfg).

[nedbat]

Original comment by Miguel Sánchez de León Peque (Bitbucket: Peque, GitHub: Peque)

---

@ionelmc Thanks. That helped and it currently finishes the report. However, the source and omit parameters are ignored.

Have not figured out why. Here is the repo/branch: https://github.com/Peque/osbrain/tree/coverage

[joshfriend]

One simple solution is to move your coverage.py settings into a .coveragerc file. If you do that, things work.

My project uses a .coveragerc file and coverage reporting still fails with this error what seems like 1 in 3 runs.

[nedbat]

@joshfriend can you give us specific instructions to reproduce what you are seeing, even if it's cloning your full project?

[nedbat]

Feel free to re-open if you get more information.

[charmoniumQ]

My question is: does turning off branch = true fix the crash in your repo?

[joshfriend]

The issue is not able to be replicated reliably in my projects, so unfortunately I could not tell you

[wohali]

I get this error when I use branch = true. @wohali @joshfriend can you confirm this with your repo?

Yes, I use branch = true. As I understand it this changes the format of the file, as I mentioned above, so the results cannot be combined with runs that don't include that. Something to do with multiprocess runs and the toml file not being included caused the mixed format issue for me.

[dmartin]

Just ran into this bug and worked around it with --cov-config=.coveragerc. Symptomology is similar to @joshfriend above, but my project uses pyproject.toml and expected coverage.py to find its configuration there in multiprocess Processes.

We also use pyproject.toml for our coverage configuration, and wanted to mention that I both encountered and was (seemingly) able to work around this issue by explicitly specifying --cov-config=pyproject.toml as an argument when invoking pytest. Explicitly passing the path of the config file seemed to be sufficient without also extracting it out into .coveragerc format.

[joshfriend]

I have been using explicit --cov-config=path/to/.coveragerc the entire time as my projects use a git submodule to share the coverage/linter/etc configs across projects.

I have not changed anything about my coverage setup other than to update coverage.py as new versions come out periodically, and I rarely see this issue anymore. It used to happen quite frequently, but recently is so infrequent that I can't try anything to debug it, but it is also not annoying anymore so I stopped trying.

[wohali]

We also use pyproject.toml for our coverage configuration, and wanted to mention that I both encountered and was (seemingly) able to work around this issue by explicitly specifying --cov-config=pyproject.toml as an argument when invoking pytest. Explicitly passing the path of the config file seemed to be sufficient without also extracting it out into .coveragerc format.

Tried this and it didn't work -- is your project multiprocess as well?

[dmartin]

Tried this and it didn't work -- is your project multiprocess as well?

Hmm, our project itself doesn't use multiprocessing so I suppose the scenario is slightly different, but we do use pytest-xdist to distribute test cases across cores, which seems to be the cause of the error in our case.

[CharlesGaydon]

Original comment by Ionel Cristian Mărieș (Bitbucket: ionelmc, GitHub: ionelmc)

I've taken a look at this as well, my opinion is that a combination of configuration, subprocess use and cwd changes cause the problem. That's why you get the data file saved without arcs (branch=False).

There are thee ways to deal with this:

* Specify `--cov-config=setup.cfg` (pytest-cov will absolutize it before `tmpworkdir` is used). Alternatively, you could use a `.coveragerc` (pytest-cov absolutize that if it exists).

* Fix the test to not use subprocesses. Afaik the test spawns a suprocess that sigterms the parent. This makes no sense to me - you can do that with no subprocess at all. Eg: `signal.kill(os.getpid(), signal.SIGTERM)` or even a thread.

* Stop changing current working directory (`tmpworkdir` fixture).

There is no bug in either coverage or pytest-cov here.

Also, note that concurrency = multiprocessing is completely un-necessary if you use pytest-cov (it completely manages coverage measurements in subprocesses for you).

Specifying --cov-config=setup.cfg as an addopted parameter in my setup.cfg solved the issue for me. Thanks!

Here are a few keywords on my setting so others might find this solution:
I use hydra config manager (which changes the current directory), and have a subprocess called thanks to sh.python(command). In addition, I use the tmp_dir fixture.

[christoph-blessing]

I think I am running into a similar issue in my project.

Traceback

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pytest_cov/plugin.py", line 271, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pytest_cov/engine.py", line 44, in ensure_topdir_wrapper
INTERNALERROR>     return meth(self, *args, **kwargs)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/pytest_cov/engine.py", line 230, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/coverage/control.py", line 748, in combine
INTERNALERROR>     combine_parallel_data(
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/coverage/data.py", line 132, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "/src/.venv/lib/python3.8/site-packages/coverage/sqldata.py", line 606, in update
INTERNALERROR>     raise DataError("Can't combine arc data with line data")
INTERNALERROR> coverage.exceptions.DataError: Can't combine arc data with line data

If you want you can reproduce the error using my project (Docker required):

git clone https://github.com/cblessing24/compenv
cd compenv
git checkout b0aafca39898bdd4487ae57869adc43c49b093fe
docker build -t cblessing24/compenv .
touch .env
docker compose run pytest_github

Adding "--cov-config=pyproject.toml" to the command field of the pytest_github service in the docker-compose.yml file fixes the issue. Interestingly adding the same argument to addopts in pyproject.toml does not fix it.

[nedbat]

Thanks all. I'm going to close this issue because: 1) it seems to be about pytest-cov, not coverage.py, and 2) there's a fix, which is to explicitly name your coverage configuration file with --cov-config.

[kdeldycke]

Do we still need the workaround of passing explicitly --cov-config=pyproject.toml now that pytest is defining the root dir relative to pyproject.toml as of 8.1.0? I am referring to: pytest-dev/pytest#11962