Modules imported with importlib don't record coverage

[timofurrer]

Describe the bug

When a module is dynamically imported with importlib it won't record any coverage data.

To Reproduce

• Python: 3.7
• Coverage: 5.1
• No 3rd-party packages involved

I've created a minimal example illustrating the problem here: https://github.com/timofurrer/coverage-importlib-test
The repository consists of a single Python package with a single module src/app/core.py which uses
the importlib code from here to dynamically import src/app/plugins/plugin_mod.py.

When running the tests with tox the output shows the following:

src/app/__init__.py                 0      0      0      0   100%
src/app/core.py                    10      0      0      0   100%
src/app/plugins/__init__.py         0      0      0      0   100%
src/app/plugins/plugin_mod.py       1      1      0      0     0%   1
---------------------------------------------------------------------------
TOTAL                              11      1      0      0    91%

So no coverage for plugin_mod.py ...

Expected behavior

... However, I'd expect it to have 100% coverage as the others do.
Importing the module statically using import app.plugins.plugin_mod.py in src/app/core.py instead of with importlib works like a charm.

Additional information

The following .coveragerc configuration is used:

[run]
branch = True
source =
    app

[paths]
source =
    src
    .tox/*/site-packages

[report]
show_missing = True

Edit:

The problem seems to be related to the [run] source config - removing it, correctly reports coverage for the plugin module,
but it also includes modules not from the app packages into the coverage report - which is undesired.
Adding the app.plugins package to [run] source doesn't help.

The config also seems like a common one - e.g. attrs uses it, too

[nedbat]

I'm trying the repo you've made (thanks!) and I can see that the plugin is reported as 0%. But I cannot reproduce what you say:

The problem seems to be related to the [run] source config - removing it, correctly reports coverage for the plugin module,
but it also includes modules not from the app packages into the coverage report

When I comment out the [run] source setting in the .coveragerc file, I get:

py37 run-test: commands[2] | coverage report
No source for code: '/private/tmp/bug1002/coverage-importlib-test/src/_pytest/__init__.py'.

I'm using --debug=trace to see what files are being imported and whether they are traced, but plugin_mod.py isn't mentioned, I'm not sure why yet.

[nedbat]

The source=app setting is internally converted into a ModuleMatcher. That is, because "app" is an importable module, it's taken to mean, "the source is everything in the app package tree". But when plugin_mod is imported, its module name is "plugin_mod", so it isn't included in the trace.

I thought there was an issue already asking for a way to indicate that source=app is definitely the directory "app", and not the module "app", but I don't see it now.

[nedbat]

Here is the issue about the directory/package confusion: #268. Though here, it's a package because you don't have an "app" directory (my mistake).

[timofurrer]

But I cannot reproduce what you say:

I wasn't clear enough, sorry. You'd also have to comment the [paths] config like, so:

[run]
branch = True
#source =
#    app

#[paths]
#source =
#    src
#    .tox/*/site-packages

[report]
show_missing = True

But when plugin_mod is imported, its module name is "plugin_mod", so it isn't included in the trace.

With "module name" are you referring to the key used for sys.modules or __name__ or something else? Do you know if I can workaround the current behavior by setting the module name correctly for coverage ?

[timofurrer]

I'm using --debug=trace to see what files are being imported and whether they are traced, but plugin_mod.py isn't mentioned, I'm not sure why yet.

What is actually printed here? I don't see the covered modules in the output either - but might be because they are traced correctly ?

[nedbat]

Adding --debug=trace shows more information, but you also need to add -s to pytest, or it captures all the output (this confused me, and is why I say plugin_mod.py wasn't mentioned in the output).

The name plugin_mod is from __name__ in the module's globals.

[nedbat]

BTW, I just pushed some changes on the master branch to add more information in the --debug=trace output that make this easier to debug.

[nedbat]

Is this still an issue for you?

[timofurrer]

Running on Python 3.10 with coverage 6.1.2 it still is an issue. The initially references repo still works.

If I find some time I'll check out the new --debug=trace option and my I can spot something. If you have any hint, please keep me posted.

[nedbat]

The --debug=trace output includes (among many other lines):

Source matching against modules <ModuleMatcher source_pkgs ['app']>
Not tracing '/System/Volumes/Data/root/src/bugs/bug1002/coverage-importlib-test/.tox/py37/lib/python3.7/site-packages/app/plugins/plugin_mod.py': module 'plugin_mod' falls outside the --source spec

Coverage.py implements your source=app setting as a check that modules are inside the "app" package. Then when plugin_mod is imported, its full name is "plugin_mod", not "app.plugins.plugin_mod". That's why the module isn't measured.

If you change this line in core.py, it works:

diff --git a/src/app/core.py b/src/app/core.py
index e0c9b4c..952db9c 100644
--- a/src/app/core.py
+++ b/src/app/core.py
@@ -2,7 +2,7 @@
 import importlib.util
 from pathlib import Path

-module_name = "plugin_mod"
+module_name = "app.plugins.plugin_mod"
 file_path = Path(__file__).parent / "plugins" / "plugin_mod.py"

 spec = importlib.util.spec_from_file_location(module_name, file_path)

Is that a reasonable change to your code? Or, if there's another package that plugins live in, it could be included in the source= setting.

[MLNW]

@nedbat's comment helped me solve this exact issue on my very own codebase! The name argument in the importlib call matters!

[nedbat]

@MLNW thanks for the confirmation, that is good to hear. Others: feel free to re-open with more details if this is still a problem for you.