ImportWarning: can't resolve package from __spec__ or __package__

[blueyed]

Given the following directory structuce:

.
└── tests
    ├── conftest.py
    ├── __init__.py
    ├── relative.py
    └── test_foo.py

With tests/conftest.py:

from .relative import relative

print("conftest")

tests/relative.py:

relative = 1

tests/test_foo.py:

def test_foo():
    pass

Running PYTHONWARNINGS=all pytest tests -s shows:

…/pyenv/tmp-system-ImportWarning-ZniFiE/lib/python3.6/site.py:165: DeprecationWarning: 'U' mode is deprecated
  f = open(fullname, "rU")
…/Vcs/pytest/_pytest/assertion/rewrite.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
…/Vcs/pytest/ImportWarning/tests/conftest.py:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from .relative import relative
conftest
==================================== test session starts ====================================
platform linux -- Python 3.6.3, pytest-3.3.2.dev14+gd3b54d03.d20171225, py-1.5.2, pluggy-0.6.0
rootdir: …/Vcs/pytest, inifile: tox.ini
collected 1 item                                                                            

tests/test_foo.py .                                                                   [100%]

================================= 1 passed in 0.01 seconds ==================================

The ImportWarning is:

tests/conftest.py:1: ImportWarning: can't resolve package from spec or package, falling back on name and path
from .relative import relative

It only shows up when using -s.

This came up for pytest-django.
See pytest-dev/pytest-django#546 (comment).

Defining __package__ explicitly fixes it, but it looks like this should not be necessary.

[pytestbot]

GitMate.io thinks the contributor most likely able to help you is @nicoddemus.

[asottile]

Now that pytest-xdist uses explicit relative imports (after pytest-dev/pytest-xdist#242) this is also breaking the py36-xdist suite for pytest's tests.

[nicoddemus]

Opened #3096 to at least ignore the warning for now so we can get clean builds again.

Anybody knows what's the proper fix here? Who's doing something wrong, pytest or pytest-xdist?

[asottile]

without any evidence my guess would be pytest (and if I were to extend that guess further, it would go away with assert rewriting off).

Let's validate those guesses:

$ PYTHONWARNINGS=all py.test tests -s --assert=plain >& assert_plain.txt
$ PYTHONWARNINGS=all py.test tests -s >& assert_rewrite.txt
$ diff -u assert_*.txt
--- assert_plain.txt	2018-01-08 14:42:20.156677858 -0800
+++ assert_rewrite.txt	2018-01-08 14:42:28.528677858 -0800
@@ -24,3 +24,5 @@
   params = attr.ib(convert=attr.converters.optional(tuple))
 /tmp/foo/venv/lib/python3.6/site-packages/_pytest/fixtures.py:846: DeprecationWarning: The `convert` argument is deprecated in favor of `converter`.  It will be removed after 2019/01.
   ids = attr.ib(default=None, convert=_ensure_immutable_ids)
+/tmp/foo/tests/conftest.py:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
+  from .relative import relative

🤔 hmmm... indeed.

Then it would probably be the way pytest is building module objects for __init__ files (with incorrect __package__)?

[asottile]

probably worth reading as well: https://www.python.org/dev/peps/pep-0451/

[BoboTiG]

I have similar warnings, let me know if it needs a separate issue or a pytest-cov issue.

platform win32 -- Python 3.6.5, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- C:\python-3.6.5\python.exe
plugins: cov-2.5.1

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:8: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import embed

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:9: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import engine

C:\python-3.6.5\lib\site-packages\pytest_cov\engine.py:11: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from .compat import StringIO

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:10: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import compat

[RonnyPfannschmidt]

its the same problem -

[blueyed]

Ping @flub - any idea?

[asottile]

This fixes the warning for me in python3.6:

diff --git a/src/_pytest/assertion/rewrite.py b/src/_pytest/assertion/rewrite.py
index eceed611..bc1f058e 100644
--- a/src/_pytest/assertion/rewrite.py
+++ b/src/_pytest/assertion/rewrite.py
@@ -4,6 +4,7 @@ import ast
 import errno
 import itertools
 import imp
+import importlib
 import marshal
 import os
 import re
@@ -207,7 +208,13 @@ class AssertionRewritingHook(object):
         # I wish I could just call imp.load_compiled here, but __file__ has to
         # be set properly. In Python 3.2+, this all would be handled correctly
         # by load_compiled.
-        mod = sys.modules[name] = imp.new_module(name)
+        if sys.version_info >= (3, 5):
+            spec = importlib.util.spec_from_file_location(
+                name, co.co_filename, loader=self,
+            )
+            mod = sys.modules[name] = importlib.util.module_from_spec(spec)
+        else:
+            mod = sys.modules[name] = imp.new_module(name)
         try:
             mod.__file__ = co.co_filename
             # Normally, this attribute is 3.2+.

The (unfortunate) bit is importlib.util.module_from_spec is py35+

I was able to reproduce by installing pytest-cov and using python -Wonce -mpytest test.py (with a trivial test file).

Another way was editing pytest.py in site-packages to have this code at the top:

import warnings
warnings.filter('error', ImportWarning)

which produces this trace:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/t/venv/lib/python3.6/site-packages/pytest.py", line 69, in <module>
    raise SystemExit(pytest.main())
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 56, in main
    config = _prepareconfig(args, plugins)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 181, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 196, in _multicall
    gen.send(outcome)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 607, in pytest_cmdline_parse
    self.parse(args)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 772, in parse
    self._preparse(args, addopts=addopts)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 724, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 397, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/tmp/t/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2318, in load
    return self.resolve()
  File "/tmp/t/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 217, in load_module
    py.builtin.exec_(co, mod.__dict__)
  File "/tmp/t/venv/lib/python3.6/site-packages/pytest_cov/plugin.py", line 8, in <module>
    from . import embed
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__

[asottile]

if I were to turn that into a patch I'd probably (at the module scope) do something like:

if sys,version_info >= (3, 5):
    import importlib.util
    def new_module(name, path, loader):
        ...
else:
    def new_module(name, path, loader):
        ...

since I should really be importing importlib.util and not depending on import side-effects by just importing importlib (and importlib.util doesn't exist in all the versions of python pytest supports, notably 2.7)

lmk and I can craft this into a patch -- no idea how to regression test it though 🙃

[asottile]

alternatively, I wonder if assigning __spec__ before calling exec_ would also resolve this warning 🤔

[asottile]

indeed, this also silences the warning:

@@ -213,6 +214,10 @@ class AssertionRewritingHook(object):
             # Normally, this attribute is 3.2+.
             mod.__cached__ = pyc
             mod.__loader__ = self
+            if sys.version_info >= (3, 5):
+                mod.__spec__ = importlib.util.spec_from_file_location(
+                    name, co.co_filename, loader=self,
+                )
             py.builtin.exec_(co, mod.__dict__)
         except:  # noqa
             if name in sys.modules:

[blueyed]

Awesome, thanks!

The following at the top of the module makes sense then IMHO:

if sys.version_info >= (3, 5):
    from importlib.util import spec_from_file_location