4.3.0: pytest is failing in tests/test_all.py::DNSTest::test_custom_resolvconf unit

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Module si build with PYCARES_USE_SYSTEM_LIB=1.
Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pycares-4.3.0-3.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pycares-4.3.0-3.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.8.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pycares-pycares-4.3.0
collected 54 items

tests/test_all.py ......F...........xx.........s.................x..xs.. [100%]

=================================== FAILURES ===================================
________________________ DNSTest.test_custom_resolvconf ________________________

self = <tests.test_all.DNSTest testMethod=test_custom_resolvconf>

    @unittest.skipIf(sys.platform == 'win32', 'skipped on Windows')
    def test_custom_resolvconf(self):
        self.result, self.errorno = None, None
        def cb(result, errorno):
            self.result, self.errorno = result, errorno
        self.channel = pycares.Channel(tries=1, timeout=2.0, resolvconf_path=os.path.join(FIXTURES_PATH, 'badresolv.conf'))
        self.channel.query('google.com', pycares.QUERY_TYPE_A, cb)
        self.wait()
        self.assertEqual(self.result, None)
>       self.assertEqual(self.errorno, pycares.errno.ARES_ETIMEOUT)
E       AssertionError: 11 != 12

tests/test_all.py:606: AssertionError
=========================== short test summary info ============================
SKIPPED [1] tests/test_all.py:430: ANY type does not work on Mac.
SKIPPED [1] tests/test_all.py:537: The site used for this test no longer returns a non-ascii SOA.
XFAIL tests/test_all.py::DNSTest::test_getnameinfo_ipv6 - reason:
XFAIL tests/test_all.py::DNSTest::test_getnameinfo_ipv6_ll - reason:
XFAIL tests/test_all.py::DNSTest::test_query_txt_bytes2 - reason:
XFAIL tests/test_all.py::DNSTest::test_query_txt_multiple_chunked_with_non_ascii_content - reason:
FAILED tests/test_all.py::DNSTest::test_custom_resolvconf - AssertionError: 1...
============= 1 failed, 47 passed, 2 skipped, 4 xfailed in 11.29s ==============

Here is list of installed modules in build env

Package                       Version
----------------------------- -----------------
alabaster                     0.7.13
attrs                         22.2.0
Babel                         2.12.1
build                         0.10.0
cffi                          1.15.1
charset-normalizer            3.1.0
distro                        1.8.0
docutils                      0.19
exceptiongroup                1.0.0
gpg                           1.18.0-unknown
idna                          3.4
imagesize                     1.4.1
importlib-metadata            6.1.0
iniconfig                     2.0.0
Jinja2                        3.1.2
libcomps                      0.1.19
MarkupSafe                    2.1.2
packaging                     23.0
pip                           23.0.1
pluggy                        1.0.0
ply                           3.11
pycparser                     2.21
Pygments                      2.14.0
pyproject_hooks               1.0.0
pytest                        7.2.2
python-dateutil               2.8.2
pytz                          2023.2
requests                      2.28.2
rpm                           4.17.0
setuptools                    65.6.3
six                           1.16.0
snowballstemmer               2.2.0
Sphinx                        6.1.3
sphinx-rtd-theme              1.2.0
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2.dev20230202
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib.jquery          3.0.0
sphinxcontrib-jsmath          1.0.1.dev20230128
sphinxcontrib-qthelp          1.0.3.dev20230128
sphinxcontrib-serializinghtml 1.1.5
tomli                         2.0.1
urllib3                       1.26.15
wheel                         0.38.4
zipp                          3.15.0

[saghul]

• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Then a number of the tests will simply fail. The test suite creates real DNS traffic.

[kloczek]

• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Then a number of the tests will simply fail. The test suite creates real DNS traffic.

So it would be good to @pytest.mark.network to those units which requires such access.
https://docs.pytest.org/en/7.1.x/example/markers.html
network mark is already quite widely used in many other python modules test suites.

[saghul]

The test suite does not use pytest, it currently uses the vanilla unittest module.

I'd take a PR improving things here, but as it stands if you are interested in running the tests you need Internet access.

[kloczek]

Correctly written unittest test suite can be handled by pytest.
Any units which needs public network should have @pytest.mark.network https://docs.pytest.org/en/7.1.x/example/markers.html as that mark is currently widely used by many other python modules test suites.

[saghul]

PRs are welcome. I currently don't have the time or energy to see if pytest would just work and add decorate functions.

If you'd like to send a PR I'd be happy to review it.