Fix unit test on Python 3.8 #56031

bdrung · 2020-01-30T17:42:45Z

Salt fails on Python 3.8:

======================================================================
ERROR: unit.grains.test_core (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: unit.grains.test_core
Traceback (most recent call last):
  File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "tests/unit/grains/test_core.py", line 37, in <module>
    import salt.grains.core as core
  File "salt/grains/core.py", line 40, in <module>
    from platform import _supported_dists
ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py)

So only try to import _supported_dists from platform for Python <=
3.7. Otherwise rely on the external distro module to not need any
special handling.

Addresses parts of #55835

bdrung · 2020-02-03T11:44:57Z

I pushed more commits to fix all Python 3.8 related unit tests failures.

s0undt3ch · 2020-02-07T19:00:48Z

FYI, I also did some work on this here.
I'm not devoting much time to it right now, so feel free to rebase your work on top of that branch, it would be much appreciated.

bdrung · 2020-02-10T12:23:46Z

I picked the commits from your branch that I haven't addressed yet.

Apply saltstack/salt#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

Apply saltstack#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

Apply saltstack/salt#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

cmcmarrow

@bdrung thanks for doing this! You’re really close. Just need to get the Windows and Mac test working. Once you have that I think this PR will be good to merge.

bdrung · 2020-03-26T16:36:22Z

The failing Windows and Mac tests are identical with the failings tests of the master branch. So this merge request does not introduce any regression.

bdrung · 2020-04-24T09:11:31Z

I rebased this patch set on master to resolve the merge commits. This rebase dropped some commits which are already part of the master branch.

s0undt3ch · 2020-04-24T09:25:50Z

It has some lint issues that need fixing now

DeprecationWarning: Using or importing the ABCs from `collections` instead of from `collections.abc` is deprecated since Python 3.3, and in 3.9 it will stop working. Therefore try to import the abstract base classes from `collections.abc` before falling back to `collections`. Signed-off-by: Benjamin Drung <[email protected]>

Salt fails on Python 3.8: ``` ====================================================================== ERROR: unit.grains.test_core (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: unit.grains.test_core Traceback (most recent call last): File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name __import__(name) File "tests/unit/grains/test_core.py", line 37, in <module> import salt.grains.core as core File "salt/grains/core.py", line 40, in <module> from platform import _supported_dists ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py) ``` So only try to import `_supported_dists` from `platform` for Python <= 3.7. Otherwise rely on the external `distro` module to not need any special handling. Addresses parts of saltstack#55835 Signed-off-by: Benjamin Drung <[email protected]>

The following unit tests fail on Python 3.8: ``` ====================================================================== ERROR: test_state_config (unit.renderers.test_stateconf.StateConfigRendererTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/renderers/test_stateconf.py", line 74, in test_state_config result = self._render_sls(''' File "/<<PKGBUILDDIR>>/tests/unit/renderers/test_stateconf.py", line 66, in _render_sls return self._renderers['stateconf']( File "/<<PKGBUILDDIR>>/salt/renderers/stateconf.py", line 227, in render for k in six.iterkeys(tmplctx): # iterate over a copy of keys RuntimeError: dictionary keys changed during iteration ====================================================================== ERROR: test_apply_cloud_providers_config_extend (unit.test_config.ConfigTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/test_config.py", line 1243, in test_apply_cloud_providers_config_extend salt.config.apply_cloud_providers_config( File "/<<PKGBUILDDIR>>/salt/config/__init__.py", line 3196, in apply_cloud_providers_config for driver, details in six.iteritems(entries): RuntimeError: dictionary keys changed during iteration ====================================================================== ERROR: test_apply_cloud_providers_config_extend_multiple (unit.test_config.ConfigTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/test_config.py", line 1334, in test_apply_cloud_providers_config_extend_multiple self.assertEqual(ret, salt.config.apply_cloud_providers_config(overrides, defaults=DEFAULT)) File "/<<PKGBUILDDIR>>/salt/config/__init__.py", line 3196, in apply_cloud_providers_config for driver, details in six.iteritems(entries): RuntimeError: dictionary keys changed during iteration ====================================================================== ``` Replace the affected for loop of the first case by a dictionary comprehension to construct the modified dictionary. For the remaining cases, switch from `iteritems` to `iterkeys`, since the dictionary values will be modified. Signed-off-by: Benjamin Drung <[email protected]>

The test case `unit.modules.test_virt.VirtTestCase.test_update` fails on Ubuntu 20.04 with Python 3.8: ``` ====================================================================== FAIL: test_update (unit.modules.test_virt.VirtTestCase) [CPU:0.0%|MEM:17.0%] ---------------------------------------------------------------------- Traceback (most recent call last): File "tests/unit/modules/test_virt.py", line 1313, in test_update self.assertEqual({ AssertionError: {'definition': False, 'disk': {'attached': [], 'detached[49 chars] []}} != {'definition': True, 'disk': {'attached': [], 'detached'[74 chars]True} + {'cpu': True, - {'definition': False, ? ^ ^^^^ + 'definition': True, ? ^ ^^^ 'disk': {'attached': [], 'detached': []}, - 'interface': {'attached': [], 'detached': []}} ? ^ + 'interface': {'attached': [], 'detached': []}, ? ^ + 'mem': True} ====================================================================== ``` `virt.update` falsely detects a change to the graphics setting in the "no diff case". The old XML part specifies: ```xml <graphics type="spice" port="5900" autoport="yes" listen="127.0.0.1"> <listen type="address" address="127.0.0.1" /> </graphics> ``` The newly generated XML misses the port setting: ```xml <graphics type="spice" listen="127.0.0.1" autoport="yes"> <listen type="address" address="127.0.0.1" /> </graphics> ``` So adjust the initial XML file of the template, because setting the port to the default 5900, will change `autoport` to `no`. Signed-off-by: Benjamin Drung <[email protected]>

bdrung · 2020-04-24T09:28:43Z

Fixed the lint issues.

This was missed in saltstack#56031.

This was missed in #56031.

Apply saltstack/salt#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

waynew · 2020-05-06T17:59:15Z

applies to #55835

OrangeDog · 2020-05-10T11:21:51Z

Why did this bother with the try/except on collections.abc, when the old location has been deprecated since 3.3, and the minimum supported version for Sodium is 3.5?

bdrung · 2020-05-19T15:33:12Z

Good point. Please open a merge request to drop falling back to the old location.

Apply saltstack/salt#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

Apply saltstack#56031 to support Python 3.8, which removed a deprecated module and changed some behaviour. Add a {Build,}Requires on python-distro, since it is now required.

bdrung requested a review from a team as a code owner January 30, 2020 17:42

ghost requested a review from cmcmarrow January 30, 2020 17:42

bdrung force-pushed the master-python-3.8 branch 2 times, most recently from fb6cfa7 to cc541a2 Compare February 3, 2020 11:44

bdrung changed the title ~~Support distro.linux_distribution~~ Fix unit test on Python 3.8 Feb 3, 2020

bdrung mentioned this pull request Feb 3, 2020

Epic: Support for Python 3.8 #55835

Closed

Akm0d added this to the Approved milestone Feb 7, 2020

Akm0d self-assigned this Feb 7, 2020

Akm0d added the Core relates to code central or existential to Salt label Feb 7, 2020

s-t-e-v-e-n-k mentioned this pull request Feb 17, 2020

Apply patch from upstream to support Python 3.8 openSUSE/salt#211

Merged

bdrung force-pushed the master-python-3.8 branch 3 times, most recently from 3833c85 to bc40225 Compare February 21, 2020 14:36

sagetherage added the ZRelease-Sodium retired label label Mar 24, 2020

cmcmarrow suggested changes Mar 25, 2020

View reviewed changes

bdrung force-pushed the master-python-3.8 branch from 54487c4 to 35d5b2a Compare March 26, 2020 16:35

bdrung dismissed waynew’s stale review via 3f34672 April 24, 2020 09:10

bdrung force-pushed the master-python-3.8 branch from b9e1609 to 3f34672 Compare April 24, 2020 09:10

bdrung and others added 6 commits April 24, 2020 11:27

Update PyTestSalt requirement(because we now bundle tornado)

01fb8fc

Run the full test suite on Arch under Py3

26d3a77

bdrung force-pushed the master-python-3.8 branch from 3f34672 to 26d3a77 Compare April 24, 2020 09:28

dwoz approved these changes Apr 24, 2020

View reviewed changes

dwoz merged commit 0f58560 into saltstack:master Apr 24, 2020

bdrung deleted the master-python-3.8 branch April 24, 2020 22:13

terminalmage mentioned this pull request Apr 25, 2020

Remove import of platform._supported_dists #56896

Merged

terminalmage added a commit to terminalmage/salt that referenced this pull request Apr 25, 2020

Remove import of platform._supported_dists

cf82b01

This was missed in saltstack#56031.

dwoz pushed a commit that referenced this pull request Apr 26, 2020

Remove import of platform._supported_dists

4ef2e20

This was missed in #56031.

OrangeDog mentioned this pull request May 11, 2020

[BUG] Ubuntu 20.04 py3 minion fails to execute jobs due to missing grain os_family #57194

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix unit test on Python 3.8 #56031

Fix unit test on Python 3.8 #56031

bdrung commented Jan 30, 2020

bdrung commented Feb 3, 2020

s0undt3ch commented Feb 7, 2020

bdrung commented Feb 10, 2020

cmcmarrow left a comment

bdrung commented Mar 26, 2020

bdrung commented Apr 24, 2020

s0undt3ch commented Apr 24, 2020

bdrung commented Apr 24, 2020

waynew commented May 6, 2020

OrangeDog commented May 10, 2020

bdrung commented May 19, 2020

Fix unit test on Python 3.8 #56031

Fix unit test on Python 3.8 #56031

Conversation

bdrung commented Jan 30, 2020

bdrung commented Feb 3, 2020

s0undt3ch commented Feb 7, 2020

bdrung commented Feb 10, 2020

cmcmarrow left a comment

Choose a reason for hiding this comment

bdrung commented Mar 26, 2020

bdrung commented Apr 24, 2020

s0undt3ch commented Apr 24, 2020

bdrung commented Apr 24, 2020

waynew commented May 6, 2020

OrangeDog commented May 10, 2020

bdrung commented May 19, 2020