Python3.8 compatibility PR's (saltstack#235)

* Update static requirements to include Py3.8 and Py3.9 (except windows) Windows required package pywin32 doesn't state that it support any python version above Py3.7 * Allow running the test suite against Py3.8 and Py3.9 * Fix deprecation warnings for imports from collections DeprecationWarning: Using or importing the ABCs from `collections` instead of from `collections.abc` is deprecated since Python 3.3, and in 3.9 it will stop working. Therefore try to import the abstract base classes from `collections.abc` before falling back to `collections`. Signed-off-by: Benjamin Drung <[email protected]> * Support distro.linux_distribution Salt fails on Python 3.8: ``` ====================================================================== ERROR: unit.grains.test_core (unittest.loader._FailedTest) ---------------------------------------------------------------------- ImportError: Failed to import test module: unit.grains.test_core Traceback (most recent call last): File "/usr/lib/python3.8/unittest/loader.py", line 436, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name __import__(name) File "tests/unit/grains/test_core.py", line 37, in <module> import salt.grains.core as core File "salt/grains/core.py", line 40, in <module> from platform import _supported_dists ImportError: cannot import name '_supported_dists' from 'platform' (/usr/lib/python3.8/platform.py) ``` So only try to import `_supported_dists` from `platform` for Python <= 3.7. Otherwise rely on the external `distro` module to not need any special handling. Addresses parts of saltstack#55835 Signed-off-by: Benjamin Drung <[email protected]> * Fix RuntimeError: dictionary keys changed during iteration The following unit tests fail on Python 3.8: ``` ====================================================================== ERROR: test_state_config (unit.renderers.test_stateconf.StateConfigRendererTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/renderers/test_stateconf.py", line 74, in test_state_config result = self._render_sls(''' File "/<<PKGBUILDDIR>>/tests/unit/renderers/test_stateconf.py", line 66, in _render_sls return self._renderers['stateconf']( File "/<<PKGBUILDDIR>>/salt/renderers/stateconf.py", line 227, in render for k in six.iterkeys(tmplctx): # iterate over a copy of keys RuntimeError: dictionary keys changed during iteration ====================================================================== ERROR: test_apply_cloud_providers_config_extend (unit.test_config.ConfigTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/test_config.py", line 1243, in test_apply_cloud_providers_config_extend salt.config.apply_cloud_providers_config( File "/<<PKGBUILDDIR>>/salt/config/__init__.py", line 3196, in apply_cloud_providers_config for driver, details in six.iteritems(entries): RuntimeError: dictionary keys changed during iteration ====================================================================== ERROR: test_apply_cloud_providers_config_extend_multiple (unit.test_config.ConfigTestCase) [CPU:0.0%|MEM:56.6%] ---------------------------------------------------------------------- Traceback (most recent call last): File "/<<PKGBUILDDIR>>/tests/unit/test_config.py", line 1334, in test_apply_cloud_providers_config_extend_multiple self.assertEqual(ret, salt.config.apply_cloud_providers_config(overrides, defaults=DEFAULT)) File "/<<PKGBUILDDIR>>/salt/config/__init__.py", line 3196, in apply_cloud_providers_config for driver, details in six.iteritems(entries): RuntimeError: dictionary keys changed during iteration ====================================================================== ``` Replace the affected for loop of the first case by a dictionary comprehension to construct the modified dictionary. For the remaining cases, switch from `iteritems` to `iterkeys`, since the dictionary values will be modified. Signed-off-by: Benjamin Drung <[email protected]> * Update PyTestSalt requirement(because we now bundle tornado) * Run the full test suite on Arch under Py3 * Fix deprecation warnings for imports from collections DeprecationWarning: Using or importing the ABCs from `collections` instead of from `collections.abc` is deprecated since Python 3.3, and in 3.9 it will stop working. Therefore try to import the abstract base classes from `collections.abc` before falling back to `collections`. Signed-off-by: Benjamin Drung <[email protected]> * Replace deprecated inspect.formatargspec Python 3.7 raises a deprecation warning: salt/utils/decorators/signature.py:31: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly `inspect.formatargspec` is only used in `salt.utils.decorators.signature.identical_signature_wrapper` which is only used in `salt.utils.decorators.path` for decorating the `which` and `which_bin` functions. The function `identical_signature_wrapper` can be simply replaced by Python's `functools.wraps` which is available since at least Python 2.7. When inspecting those wrapped functions, the underlying function (stored in the `__wrapped__` attribute) needs to be inspect instead. fixes saltstack#50911 Signed-off-by: Benjamin Drung <[email protected]> Co-authored-by: Pedro Algarvio <[email protected]> Co-authored-by: Benjamin Drung <[email protected]>
meaksh · May 12, 2020 · 1ce138a · 1ce138a
1 parent 9c2a454
commit 1ce138a
Show file tree

Hide file tree

Showing 38 changed files with 1,193 additions and 83 deletions.
diff --git a/.ci/kitchen-archlts-py3 b/.ci/kitchen-archlts-py3
@@ -8,7 +8,7 @@ runTestSuite(
     golden_images_branch: 'master',
     jenkins_slave_label: 'kitchen-slave',
     nox_env_name: 'runtests-zeromq',
-    nox_passthrough_opts: '-n integration.modules.test_pkg',
+    nox_passthrough_opts: '',
     python_version: 'py3',
     testrun_timeout: 6,
     use_spot_instances: true)

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -412,6 +412,234 @@ repos:
           - --py-version=3.7
           - --platform=linux
 
+      - id: pip-tools-compile
+        alias: compile-linux-py3.8-zmq-requirements
+        name: Linux Py3.8 ZeroMQ Requirements
+        files: ^requirements/((base|zeromq|pytest)\.txt|static/linux\.in)$
+        exclude: ^requirements/static/(centos-6|amzn-2018\.03|lint|cloud|docs|darwin|windows)\.in$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=linux
+          - --include=requirements/base.txt
+          - --include=requirements/zeromq.txt
+          - --include=requirements/pytest.txt
+
+      - id: pip-tools-compile
+        alias: compile-darwin-py3.8-zmq-requirements
+        name: Darwin Py3.8 ZeroMQ Requirements
+        files: ^(pkg/osx/(req|req_ext)\.txt|requirements/((base|zeromq|pytest)\.txt|static/darwin\.in))$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=darwin
+          - --include=pkg/osx/req.txt
+          - --include=pkg/osx/req_ext.txt
+          - --include=requirements/base.txt
+          - --include=requirements/zeromq.txt
+          - --include=requirements/pytest.txt
+          - --passthrough-line-from-input=^pyobjc(.*)$
+
+# Commented out since pywin32 and pymssql do not have packages or support for Py >= 3.8
+#      - id: pip-tools-compile
+#        alias: compile-windows-py3.8-zmq-requirements
+#        name: Windows Py3.8 ZeroMQ Requirements
+#        files: ^(pkg/windows/(req|req_win)\.txt|requirements/((base|zeromq|pytest)\.txt|static/windows\.in))$
+#        args:
+#          - -v
+#          - --py-version=3.8
+#          - --platform=windows
+#          - --include=pkg/windows/req.txt
+#          - --include=pkg/windows/req_win.txt
+#          - --include=requirements/base.txt
+#          - --include=requirements/zeromq.txt
+#          - --include=requirements/pytest.txt
+
+      - id: pip-tools-compile
+        alias: compile-cloud-py3.8-requirements
+        name: Cloud Py3.8 Requirements
+        files: ^requirements/(static/cloud\.in)$
+        args:
+          - -v
+          - --py-version=3.8
+
+      - id: pip-tools-compile
+        alias: compile-doc-requirements
+        name: Docs Py3.8 Requirements
+        files: ^requirements/((base|zeromq|pytest)\.txt|static/docs\.in)$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=linux
+
+      - id: pip-tools-compile
+        alias: compile-linux-crypto-py3.8-requirements
+        name: Linux Py3.8 Crypto Requirements
+        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=linux
+          - --out-prefix=linux
+
+      - id: pip-tools-compile
+        alias: compile-darwin-crypto-py3.8-requirements
+        name: Darwin Py3.8 Crypto Requirements
+        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=darwin
+          - --out-prefix=darwin
+
+# Commented out since pywin32 and pymssql do not have packages or support for Py >= 3.8
+#      - id: pip-tools-compile
+#        alias: compile-windows-crypto-py3.8-requirements
+#        name: Windows Py3.8 Crypto Requirements
+#        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+#        args:
+#          - -v
+#          - --py-version=3.8
+#          - --platform=windows
+#          - --out-prefix=windows
+
+      - id: pip-tools-compile
+        alias: compile-lint-py3.8-requirements
+        name: Lint Py3.8 Requirements
+        files: ^requirements/static/lint\.in$
+        args:
+          - -v
+          - --py-version=3.8
+          - --platform=linux
+
+
+      - id: pip-tools-compile
+        alias: compile-linux-py3.9-zmq-requirements
+        name: Linux Py3.9 ZeroMQ Requirements
+        files: ^requirements/((base|zeromq|pytest)\.txt|static/linux\.in)$
+        exclude: ^requirements/static/(centos-6|amzn-2018\.03|lint|cloud|docs|darwin|windows)\.in$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=linux
+          - --include=requirements/base.txt
+          - --include=requirements/zeromq.txt
+          - --include=requirements/pytest.txt
+
+      - id: pip-tools-compile
+        alias: compile-darwin-py3.9-zmq-requirements
+        name: Darwin Py3.9 ZeroMQ Requirements
+        files: ^(pkg/osx/(req|req_ext)\.txt|requirements/((base|zeromq|pytest)\.txt|static/darwin\.in))$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=darwin
+          - --include=pkg/osx/req.txt
+          - --include=pkg/osx/req_ext.txt
+          - --include=requirements/base.txt
+          - --include=requirements/zeromq.txt
+          - --include=requirements/pytest.txt
+          - --passthrough-line-from-input=^pyobjc(.*)$
+
+# Commented out since pywin32 and pymssql do not have packages or support for Py >= 3.8
+#      - id: pip-tools-compile
+#        alias: compile-windows-py3.9-zmq-requirements
+#        name: Windows Py3.9 ZeroMQ Requirements
+#        files: ^(pkg/windows/(req|req_win)\.txt|requirements/((base|zeromq|pytest)\.txt|static/windows\.in))$
+#        args:
+#          - -v
+#          - --py-version=3.9
+#          - --platform=windows
+#          - --include=pkg/windows/req.txt
+#          - --include=pkg/windows/req_win.txt
+#          - --include=requirements/base.txt
+#          - --include=requirements/zeromq.txt
+#          - --include=requirements/pytest.txt
+
+      - id: pip-tools-compile
+        alias: compile-cloud-py3.9-requirements
+        name: Cloud Py3.9 Requirements
+        files: ^requirements/(static/cloud\.in)$
+        args:
+          - -v
+          - --py-version=3.9
+
+      - id: pip-tools-compile
+        alias: compile-doc-requirements
+        name: Docs Py3.9 Requirements
+        files: ^requirements/((base|zeromq|pytest)\.txt|static/docs\.in)$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=linux
+
+      - id: pip-tools-compile
+        alias: compile-linux-crypto-py3.9-requirements
+        name: Linux Py3.9 Crypto Requirements
+        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=linux
+          - --out-prefix=linux
+
+      - id: pip-tools-compile
+        alias: compile-darwin-crypto-py3.9-requirements
+        name: Darwin Py3.9 Crypto Requirements
+        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=darwin
+          - --out-prefix=darwin
+
+# Commented out since pywin32 and pymssql do not have packages or support for Py >= 3.8
+#      - id: pip-tools-compile
+#        alias: compile-windows-crypto-py3.9-requirements
+#        name: Windows Py3.9 Crypto Requirements
+#        files: ^requirements/(crypto\.txt|static/crypto\.in)$
+#        args:
+#          - -v
+#          - --py-version=3.9
+#          - --platform=windows
+#          - --out-prefix=windows
+
+      - id: pip-tools-compile
+        alias: compile-lint-py3.9-requirements
+        name: Lint Py3.9 Requirements
+        files: ^requirements/static/lint\.in$
+        args:
+          - -v
+          - --py-version=3.9
+          - --platform=linux
+
+  - repo: https://github.com/timothycrosley/isort
+    rev: "1e78a9acf3110e1f9721feb591f89a451fc9876a"
+    hooks:
+      - id: isort
+        additional_dependencies: ['toml']
+        # This tells pre-commit not to pass files to isort.
+        # This should be kept in sync with pyproject.toml
+        exclude: >
+            (?x)^(
+                templates/.*|
+                salt/ext/.*|
+                tests/kitchen/.*
+            )$
+
+  - repo: https://github.com/psf/black
+    rev: stable
+    hooks:
+      - id: black
+        # This tells pre-commit not to pass files to black.
+        # This should be kept in sync with pyproject.toml
+        exclude: >
+            (?x)^(
+                templates/.*|
+                salt/ext/.*|
+                tests/kitchen/.*
+            )$
+
 
   - repo: https://github.com/saltstack/salt-nox-pre-commit
     rev: master

diff --git a/noxfile.py b/noxfile.py
@@ -39,7 +39,7 @@
 IS_DARWIN = sys.platform.lower().startswith('darwin')
 IS_WINDOWS = sys.platform.lower().startswith('win')
 # Python versions to run against
-_PYTHON_VERSIONS = ('2', '2.7', '3', '3.4', '3.5', '3.6', '3.7')
+_PYTHON_VERSIONS = ("2", "2.7", "3", "3.4", "3.5", "3.6", "3.7", "3.8", "3.9")
 
 # Nox options
 #  Reuse existing virtualenvs
@@ -167,7 +167,7 @@ def _install_system_packages(session):
         '{id}-{version_parts[major]}'.format(**distro)
     ]
     version_info = _get_session_python_version_info(session)
-    py_version_keys = [
+    py_version_keys = +SITECUSTOMIZE_DIR[
         '{}'.format(*version_info),
         '{}.{}'.format(*version_info)
     ]

diff --git a/pkg/windows/req.txt b/pkg/windows/req.txt
@@ -1,6 +1,6 @@
 -r req_win.txt
 backports-abc==0.5; python_version < '3.0'
-backports.ssl-match-hostname==3.7.0.1
+backports.ssl-match-hostname==3.7.0.1; python_version < '3.7'
 certifi
 cffi==1.12.2
 CherryPy==17.4.1

diff --git a/pkg/windows/req_testing.txt b/pkg/windows/req_testing.txt
diff --git a/pkg/windows/req_win.txt b/pkg/windows/req_win.txt
@@ -1,2 +1,2 @@
-pywin32==224
+pywin32==227
 WMI==1.4.9
diff --git a/requirements/pytest.txt b/requirements/pytest.txt
@@ -1,7 +1,7 @@
 mock >= 3.0.0
 # PyTest
 pytest >=4.6.6,<4.7   # PyTest 4.6.x are the last Py2 and Py3 releases
-pytest-salt >= 2019.12.27
+pytest-salt >= 2020.1.27
 pytest-tempdir >= 2019.10.12
 pytest-helpers-namespace >= 2019.1.8
 pytest-salt-runtests-bridge >= 2019.7.10
diff --git a/requirements/static/py3.5/darwin.txt b/requirements/static/py3.5/darwin.txt
@@ -85,7 +85,7 @@ pyparsing==2.4.5          # via packaging
 pyserial==3.4             # via junos-eznc
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0

diff --git a/requirements/static/py3.5/linux.txt b/requirements/static/py3.5/linux.txt
@@ -85,7 +85,7 @@ pyparsing==2.4.5          # via packaging
 pyserial==3.4             # via junos-eznc
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0    # via botocore, croniter, kubernetes, moto, vcert

diff --git a/requirements/static/py3.5/windows.txt b/requirements/static/py3.5/windows.txt
@@ -10,7 +10,7 @@ atomicwrites==1.3.0       # via pytest
 attrs==19.1.0             # via pytest
 aws-xray-sdk==0.95        # via moto
 backports.functools-lru-cache==1.5  # via cheroot
-backports.ssl-match-hostname==3.7.0.1
+backports.ssl-match-hostname==3.7.0.1 ; python_version < "3.7"
 boto3==1.9.132
 boto==2.49.0
 botocore==1.12.132        # via boto3, moto, s3transfer
@@ -80,7 +80,7 @@ pyopenssl==19.0.0
 pyparsing==2.4.5          # via packaging
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0
@@ -90,7 +90,7 @@ python-jose==2.0.2        # via moto
 pythonnet==2.3.0
 pytz==2019.1              # via moto, tempora
 pyvmomi==6.7.1.2018.12
-pywin32==224
+pywin32==227
 pyyaml==5.1.2
 pyzmq==18.0.1 ; python_version != "3.4"
 requests==2.21.0

diff --git a/requirements/static/py3.6/darwin.txt b/requirements/static/py3.6/darwin.txt
@@ -84,7 +84,7 @@ pyparsing==2.4.5          # via packaging
 pyserial==3.4             # via junos-eznc
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0

diff --git a/requirements/static/py3.6/linux.txt b/requirements/static/py3.6/linux.txt
@@ -84,7 +84,7 @@ pyparsing==2.4.5          # via packaging
 pyserial==3.4             # via junos-eznc
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0    # via botocore, croniter, kubernetes, moto, vcert

diff --git a/requirements/static/py3.6/windows.txt b/requirements/static/py3.6/windows.txt
@@ -10,7 +10,7 @@ atomicwrites==1.3.0       # via pytest
 attrs==19.1.0             # via pytest
 aws-xray-sdk==0.95        # via moto
 backports.functools-lru-cache==1.5  # via cheroot
-backports.ssl-match-hostname==3.7.0.1
+backports.ssl-match-hostname==3.7.0.1 ; python_version < "3.7"
 boto3==1.9.132
 boto==2.49.0
 botocore==1.12.132        # via boto3, moto, s3transfer
@@ -79,7 +79,7 @@ pyopenssl==19.0.0
 pyparsing==2.4.5          # via packaging
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0
@@ -89,7 +89,7 @@ python-jose==2.0.2        # via moto
 pythonnet==2.3.0
 pytz==2019.1              # via moto, tempora
 pyvmomi==6.7.1.2018.12
-pywin32==224
+pywin32==227
 pyyaml==5.1.2
 pyzmq==18.0.1 ; python_version != "3.4"
 requests==2.21.0

diff --git a/requirements/static/py3.7/darwin.txt b/requirements/static/py3.7/darwin.txt
@@ -84,7 +84,7 @@ pyparsing==2.4.5          # via packaging
 pyserial==3.4             # via junos-eznc
 pytest-helpers-namespace==2019.1.8
 pytest-salt-runtests-bridge==2019.7.10
-pytest-salt==2019.12.27
+pytest-salt==2020.1.27
 pytest-tempdir==2019.10.12
 pytest==4.6.6
 python-dateutil==2.8.0