Python 3.12 support #1032

moreati · 2023-11-14T14:12:10Z

No description provided.

This prevents unreleased versions appearing on the website (e.g. 0.3.5.dev0), but introduces the risk of forgetting to update the website after a release. A better fix requires deeper design/workflow thought. refs mitogen-hq#1028

With current macOS 11 runner images (20231216.1) the `python` on `$PATH` is Python 3.12 and setuptools isn't installed by default. E.g. ``` python -mtox -e "py27-mode_localhost-ansible4" ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /Users/runner/work/_temp/93a29c4c-f606-45e4-8dbd-a4a5f51b8730.sh GLOB sdist-make: /Users/runner/work/1/s/setup.py ERROR: invocation failed (exit code 1), logfile: /Users/runner/work/1/s/.tox/log/GLOB-0.log ================================== log start =================================== Traceback (most recent call last): File "/Users/runner/work/1/s/setup.py", line 32, in <module> from setuptools import find_packages, setup ModuleNotFoundError: No module named 'setuptools' ``` Installing setuptools under Python 3.12 chooses package versions incompatible with Python 2.7. Additionally Mitogen isn't yet compatible with Python 3.12 (mitogen-hq#1033), so tests that call a local context with `python` fail.

moreati · 2024-03-06T21:53:37Z

docs/changelog.rst

@@ -21,6 +21,11 @@ Unreleased
 ----------

 * :gh:issue:`987` Support Python 3.11
+* :gh:issue:`885` Fix :py:exc:`PermissionError` in :py:mod:`importlib` when
+  becoming an unprvileged user with Python 3.x


psvmcc · 2024-03-07T07:58:44Z

Hi, just checked, my setup:

ansible [core 2.16.4]
  config file = /Users/ps/infra/ansible.cfg
  configured module search path = ['/Users/ps/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/ps/infra/.venv/lib/python3.12/site-packages/ansible
  ansible collection location = /Users/ps/infra/.collections
  executable location = /Users/ps/infra/.venv/bin/ansible
  python version = 3.12.1 (main, Dec  7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (/Users/ps/infra/.venv/bin/python)
  jinja version = 3.1.3
  libyaml = True

# git+https://github.com/moreati/mitogen.git@docs-download-url
mitogen                   0.3.5.dev0

ANSIBLE_VERSION_MAX patched to 2.16

Packaged modules(dnf/apt/...) doesn't works:

ANSIBLE_STRATEGY=mitogen_linear ansible localhost -m package -a 'pkg=vim' -D -C -i localhost, --check

output:

    "msg": "Unexpected failure during module execution: mitogen.core.StreamError: cannot unpickle 'ansible.utils.unsafe_proxy'/'AnsibleUnsafeText'\n  File \"<stdin>\", line 3868, in _dispatch_one\n  File \"<stdin>\", line 3851, in _parse_request\n  File \"<stdin>\", line 998, in unpickle\n  File \"<stdin>\", line 789, in find_class\n  File \"<stdin>\", line 899, in _find_global\n",

Hotfix in #1034 (comment) fixes this issue.

I'm not sure, it related or not to this MR.

I don't see any other issues here.

moreati · 2024-03-07T10:39:55Z

python version = 3.12.1 (main, Dec 7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (/Users/ps/infra/.venv/bin/python)
mitogen git+https://github.com/moreati/mitogen.git@docs-download-url
ANSIBLE_VERSION_MAX patched to 2.16
...

Unexpected failure during module execution: mitogen.core.StreamError: cannot unpickle 'ansible.utils.unsafe_proxy'/'AnsibleUnsafeText' ...

I don't see any other issues here.

Thanks, that's good to hear. My rough plan is to release 0.3.5 with this, followed by 0.3.6 with newer Ansible support.

upekkha · 2024-03-07T13:20:52Z

Same here: no issues so far. (Python 3.12.2, Ansible 9.3.0, this mitogen branch with the manual patch above.)

azmeuk · 2024-03-07T17:45:30Z

Hi @moreati.
Nice to see that you have a bit of time to spend on mitogen, and thank you for the hard work 🙏
I just tried to check the py312 PR after your call for comments on mastodon, but I am stuck to 0.3.3 because of #726. Any attempt to any of my servers fails with a sudo password is incorrect error with the master branch or this PR + python 3.12.

philfry · 2024-03-08T09:55:05Z

Great to see this project is still alive, many thanks @moreati!

I tested this pr together with #833, #991, #1034 (comment) and bumping ANSIBLE_VERSION_MAX to 2.16 on Fedora 39 (controller node) and OracleLinux 8 & 9 (managed nodes). Works without any errors 👍

kitos9112 · 2024-03-12T15:17:38Z

Great to see you're spending time on this project @moreati - I'm looking forward to testing this branch but we're in the the same boat as @azmeuk - #726 is a huge blocker for us.

…eadable On macOS when using a become plugin as an unprivileged user, to another unprivileged user it is likely that the current working directory can't be read. In this case os.cwd() raises PermissionError. On versions of Python currently in the wild (March 2024, CPython <= 3.13) if any non-builtin or non-frozen module (e.g. zlib, base64) is imported then `importlib._bootstrap_external.PathFinder._path_importer_cache()` attempts to call os.cwd() without catching PermissionError. The previous comment about needing an extra .encode() appears to be wrong, atleast for Python 3.x >= 3.6. Command size increased by 54 bytes, bootstrap by 804 bytes. Changed from codecs module to binascii & zlib because they're extensions, and importing them triggers fewer supporting imports (e.g. encodings module). Before ``` ✗ ./preamble_size.py SSH command size: 705 Bootstrap (mitogen.core) size: 17078 (16.68KiB) Original Minimized Compressed mitogen.parent 97884 95.6KiB 50515 49.3KiB 51.6% 12727 12.4KiB 13.0% mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5% mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4% mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6% mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8% mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB 14.1% mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB 16.8% mitogen.master 51398 50.2KiB 25715 25.1KiB 50.0% 6886 6.7KiB 13.4% ``` After ``` ✗ ./preamble_size.py SSH command size: 759 Bootstrap (mitogen.core) size: 17882 (17.46KiB) Original Minimized Compressed mitogen.parent 98173 95.9KiB 50571 49.4KiB 51.5% 12747 12.4KiB 13.0% mitogen.fork 8436 8.2KiB 4130 4.0KiB 49.0% 1648 1.6KiB 19.5% mitogen.ssh 10892 10.6KiB 6952 6.8KiB 63.8% 2113 2.1KiB 19.4% mitogen.sudo 12089 11.8KiB 5924 5.8KiB 49.0% 2249 2.2KiB 18.6% mitogen.select 12325 12.0KiB 2929 2.9KiB 23.8% 964 0.9KiB 7.8% mitogen.service 41699 40.7KiB 22477 22.0KiB 53.9% 5885 5.7KiB 14.1% mitogen.fakessh 15577 15.2KiB 7989 7.8KiB 51.3% 2623 2.6KiB 16.8% mitogen.master 56116 54.8KiB 29427 28.7KiB 52.4% 7627 7.4KiB 13.6% ``` Fixes mitogen-hq#885 Refs python/cpython#115911

importlib.machinery.ModuleSpec and find_spec() were introduced in Python 3.4 under PEP 451. They replace the find_module() API of PEP 302, which was deprecated from Python 3.4. They were removed in Python 3.12 along with the imp module. This change adds support for the PEP 451 APIs. Mitogen should no longer import imp on Python versions that support ModuleSpec. Tests have been added to cover the new APIs. CI jobs have been added to cover Python 3.x on macOS. Refs mitogen-hq#1033 Co-authored-by: Witold Baryluk <[email protected]>

…7/Vanilla

… 3.12

Most of the necessary changes were made in recent PEP 451 commits. This bumps the CI jobs, and declares the support. Test dependendancies are bumped to latest supportted/available versions. refs mitogen-hq#1033

moreati added 2 commits November 14, 2023 14:05

docs: Bypass networkgenomics.com/try/ -> PyPI redirect

e580258

refs mitogen-hq#1028

moreati force-pushed the docs-download-url branch from 886a8c3 to 4029fd9 Compare November 14, 2023 15:28

moreati force-pushed the docs-download-url branch 6 times, most recently from b3d0d57 to 0603f66 Compare January 9, 2024 16:47

moreati added 2 commits January 10, 2024 10:04

non functional: Add comments about imp module removal in Python 3.12

fc3e788

moreati force-pushed the docs-download-url branch 13 times, most recently from 6f6726f to 56b354a Compare February 8, 2024 14:21

moreati force-pushed the docs-download-url branch 6 times, most recently from fdcd596 to fa14b42 Compare February 9, 2024 15:43

moreati force-pushed the docs-download-url branch 13 times, most recently from 7aff7a0 to 1801378 Compare March 6, 2024 13:05

moreati changed the title ~~Fix website download URL~~ Python 3.12 support Mar 6, 2024

moreati commented Mar 6, 2024

View reviewed changes

moreati mentioned this pull request Mar 6, 2024

Python 3.12 support #1033

Closed

moreati added 5 commits March 17, 2024 14:54

tests: Skip "discovered python matches invoked" on macOS 11/Python 2.…

92c00d9

…7/Vanilla

ansible_mitogen: Remove use of distutils, which was removed in Python…

fe8a3a7

… 3.12

mitogen: Support Python 3.12

123efa7

Most of the necessary changes were made in recent PEP 451 commits. This bumps the CI jobs, and declares the support. Test dependendancies are bumped to latest supportted/available versions. refs mitogen-hq#1033

moreati force-pushed the docs-download-url branch from 1801378 to 123efa7 Compare March 17, 2024 14:58

moreati merged commit a210c37 into mitogen-hq:master Mar 17, 2024
43 checks passed

moreati deleted the docs-download-url branch March 17, 2024 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.12 support #1032

Python 3.12 support #1032

moreati commented Nov 14, 2023

moreati Mar 6, 2024

psvmcc commented Mar 7, 2024

moreati commented Mar 7, 2024

upekkha commented Mar 7, 2024

azmeuk commented Mar 7, 2024

philfry commented Mar 8, 2024

kitos9112 commented Mar 12, 2024

Python 3.12 support #1032

Python 3.12 support #1032

Conversation

moreati commented Nov 14, 2023

moreati Mar 6, 2024

Choose a reason for hiding this comment

psvmcc commented Mar 7, 2024

moreati commented Mar 7, 2024

upekkha commented Mar 7, 2024

azmeuk commented Mar 7, 2024

philfry commented Mar 8, 2024

kitos9112 commented Mar 12, 2024