gh-98040: Remove just the `imp` module #98573

warsaw · 2022-10-23T18:55:47Z

@ericsnowcurrently - here's a branch that removes just the imp module. It still has failures because of this pip issue.

Issue: Remove more deprecated importlib APIs from Python 3.12 #98040

ericsnowcurrently

LGTM

(pending the pip fix, of course)

ericsnowcurrently · 2022-11-02T16:51:14Z

Thanks for splitting this out. It was much easier to review. 😄

iritkatriel

This has a merge conflict now.

bedevere-bot · 2022-11-25T22:44:14Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

ericsnowcurrently · 2023-03-03T18:24:37Z

Sorry about the churn that caused conflicts for you, @warsaw! I don't expect I'll be making any more changes to imp.py or test_imp.py before you merge this.

ericsnowcurrently · 2023-03-09T16:40:42Z

@warsaw, FYI, I'm moving the tests I recently added to test_imp over to test_import: gh-102561.

While I was doing that, I noticed there are other tests in test_imp that seem to be more about exercising the import system than the imp module itself, which may make it worth it to keep them. Assuming those ones really aren't imp-specific, do you know if any of them are not already covered by other tests, e.g. in test_import or test_importlib? If so, we should move them out before test_imp is deleted. I'd be glad to do so if it would help (if there are any).

cpython-cla-bot · 2023-04-11T17:52:01Z

All commit authors signed the Contributor License Agreement.

arhadthedev · 2023-04-11T18:26:10Z

It seems that whoever wanted to merge main into this PR cherry-picked it instead.

The previous head was f45d6a4. I would like to leave the restoration force-push to core developers.

CAM-Gerlach · 2023-04-11T20:20:49Z

FYI, the actual head of this PR branch was 6159e52 . It actually turned out to be fairly straightforward to fix with some rebaseing, i.e.

git fetch upstream
git rebase upstream/main 6159e52ea7000499260d18cee833761811a67fa1

I've (force) pushed the fixed version. (Also accidentally hit the "ready for review button", which I've reversed for now).

CAM-Gerlach · 2023-04-11T20:44:52Z

So, grepping a bit, looks like imp is still referred to various potentially undesired places:

A few places in the docs, including at least one spot that's causing the docs build to fail, as its resulting in a broken reference in a clean file—I can help fix those
The stdlib_module_names.h file, which is causing the check for generated files to fail, which I'm assuming is due to Eric & Co's work and can be fixed by regenerating it via Tools/build/generate_stdlib_module_names.py — it doesn't seem to work on Windows
Some comments in the code that still refer to it
A couple references in the opcode generation as well
Used in the (possibly legacy) importbench script in Tools—should that code or the whole script just be removed, or does it need to be manually converted to importlib? Not sure if its still actually used now that we have a lot of other comprehensive benchmark sources like pyperformance—the script itself mentions this (was last touched 4 years ago).

I'm assuming the test failures due to ensurepip are expected, as referred to above, so I ignored those. I also ignored the references to the very low-level C _imp module used by importlib (and previously imp).

Here's a full list, pruned for false positives and instances that should stay (NEWS, what's new, historical notes, etc):

CAM-Gerlach · 2023-04-11T22:00:26Z

Also, not totally sure why, but when I build this commit, I get this delta in the generated files which I cannot reproduce on the main parent commit, adding zipimporter a few places:

diff --git a/Include/internal/pycore_global_objects_fini_generated.h b/Include/internal/pycore_global_objects_fini_generated.h
index 14dfd9ea582..d79b56574f7 100644
--- a/Include/internal/pycore_global_objects_fini_generated.h
+++ b/Include/internal/pycore_global_objects_fini_generated.h
@@ -1232,6 +1232,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) {
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(x));
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(year));
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(zdict));
+    _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(zipimporter));
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_SINGLETON(strings).ascii[0]);
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_SINGLETON(strings).ascii[1]);
     _PyStaticObject_CheckRefcnt((PyObject *)&_Py_SINGLETON(strings).ascii[2]);
diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h
index 6f430bb25eb..77169765294 100644
--- a/Include/internal/pycore_global_strings.h
+++ b/Include/internal/pycore_global_strings.h
@@ -718,6 +718,7 @@ struct _Py_global_strings {
         STRUCT_FOR_ID(x)
         STRUCT_FOR_ID(year)
         STRUCT_FOR_ID(zdict)
+        STRUCT_FOR_ID(zipimporter)
     } identifiers;
     struct {
         PyASCIIObject _ascii;
diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h
index 0452c4c6155..82014a8b10d 100644
--- a/Include/internal/pycore_runtime_init_generated.h
+++ b/Include/internal/pycore_runtime_init_generated.h
@@ -1224,6 +1224,7 @@ extern "C" {
     INIT_ID(x), \
     INIT_ID(year), \
     INIT_ID(zdict), \
+    INIT_ID(zipimporter), \
 }

 #define _Py_str_ascii_INIT { \
diff --git a/Include/internal/pycore_unicodeobject_generated.h b/Include/internal/pycore_unicodeobject_generated.h
index 7114a5416f2..6d4dcc229aa 100644
--- a/Include/internal/pycore_unicodeobject_generated.h
+++ b/Include/internal/pycore_unicodeobject_generated.h
@@ -2007,6 +2007,9 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) {
     string = &_Py_ID(zdict);
     assert(_PyUnicode_CheckConsistency(string, 1));
     _PyUnicode_InternInPlace(interp, &string);
+    string = &_Py_ID(zipimporter);
+    assert(_PyUnicode_CheckConsistency(string, 1));
+    _PyUnicode_InternInPlace(interp, &string);
 }
 /* End auto-generated code */
 #ifdef __cplusplus

warsaw · 2023-04-11T23:50:52Z

@warsaw, FYI, I'm moving the tests I recently added to test_imp over to test_import: gh-102561.

While I was doing that, I noticed there are other tests in test_imp that seem to be more about exercising the import system than the imp module itself, which may make it worth it to keep them. Assuming those ones really aren't imp-specific, do you know if any of them are not already covered by other tests, e.g. in test_import or test_importlib? If so, we should move them out before test_imp is deleted. I'd be glad to do so if it would help (if there are any).

I missed this comment earlier but yes, we should audit test_imp.py for general import system tests and move them over. I plan on working on this at Pycon.

warsaw · 2023-04-11T23:52:03Z

So, grepping a bit, looks like imp is still referred to various potentially undesired places

Thanks for doing this audit @CAM-Gerlach !

vstinner · 2023-04-13T12:56:37Z

The imp module is deprecated for 9 years, since Python 3.4 (2014). Problem: a search for (from imp\b|import imp\b) regex in PyPI top 5,000 projects match 153 projects (260 lines).

That's why I didn't propose a PR to remove it in Python 3.12.

It would be nice for the Python community to collaborate with these projects to reduce the number of impacted projects before removing the imp module.

Example of projects:

ansible-7.4.0
boto-2.49.0
cffi-1.15.1
conda-4.3.16
django-compat-1.0.15
eventlet-0.33.3
lxml-4.9.2
matplotlib-3.7.1
mercurial-6.4.1
nose-1.3.7
Paste-3.5.2
pip-23.0.1
pipenv-2023.3.20
reportlab-3.6.12
setuptools-67.6.1
Twisted-22.10.0
virtualenv-20.21.0

Full list of projects:

* aiosmtpd-1.4.4.post2 * ansible-7.4.0 * ansible-core-2.14.4 * ansiwrap-0.8.4 * apache-libcloud-3.7.0 * argcomplete-3.0.5 * Arpeggio-2.0.0 * asttokens-2.2.1 * audioread-3.0.0 * auth-0.5.3 * Autologging-1.3.2 * autopep8-2.0.2 * better_exceptions-0.3.3 * boto-2.49.0 * bottle-0.12.25 * cachy-0.3.0 * casadi-3.6.0 * cement-3.0.8 * Cerberus-1.3.4 * cffi-1.15.1 * Cheetah3-3.2.6.post1 * circus-0.18.0 * comtypes-1.1.14 * conda-4.3.16 * conda-pack-0.6.0 * coremltools-6.3.0 * crochet-2.0.0 * Cython-0.29.34 * databricks-cli-0.17.6 * datalab-1.2.1 * dateparser-1.1.8 * dbnd-1.0.12.12 * ddtrace-1.11.2 * debugpy-1.6.7 * diff-match-patch-20200713 * distlib-0.3.6 * distribute-0.7.3 * django-compat-1.0.15 * django_compressor-4.3.1 * django-configurations-2.4.1 * django-filter-23.1 * django-grappelli-3.0.5 * django-heroku-0.3.1 * django-nine-0.2.7 * django-user_agents-0.4.0 * dominate-2.7.0 * dtale-2.14.0 * dvc-2.54.0 * eventlet-0.33.3 * fastrlock-0.8.1 * fire-0.5.0 * flasgger-0.9.5 * Flask-AppBuilder-4.3.1 * flatbuffers-23.3.3 * func_timeout-4.3.5 * future-0.18.3 * gevent-22.10.2 * glom-23.3.0 * google-compute-engine-2.8.13 * grafanalib-0.7.0 * gsutil-5.23 * h2o-3.40.0.3 * hachoir-3.2.0 * hdfs-2.7.0 * hjson-3.1.0 * hyper-0.7.0 * imageio-2.27.0 * IMAPClient-2.3.1 * importlab-0.8 * invoke-2.0.0 * j2cli-0.3.10 * joblib-1.2.0 * Kivy-2.1.0 * logilab-common-1.9.8 * lxml-4.9.2 * markdown2-2.4.8 * matplotlib-3.7.1 * mercurial-6.4.1 * metaflow-2.8.3 * ml_collections-0.1.1 * multiprocessing-2.6.2.1 * mysql-connector-2.2.9 * mysql-connector-python-rf-2.2.2 * newrelic-8.8.0 * nose-1.3.7 * numba-0.56.4 * oletools-0.60.1 * os_sys-2.1.4 * Paste-3.5.2 * pathtools-0.1.2 * pbr-5.11.1 * pdm-2.5.2 * pep562-1.1 * pex-2.1.131 * pip-23.0.1 * pipenv-2023.3.20 * plac-1.3.5 * ply-3.11 * psutil-5.9.4 * ptvsd-4.3.2 * pvlib-0.9.5 * py2neo-2021.2.3 * pycryptodome-3.17 * pycryptodomex-3.17 * pydevd-2.9.5 * pydevd-pycharm-231.8109.197 * pydoop-2.0.0 * pyexcel-0.7.0 * pyhocon-0.3.60 * pyinstaller-5.10.0 * pyLDAvis-3.4.0 * Pympler-1.0.1 * pynose-1.4.2 * pyodbc-4.0.38 * pyopencl-2022.3.1 * pyrepl-0.9.0 * Pyro4-4.82 * pysaml2-7.4.1 * pysmi-0.3.4 * pysnmp-4.4.12 * pytest-openfiles-0.5.0 * pytest-shutil-1.7.0 * python-crfsuite-0.9.9 * python-i18n-0.3.9 * PyXB-1.2.6 * pyxdg-0.28 * raven-6.10.0 * rcssmin-1.1.1 * reportlab-3.6.12 * rjsmin-1.2.1 * SCons-4.5.2 * scrypt-0.8.20 * selinux-0.3.0 * setupmeta-3.4.0 * setuptools-67.6.1 * simplejson-3.19.1 * snapshottest-0.6.0 * social-auth-core-4.4.1 * sox-1.4.1 * sqlalchemy-migrate-0.13.0 * stone-3.3.1 * strip-hints-0.1.10 * sympy-1.11.1 * tecton-0.6.5 * Theano-1.0.5 * Twisted-22.10.0 * twofish-0.3.0 * uwsgi-2.0.21 * virtualenv-20.21.0 * vowpalwabbit-9.8.0 * workflow-2.1.6 * yacs-0.1.8 * Yapsy-1.12.2

vstinner · 2023-04-13T13:02:05Z

pip and setuptools:

PYPI-2023-04-13/pip-23.0.1.tar.gz: pip-23.0.1/src/pip/_vendor/distlib/wheel.py: import imp
PYPI-2023-04-13/pip-23.0.1.tar.gz: pip-23.0.1/src/pip/_vendor/pkg_resources/__init__.py: import imp as _imp
PYPI-2023-04-13/setuptools-67.6.1.tar.gz: setuptools-67.6.1/pkg_resources/__init__.py: import imp as _imp

The `imp` module has been deprecated from py3.4 [1] and will be removed in py3.12 [2]. This patch is required to use pythran on the current main branch of Python. [1] https://docs.python.org/3.11/library/imp.html?highlight=imp#module-imp [2] python/cpython#98573

bedevere-bot added the awaiting core review label Oct 23, 2022

ericsnowcurrently approved these changes Nov 2, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Nov 2, 2022

iritkatriel previously requested changes Nov 25, 2022

View reviewed changes

bedevere-bot removed the awaiting merge label Nov 25, 2022

bedevere-bot added the awaiting changes label Nov 25, 2022

bedevere-bot mentioned this pull request Jan 12, 2023

Remove more deprecated importlib APIs from Python 3.12 #98040

Closed

6 tasks

matusvalo mentioned this pull request Mar 4, 2023

Cython 0.29.x Python 3.12.0a5+ tracker cython/cython#5285

Closed

ericsnowcurrently mentioned this pull request Mar 9, 2023

gh-98040: Move the Single-Phase Init Tests Out of test_imp #102561

Merged

warsaw added 3 commits April 11, 2023 15:14

pythongh-98040: Remove just the imp module

44cd65d

Fix some markup

e4f857d

Remove imp module documentation

cc469c4

CAM-Gerlach force-pushed the remove-imp branch from 0878460 to cc469c4 Compare April 11, 2023 20:18

CAM-Gerlach marked this pull request as ready for review April 11, 2023 20:18

CAM-Gerlach marked this pull request as draft April 11, 2023 20:21

Merge branch 'main' into remove-imp

b3c3517

tacaswell mentioned this pull request May 1, 2023

[BUG]: Python 3.12 will remove deprecated ImpImporter from pgkutil pybind/pybind11#4646

Closed

3 tasks

This was referenced May 3, 2023

Meta issue for cleaning up import system cruft #97850

Open

Replace deprecated imp module with importlib itamarst/crochet#145

Closed

Replace deprecated imp module with importlib fabioz/PyDev.Debugger#251

Open

hugovk mentioned this pull request May 11, 2023

Importing imp in standard_library raises a PendingDeprecationWarning PythonCharmers/python-future#246

Open

This was referenced Jun 8, 2023

Module imp was removed in 3.12 python/typeshed#10278

Merged

Update pkgutil.pyi for Python3.12 python/typeshed#10281

Merged

sarnold mentioned this pull request Jul 17, 2023

setuptools in rawhide broken on Python 3.12 ComplianceAsCode/content#10862

Closed

encukou mentioned this pull request Oct 7, 2024

Improve Testing For Importing Multiple Extension Modules From One File (and Deal With _testimportmultiple.c) #124978

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-98040: Remove just the `imp` module #98573

gh-98040: Remove just the `imp` module #98573

warsaw commented Oct 23, 2022 •

edited by bedevere-bot

Loading

ericsnowcurrently left a comment

ericsnowcurrently commented Nov 2, 2022

iritkatriel left a comment

bedevere-bot commented Nov 25, 2022

ericsnowcurrently commented Mar 3, 2023

ericsnowcurrently commented Mar 9, 2023

cpython-cla-bot bot commented Apr 11, 2023 •

edited

Loading

arhadthedev commented Apr 11, 2023

CAM-Gerlach commented Apr 11, 2023 •

edited

Loading

CAM-Gerlach commented Apr 11, 2023 •

edited by warsaw

Loading

CAM-Gerlach commented Apr 11, 2023

warsaw commented Apr 11, 2023

warsaw commented Apr 11, 2023

vstinner commented Apr 13, 2023

vstinner commented Apr 13, 2023

gh-98040: Remove just the imp module #98573

gh-98040: Remove just the imp module #98573

Conversation

warsaw commented Oct 23, 2022 • edited by bedevere-bot Loading

ericsnowcurrently left a comment

Choose a reason for hiding this comment

ericsnowcurrently commented Nov 2, 2022

iritkatriel left a comment

Choose a reason for hiding this comment

bedevere-bot commented Nov 25, 2022

ericsnowcurrently commented Mar 3, 2023

ericsnowcurrently commented Mar 9, 2023

cpython-cla-bot bot commented Apr 11, 2023 • edited Loading

arhadthedev commented Apr 11, 2023

CAM-Gerlach commented Apr 11, 2023 • edited Loading

CAM-Gerlach commented Apr 11, 2023 • edited by warsaw Loading

CAM-Gerlach commented Apr 11, 2023

warsaw commented Apr 11, 2023

warsaw commented Apr 11, 2023

vstinner commented Apr 13, 2023

vstinner commented Apr 13, 2023

gh-98040: Remove just the `imp` module #98573

gh-98040: Remove just the `imp` module #98573

warsaw commented Oct 23, 2022 •

edited by bedevere-bot

Loading

cpython-cla-bot bot commented Apr 11, 2023 •

edited

Loading

CAM-Gerlach commented Apr 11, 2023 •

edited

Loading

CAM-Gerlach commented Apr 11, 2023 •

edited by warsaw

Loading