Deprecate lib2to3 (and 2to3) for future removal #84540
Comments
|
Based on the PEP-617 acceptance thread on python-dev, lib2to3 is eventually going to run into trouble parsing modern syntax a few releases from now. It would be better off maintained outside of the standard library. It gets used by a lot of things and is generally useful, but would make a lot more sense as a PyPI project than as something only quasi-maintained within the stdlib (it only gained the ability to parse a couple modern syntax features in via bugfix contributions to the stdlib the past month or two... meaning a lot of versions of it out there cannot) Black has already forked it. goal: PendingDeprecationWarning and documentation as such in 3.9. Move to DeprecationWarning in 3.10 or 3.11 and remove it by ~3.12. Subject to our existing deprecation process guidelines. |
gpshead
added
type-feature
|
I am in favor of this. We could promote LibCST, which is based on Parso, which uses a forked version of pgen2 (the parser in lib2to3). I believe one of these could switch to a fork of pegen as its parser, so it will be able to handle new PEG based syntax in 3.10+. Removal by 3.12 might be feasible. |
|
I volunteered in the python-dev thread to write a patch to the docs clarifying future status of lib2to3; happy to include the PendingDeprecationWarning as well. Re linking to alternatives, we want to make sure we link to alternatives that are committed to updating to support newer Python versions' syntax. This definitely includes LibCST; I can inquire with the parso maintainer about whether it also includes parso. In future it could also include a third-party-maintained copy of lib2to3, if someone picks that up. |
|
I opened a PR. It deprecates the lib2to3 library to discourage future use of it for Python3, but not the 2to3 tool. This of course means that the lib2to3 module will in practice stick around in the stdlib as long as 2to3 is still bundled with Python. It seems like the idea in this issue is to deprecate and remove both. I'm not sure what we typically do to deprecate a command-line utility bundled with Python. Given warnings are silent by default, the deprecation warning for lib2to3 won't be visible to users of 2to3. Should I add something to its |
|
Okay,the pending deprecation is in. Keeping open as a reminder to turn that into a real DeprecationWarning in 3.10 after the 3.9 branch is cut. We'll then want to track reminding us to remove it in 3.12. |
|
@gregory.p.smith What do you think about the question I raised above about how to make this deprecation visible to users of the 2to3 CLI tool, assuming the plan is to remove both? |
|
I think what we're doing with the documentation update is fine. We can add a warning on stderr to the tool in 3.11. But I don't expect people will be using the tool _from_ the latest CPython 3.x by then. 2to3 is already included with Python 2.7 and the only real use for it is for people who still have code they maintain on 2.7 so they've got a copy already. There is no value in running a 2to3 shipped with Python 3 vs the latest 2.7. Meaningful updates to it were already back ported to 2.7 over time as it was intentionally exempt from feature freeze. We should have sorted out a PyPI home for lib2to3 by 3.11 time and can also create a PyPI package for the 2to3 tool itself at that point. I _think_ there is support for running 2to3 on sources at package install time from setup.py? But I don't expect anything actually maintained and widely used to require that by the time this deprecation lands. If it does, that becomes a plumbing issue within package tools to know that requiring 2to3 at either build or install time adds an implicit tool dependency on the new pypi package to get it. Maybe I'm just in a good mood about all of this, but none of this seems worrisome. |
|
The documentation change gives two possible successors: https://libcst.readthedocs.io/ (https://github.com/Instagram/LibCST) And I've also seen this mentioned: https://github.com/pyga/awpa Is it possible to settle on one of these as the successor to the lib2to3 parser? It would be nice to avoid a 2nd deprecation in the future ... |
|
It's typically not up to the core devs to pick a winning third party library; we tend to recommend libraries that are already essentially category winners, like requests. In a sense pointing to LibCST *and* parso is redundant because LibCST builds on parso. Comparing stars on GitHub:
|
|
Right, although I think it still makes sense to link both LibCST and parso since they provide different levels of abstraction that would be suitable for different types of tools (e.g. I would rather write an auto-formatter on top of parso, because LibCST's careful parsing and assignment of whitespace would mostly just get in the way, but I'd rather write any kind of refactoring tooling on top of LibCST.) Another tool that escaped my mind when writing the PR that should probably be linked also is Baron/RedBaron (https://github.com/PyCQA/redbaron); 457 stars makes it slightly more popular than LibCST (but it's also been around a lot longer.) |
|
Coul you please add a what's new entry for this change? |
|
I don't understand why there is a PendingDeprecationWarning and not a DeprecationWarning. See https://discuss.python.org/t/pendingdeprecationwarning-is-really-useful/1038/4 and bpo-36404 |
The committed change already included an entry in NEWS. Is a "What's New" entry something different?
Purely because I was following gps' recommendation in the first comment on this issue. Getting rid of PendingDeprecationWarning seems like an orthogonal decision; if it happens, this can trivially be upgraded to DeprecationWarning as part of a removal sweep. |
|
A "What's New" entry would go into Doc/whatsnew/3.9.rst and is much more visible to users looking for exciting bits in the new release (the NEWS file is very large, see e.g. https://docs.python.org/3/whatsnew/changelog.html#changelog. The What's New doc typically has a section collecting all the deprecations, e.g. https://docs.python.org/3/whatsnew/3.8.html#deprecated. |
My thought was that the decision was already made to do so. Hence adding new PendingDeprecationWarnings goes against that decision. But maybe I misunderstand and that decision was not made. |
|
IIRC PendingDeprecationError does not mean that the decision hasn't been made yet. It just means it's less urgent for folks to worry about. I believe we tend to change PendingDeprecationError to DeprecationError in the last release before something is removed. |
|
Thanks for the explanation. I plan to send a PR to add this to the What's new in 3.9 page early next week. Anyone, feel free to beat me to it. |
|
FYI the autopep8 project uses lib2to3. |
|
Looking at the suggested successor tools (redbaron, libCST, parso, awpa) ... all of them appear to use some variant of pgen2. But at some point Python will be using a PEG approach (PEP-617), and therefor the pgen2 approach apparently won't work. For a number of projects, it's important to have a parse tree that contains all the "whitespace" information (indent, dedent, comment, newline, etc.) As far as I can tell, the new PEG parser won't provide that, and it seems that none of the successor tools will be able to handle future versions of Python syntax. So, three questions:
If Lib/ast.py continues to be supported, I think I can see a way of providing functionality similar to lib2to3 (in terms of an AST-ish thing with "whitespace" from the source, sufficient for tools such as yapf, black, pykythe, pytype, mypy, etc.) as a kind of wrapper to ast.py. |
|
There's no python-dev discussion; if you want more feedback I recommend starting on python-ideas first (on either forum you may expect pushback because this is not about a proposed change to Python or its workflow). The Lib/ast.py module will continue to be the official API for the standard AST. It is a simple wrapper around the builtin parser (at least in CPython -- I don't actually know to what extent other Python implementations support it, but they certainly *could*). And in 3.9 and later the AST is already being produced using the *new* parser. We want to deprecate lib2to3 because nobody is interested in maintaining it., Having it in the stdlib, with its strict backwards compatibility requirements, makes it difficult to do a good job at updating it. This is why it's been forked repeatedly -- once forked, the owner of the fork can make changes easily, preserving the API perfectly (if so desired) and maintaining compatibility with older Python versions. My own thoughts are that libraries like LibCST and parso have two sides: an API for the AST, and a way to parse source code into an AST. Usually the parsing API is incredibly simple -- e.g. a function to parse a file and another function to parse a string. And there's no reason for the AST API to change just because the parsing algorithm has changed. Finally, we already have a (rough) Python implementation of the PEG parser too -- in fact it's included in Tools/peg_generator (and used to regenerate the metaparser). This reads the same grammar format (i.e. Grammar/python.gram) and generates Python code instead of C code to do the parsing. It's easy to retarget the tokenizer of the generated Python code. So a decent way forward might be to pick one of the 3rd party libraries (perhaps parso, which is itself a fork of lib2to3 and what LibCST builds on) and update its parser to use a PEG parser generated using the PEG generator from Tools/peg_generator (which people are welcome to fork). This might be a summer-of-code-sized project. |
|
I created PR 28116 to deprecate the lib2to3 package: replace PendingDeprecationWarning to DeprecationWarning. In 2021, I don't think that we should keep the 2to3 tool in the stdlib. Python 2 reached end of line 1 year ago. For the lib2to3 *parser*, IMO it would be better to maintain it outside the stdlib, and collaborate with other existing forks, like the parser used by Black. The change is only about *deprecating* lib2to3, not *remove* it. For the removal, we should check if major projects which used it moved to something else. It's kind of a shame that Python stdlib (lib2to3) cannot parse valid Python 3.10 code :-( IMO it's better to deprecate (and then remove) lib2to3. |
|
I retarget this issue to Python 3.11, since lib2to3 is *not* deprecated in Python 3.10. |
|
How come the deprecation didn't happen in 3.10? Were people just not |
|
I think we just forgot to make the change in time. 3.11 is fine. We're not _maintaining_ lib2to3 or describing it as fit for any modern purpose regardless. It's just code that'll sit around in the back of the 3.10 stdlib but not be able to parse the new syntax in 3.10. |
|
Guido: How come the deprecation didn't happen in 3.10? Were people just not interested? Well, if nobody deprecates it, it's not deprecated. It is simple as it it :-) IMO it's ok to only deprecate it in Python 3.11, unless Pablo *really* wants to deprecate lib2to3 before just Python 3.10 final. I dislike adding new warnings between a beta1 release and the final release :-( In my experience, it *does* break projects which care of warnings. |
|
We can add to the 3.10 docs that it is deprecated without any code change. And in 3.11 we can add a warning. |
This was already the case: The wording was a bit clumsy so I rephrased in #72309. |
|
I close the issue: lib2to3 is now deprecated in Python 3.11. I propose to open a new issue in Python 3.13 or newer to remove it. |
|
The "pending" deprecation status of lib2to3 in 3.9 and 3.10 is no worse than a vanilla deprecation in terms of visibility. It will appear just the same when run with pytest or However, upgrading the deprecation between 3.10.0rc1 and 3.10.0rc2 really was too late. So we'll have it deprecated fully in 3.11 and removed in 3.13. One more year doesn't hurt us much but might be helpful for library maintainers to have more time to move off of the included lib2to3. Invoking $ ./python.exe Tools/scripts/2to3
/private/tmp/cpy2/Tools/scripts/2to3:3: DeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+
from lib2to3.main import main
At least one file or directory argument required.
Use --help to show usage.Thanks for the patches, Gregory, Carl, and Victor! ✨ 🍰 ✨ Now we just have to remember to actually remove the damn thing in 3.13 😂 |
|
Created bpo-45544 to close all open issues and list them there. |
|
What does this mean for someone who picks up a bit of code that happens to be obsolete and then starts getting warnings about print statements and lots of other stuff? |
|
@johns1c -- It means that you should ask for help somewhere else. :-) Seriously, try explaining your problem in our Discourse server at discuss.python.org. |
|
For people who still have a Python 2 code base: download/install Python 3.12 and run 2to3 included in this version. Or give a try to 2to3 alternatives. |
|
Thanks for your reply.
This works for Python 3.11 with 2to3.exe being found in the `Scripts` directory of my Windows 11 64 bit install and 2to3.py being found in `Tools\scripts` (it was ‘tools\scripts’ in 3.8)
Interestingly the Windows install for Python 3.12 does NOT provide either the library or the runnable utility.
From now on it seems that a user wishing to upgrade old Python 2 code needs to install 2to3 using PIP. The command `PIP install 2to3` will install 2to3.exe in the `Scripts` directory under the Python install directory or in the `scripts` directory of a virtual environment.
Chris J
From: Victor Stinner
Sent: 14 November 2023 12:25
To: python/cpython
Cc: C Johnson; Mention
Subject: Re: [python/cpython] Deprecate lib2to3 (and 2to3) for future removal(#84540)
For people who still have a Python 2 code base: download/install Python 3.12 and run 2to3 included in this version. Or give a try to 2to3 alternatives.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: