Merge branch 'main' into add-warning-options-and-include-in-check-war…

…nings

.gitattributes

@@ -27,8 +27,6 @@ Lib/test/cjkencodings/*                    noeol
 Lib/test/tokenizedata/coding20731.py       noeol
 Lib/test/decimaltestdata/*.decTest         noeol
 Lib/test/test_email/data/*.txt             noeol
-Lib/test/test_importlib/resources/data01/*           noeol
-Lib/test/test_importlib/resources/namespacedata01/*  noeol
 Lib/test/xmltestdata/*                     noeol
 
 # Shell scripts should have LF even on Windows because of Cygwin

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.3.4
+    rev: v0.4.10
     hooks:
       - id: ruff
         name: Run Ruff (lint) on Doc/
@@ -20,15 +20,15 @@ repos:
         files: ^Doc/
 
   - repo: https://github.com/psf/black-pre-commit-mirror
-    rev: 24.4.2
+    rev: 24.8.0
     hooks:
       - id: black
         name: Run Black on Tools/jit/
         files: ^Tools/jit/
         language_version: python3.12
 
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.5.0
+    rev: v4.6.0
     hooks:
       - id: check-case-conflict
       - id: check-merge-conflict
@@ -42,7 +42,7 @@ repos:
         types_or: [c, inc, python, rst]
 
   - repo: https://github.com/sphinx-contrib/sphinx-lint
-    rev: v0.9.1
+    rev: v1.0.0
     hooks:
       - id: sphinx-lint
         args: [--enable=default-role]

Android/README.md

@@ -12,8 +12,12 @@ approachable user experience:
 
 ## Prerequisites
 
-Export the `ANDROID_HOME` environment variable to point at your Android SDK. If
-you don't already have the SDK, here's how to install it:
+First, make sure you have all the usual tools and libraries needed to build
+Python for your development machine.
+
+Second, you'll need an Android SDK. If you already have the SDK installed,
+export the `ANDROID_HOME` environment variable to point at its location.
+Otherwise, here's how to install it:
 
 * Download the "Command line tools" from <https://developer.android.com/studio>.
 * Create a directory `android-sdk/cmdline-tools`, and unzip the command line
@@ -37,11 +41,6 @@ development tools, which currently means Linux or macOS. This involves doing a
 cross-build where you use a "build" Python (for your development machine) to
 help produce a "host" Python for Android.
 
-First, make sure you have all the usual tools and libraries needed to build
-Python for your development machine. The only Android tool you need to install
-is the command line tools package above: the build script will download the
-rest.
-
 The easiest way to do a build is to use the `android.py` script. You can either
 have it perform the entire build process from start to finish in one step, or
 you can do it in discrete steps that mirror running `configure` and `make` for
@@ -80,12 +79,15 @@ call. For example, if you want a pydebug build that also caches the results from
 
 ## Testing
 
-The tests can be run on Linux, macOS, or Windows, although on Windows you'll
-have to build the `cross-build/HOST` subdirectory on one of the other platforms
-and copy it over.
+The test suite can be run on Linux, macOS, or Windows:
+
+* On Linux, the emulator needs access to the KVM virtualization interface, and
+  a DISPLAY environment variable pointing at an X server.
+* On Windows, you won't be able to do the build on the same machine, so you'll
+  have to copy the `cross-build/HOST` directory from somewhere else.
 
-The test suite can usually be run on a device with 2 GB of RAM, though for some
-configurations or test orders you may need to increase this. As of Android
+The test suite can usually be run on a device with 2 GB of RAM, but this is
+borderline, so you may need to increase it to 4 GB. As of Android
 Studio Koala, 2 GB is the default for all emulators, although the user interface
 may indicate otherwise. The effective setting is `hw.ramSize` in
 ~/.android/avd/*.avd/hardware-qemu.ini, whereas Android Studio displays the

Android/android.py

@@ -259,8 +259,8 @@ def setup_testbed():
              f"{temp_dir}/{outer_jar}", "gradle-wrapper.jar"])
 
 
-# run_testbed will build the app automatically, but it hides the Gradle output
-# by default, so it's useful to have this as a separate command for the buildbot.
+# run_testbed will build the app automatically, but it's useful to have this as
+# a separate command to allow running the app outside of this script.
 def build_testbed(context):
     setup_sdk()
     setup_testbed()
@@ -376,6 +376,8 @@ async def find_pid(serial):
     shown_error = False
     while True:
         try:
+            # `pidof` requires API level 24 or higher. The level 23 emulator
+            # includes it, but it doesn't work (it returns all processes).
             pid = (await async_check_output(
                 adb, "-s", serial, "shell", "pidof", "-s", APP_ID
             )).strip()
@@ -407,6 +409,7 @@ async def logcat_task(context, initial_devices):
     serial = await wait_for(find_device(context, initial_devices), startup_timeout)
     pid = await wait_for(find_pid(serial), startup_timeout)
 
+    # `--pid` requires API level 24 or higher.
     args = [adb, "-s", serial, "logcat", "--pid", pid,  "--format", "tag"]
     hidden_output = []
     async with async_process(
@@ -421,11 +424,15 @@ async def logcat_task(context, initial_devices):
                 # such messages, but other components might.
                 level, message = None, line
 
+            # Exclude high-volume messages which are rarely useful.
+            if context.verbose < 2 and "from python test_syslog" in message:
+                continue
+
             # Put high-level messages on stderr so they're highlighted in the
             # buildbot logs. This will include Python's own stderr.
             stream = (
                 sys.stderr
-                if level in ["E", "F"]  # ERROR and FATAL (aka ASSERT)
+                if level in ["W", "E", "F"]  # WARNING, ERROR, FATAL (aka ASSERT)
                 else sys.stdout
             )
 
@@ -573,8 +580,9 @@ def parse_args():
     test = subcommands.add_parser(
         "test", help="Run the test suite")
     test.add_argument(
-        "-v", "--verbose", action="store_true",
-        help="Show Gradle output, and non-Python logcat messages")
+        "-v", "--verbose", action="count", default=0,
+        help="Show Gradle output, and non-Python logcat messages. "
+        "Use twice to include high-volume messages which are rarely useful.")
     device_group = test.add_mutually_exclusive_group(required=True)
     device_group.add_argument(
         "--connected", metavar="SERIAL", help="Run on a connected device. "
@@ -591,6 +599,13 @@ def parse_args():
 
 def main():
     install_signal_handler()
+
+    # Under the buildbot, stdout is not a TTY, but we must still flush after
+    # every line to make sure our output appears in the correct order relative
+    # to the output of our subprocesses.
+    for stream in [sys.stdout, sys.stderr]:
+        stream.reconfigure(line_buffering=True)
+
     context = parse_args()
     dispatch = {"configure-build": configure_build_python,
                 "make-build": make_build_python,

Android/testbed/app/src/main/python/main.py

@@ -5,9 +5,25 @@
 import sys
 
 # Some tests use SIGUSR1, but that's blocked by default in an Android app in
-# order to make it available to `sigwait` in the "Signal Catcher" thread. That
-# thread's functionality is only relevant to the JVM ("forcing GC (no HPROF) and
-# profile save"), so disabling it should not weaken the tests.
+# order to make it available to `sigwait` in the Signal Catcher thread.
+# (https://cs.android.com/android/platform/superproject/+/android14-qpr3-release:art/runtime/signal_catcher.cc).
+# That thread's functionality is only useful for debugging the JVM, so disabling
+# it should not weaken the tests.
+#
+# There's no safe way of stopping the thread completely (#123982), but simply
+# unblocking SIGUSR1 is enough to fix most tests.
+#
+# However, in tests that generate multiple different signals in quick
+# succession, it's possible for SIGUSR1 to arrive while the main thread is busy
+# running the C-level handler for a different signal. In that case, the SIGUSR1
+# may be sent to the Signal Catcher thread instead, which will generate a log
+# message containing the text "reacting to signal".
+#
+# Such tests may need to be changed in one of the following ways:
+#   * Use a signal other than SIGUSR1 (e.g. test_stress_delivery_simultaneous in
+#     test_signal.py).
+#   * Send the signal to a specific thread rather than the whole process (e.g.
+#     test_signals in test_threadsignals.py.
 signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGUSR1])
 
 sys.argv[1:] = shlex.split(os.environ["PYTHON_ARGS"])

Doc/Makefile

@@ -222,16 +222,6 @@ dist:
 	cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-a4.tar.bz2
 	@echo "Build finished and archived!"
 
-	# archive the letter latex
-	@echo "Building LaTeX (US paper)..."
-	rm -rf build/latex
-	$(MAKE) latex PAPER=letter
-	-sed -i 's/: all-$$(FMT)/:/' build/latex/Makefile
-	(cd build/latex; $(MAKE) clean && $(MAKE) --jobs=$$((`nproc`+1)) --output-sync LATEXMKOPTS='-quiet' all-pdf && $(MAKE) FMT=pdf zip bz2)
-	cp build/latex/docs-pdf.zip dist/python-$(DISTVERSION)-docs-pdf-letter.zip
-	cp build/latex/docs-pdf.tar.bz2 dist/python-$(DISTVERSION)-docs-pdf-letter.tar.bz2
-	@echo "Build finished and archived!"
-
 	# copy the epub build
 	@echo "Building EPUB..."
 	rm -rf build/epub

Doc/c-api/type.rst

@@ -345,8 +345,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. c:function:: PyObject* PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
 
@@ -362,8 +366,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. c:function:: PyObject* PyType_FromSpec(PyType_Spec *spec)
 
@@ -378,8 +386,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. raw:: html
 

Doc/glossary.rst

@@ -36,19 +36,24 @@ Glossary
       and loaders (in the :mod:`importlib.abc` module).  You can create your own
       ABCs with the :mod:`abc` module.
 
+   annotate function
+      A function that can be called to retrieve the :term:`annotations <annotation>`
+      of an object. This function is accessible as the :attr:`~object.__annotate__`
+      attribute of functions, classes, and modules. Annotate functions are a
+      subset of :term:`evaluate functions <evaluate function>`.
+
    annotation
       A label associated with a variable, a class
       attribute or a function parameter or return value,
       used by convention as a :term:`type hint`.
 
       Annotations of local variables cannot be accessed at runtime, but
       annotations of global variables, class attributes, and functions
-      are stored in the :attr:`__annotations__`
-      special attribute of modules, classes, and functions,
-      respectively.
+      can be retrieved by calling :func:`annotationlib.get_annotations`
+      on modules, classes, and functions, respectively.
 
-      See :term:`variable annotation`, :term:`function annotation`, :pep:`484`
-      and :pep:`526`, which describe this functionality.
+      See :term:`variable annotation`, :term:`function annotation`, :pep:`484`,
+      :pep:`526`, and :pep:`649`, which describe this functionality.
       Also see :ref:`annotations-howto`
       for best practices on working with annotations.
 
@@ -366,6 +371,11 @@ Glossary
       statements.  The technique contrasts with the :term:`LBYL` style
       common to many other languages such as C.
 
+   evaluate function
+      A function that can be called to evaluate a lazily evaluated attribute
+      of an object, such as the value of type aliases created with the :keyword:`type`
+      statement.
+
    expression
       A piece of syntax which can be evaluated to some value.  In other words,
       an expression is an accumulation of expression elements like literals,

Doc/howto/annotations.rst

@@ -34,11 +34,16 @@ Accessing The Annotations Dict Of An Object In Python 3.10 And Newer
 
 Python 3.10 adds a new function to the standard library:
 :func:`inspect.get_annotations`.  In Python versions 3.10
-and newer, calling this function is the best practice for
+through 3.13, calling this function is the best practice for
 accessing the annotations dict of any object that supports
 annotations.  This function can also "un-stringize"
 stringized annotations for you.
 
+In Python 3.14, there is a new :mod:`annotationlib` module
+with functionality for working with annotations. This
+includes a :func:`annotationlib.get_annotations` function,
+which supersedes :func:`inspect.get_annotations`.
+
 If for some reason :func:`inspect.get_annotations` isn't
 viable for your use case, you may access the
 ``__annotations__`` data member manually.  Best practice
@@ -184,7 +189,11 @@ Best Practices For ``__annotations__`` In Any Python Version
 * If you do assign directly to the ``__annotations__`` member
   of an object, you should always set it to a ``dict`` object.
 
-* If you directly access the ``__annotations__`` member
+* You should avoid accessing ``__annotations__`` directly on any object.
+  Instead, use :func:`annotationlib.get_annotations` (Python 3.14+)
+  or :func:`inspect.get_annotations` (Python 3.10+).
+
+* If you do directly access the ``__annotations__`` member
   of an object, you should ensure that it's a
   dictionary before attempting to examine its contents.
 
@@ -231,3 +240,11 @@ itself be quoted.  In effect the annotation is quoted
 
 This prints ``{'a': "'str'"}``.  This shouldn't really be considered
 a "quirk"; it's mentioned here simply because it might be surprising.
+
+If you use a class with a custom metaclass and access ``__annotations__``
+on the class, you may observe unexpected behavior; see
+:pep:`749 <749#pep749-metaclasses>` for some examples. You can avoid these
+quirks by using :func:`annotationlib.get_annotations` on Python 3.14+ or
+:func:`inspect.get_annotations` on Python 3.10+. On earlier versions of
+Python, you can avoid these bugs by accessing the annotations from the
+class's ``__dict__`` (e.g., ``cls.__dict__.get('__annotations__', None)``).

Doc/howto/argparse.rst

@@ -444,7 +444,7 @@ And the output:
 
    options:
      -h, --help            show this help message and exit
-     -v {0,1,2}, --verbosity {0,1,2}
+     -v, --verbosity {0,1,2}
                            increase output verbosity
 
 Note that the change also reflects both in the error message as well as the

Doc/library/__future__.rst

@@ -64,8 +64,10 @@ language using this mechanism:
 | generator_stop   | 3.5.0b1     | 3.7          | :pep:`479`:                                 |
 |                  |             |              | *StopIteration handling inside generators*  |
 +------------------+-------------+--------------+---------------------------------------------+
-| annotations      | 3.7.0b1     | TBD [1]_     | :pep:`563`:                                 |
-|                  |             |              | *Postponed evaluation of annotations*       |
+| annotations      | 3.7.0b1     | Never [1]_   | :pep:`563`:                                 |
+|                  |             |              | *Postponed evaluation of annotations*,      |
+|                  |             |              | :pep:`649`: *Deferred evalutation of        |
+|                  |             |              | annotations using descriptors*              |
 +------------------+-------------+--------------+---------------------------------------------+
 
 .. XXX Adding a new entry?  Remember to update simple_stmts.rst, too.
@@ -115,11 +117,9 @@ language using this mechanism:
 
 .. [1]
    ``from __future__ import annotations`` was previously scheduled to
-   become mandatory in Python 3.10, but the Python Steering Council
-   twice decided to delay the change
-   (`announcement for Python 3.10 <https://mail.python.org/archives/list/[email protected]/message/CLVXXPQ2T2LQ5MP2Y53VVQFCXYWQJHKZ/>`__;
-   `announcement for Python 3.11 <https://mail.python.org/archives/list/[email protected]/message/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/>`__).
-   No final decision has been made yet. See also :pep:`563` and :pep:`649`.
+   become mandatory in Python 3.10, but the change was delayed and ultimately
+   canceled. This feature will eventually be deprecated and removed. See
+   :pep:`649` and :pep:`749`.
 
 
 .. seealso::