Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bpo-42260: PyConfig_Read() only parses argv once #23168

Merged
merged 2 commits into from
Nov 5, 2020
Merged

bpo-42260: PyConfig_Read() only parses argv once #23168

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
12fe7d1
bpo-42260: PyConfig_Read() only parses argv once
vstinner Nov 5, 2020
3903273
Add test on argv=[]
vstinner Nov 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 104 additions & 92 deletions Doc/c-api/init_config.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,55 +8,68 @@ Python Initialization Configuration

.. versionadded:: 3.8

Structures:

* :c:type:`PyConfig`
* :c:type:`PyPreConfig`
* :c:type:`PyStatus`
* :c:type:`PyWideStringList`

Functions:

* :c:func:`PyConfig_Clear`
* :c:func:`PyConfig_InitIsolatedConfig`
* :c:func:`PyConfig_InitPythonConfig`
* :c:func:`PyConfig_Read`
* :c:func:`PyConfig_SetArgv`
* :c:func:`PyConfig_SetBytesArgv`
* :c:func:`PyConfig_SetBytesString`
* :c:func:`PyConfig_SetString`
* :c:func:`PyConfig_SetWideStringList`
* :c:func:`PyPreConfig_InitIsolatedConfig`
* :c:func:`PyPreConfig_InitPythonConfig`
* :c:func:`PyStatus_Error`
* :c:func:`PyStatus_Exception`
* :c:func:`PyStatus_Exit`
* :c:func:`PyStatus_IsError`
* :c:func:`PyStatus_IsExit`
* :c:func:`PyStatus_NoMemory`
* :c:func:`PyStatus_Ok`
* :c:func:`PyWideStringList_Append`
* :c:func:`PyWideStringList_Insert`
* :c:func:`Py_ExitStatusException`
* :c:func:`Py_InitializeFromConfig`
* :c:func:`Py_PreInitialize`
* :c:func:`Py_PreInitializeFromArgs`
* :c:func:`Py_PreInitializeFromBytesArgs`
* :c:func:`Py_RunMain`
* :c:func:`Py_GetArgcArgv`

The preconfiguration (``PyPreConfig`` type) is stored in
``_PyRuntime.preconfig`` and the configuration (``PyConfig`` type) is stored in
``PyInterpreterState.config``.
Python can be initialized with :c:func:`Py_InitializeFromConfig` and the
:c:type:`PyConfig` structure. It can be preinitialized with
:c:func:`Py_PreInitialize` and the :c:type:`PyPreConfig` structure.

There are two kinds of configuration:

* The :ref:`Python Configuration <init-python-config>` can be used to build a
customized Python which behaves as the regular Python. For example,
environments variables and command line arguments are used to configure
Python.

* The :ref:`Isolated Configuration <init-isolated-conf>` can be used to embed
Python into an application. It isolates Python from the system. For example,
environments variables are ignored, the LC_CTYPE locale is left unchanged and
no signal handler is registred.

See also :ref:`Initialization, Finalization, and Threads <initialization>`.

.. seealso::
:pep:`587` "Python Initialization Configuration".

Example
=======

Example of customized Python always running in isolated mode::

int main(int argc, char **argv)
{
PyStatus status;

PyConfig config;
PyConfig_InitPythonConfig(&config);
config.isolated = 1;

/* Decode command line arguments.
Implicitly preinitialize Python (in isolated mode). */
status = PyConfig_SetBytesArgv(&config, argc, argv);
if (PyStatus_Exception(status)) {
goto exception;
}

status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto exception;
}
PyConfig_Clear(&config);

return Py_RunMain();

exception:
PyConfig_Clear(&config);
if (PyStatus_IsExit(status)) {
return status.exitcode;
}
/* Display the error message and exit the process with
non-zero exit code */
Py_ExitStatusException(status);
}


PyWideStringList
----------------
================

.. c:type:: PyWideStringList

Expand Down Expand Up @@ -95,7 +108,7 @@ PyWideStringList
List items.

PyStatus
--------
========

.. c:type:: PyStatus

Expand Down Expand Up @@ -187,7 +200,7 @@ Example::


PyPreConfig
-----------
===========

.. c:type:: PyPreConfig

Expand Down Expand Up @@ -317,7 +330,7 @@ PyPreConfig
.. _c-preinit:

Preinitialize Python with PyPreConfig
-------------------------------------
=====================================

The preinitialization of Python:

Expand All @@ -326,26 +339,35 @@ The preinitialization of Python:
* Set the :ref:`Python UTF-8 Mode <utf8-mode>`
(:c:member:`PyPreConfig.utf8_mode`)

The current preconfiguration (``PyPreConfig`` type) is stored in
``_PyRuntime.preconfig``.

Functions to preinitialize Python:

.. c:function:: PyStatus Py_PreInitialize(const PyPreConfig *preconfig)

Preinitialize Python from *preconfig* preconfiguration.

*preconfig* must not be ``NULL``.

.. c:function:: PyStatus Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv)

Preinitialize Python from *preconfig* preconfiguration.

Parse *argv* command line arguments (bytes strings) if
:c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.

*preconfig* must not be ``NULL``.

.. c:function:: PyStatus Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv)

Preinitialize Python from *preconfig* preconfiguration.

Parse *argv* command line arguments (wide strings) if
:c:member:`~PyPreConfig.parse_argv` of *preconfig* is non-zero.

*preconfig* must not be ``NULL``.

The caller is responsible to handle exceptions (error or exit) using
:c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.

Expand Down Expand Up @@ -388,7 +410,7 @@ the :ref:`Python UTF-8 Mode <utf8-mode>`::


PyConfig
--------
========

.. c:type:: PyConfig

Expand Down Expand Up @@ -449,8 +471,20 @@ PyConfig

Fields which are already initialized are left unchanged.

The :c:func:`PyConfig_Read` function only parses
:c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv`
is set to ``2`` after arguments are parsed. Since Python arguments are
strippped from :c:member:`PyConfig.argv`, parsing arguments twice would
parse the application options as Python options.

:ref:`Preinitialize Python <c-preinit>` if needed.

.. versionchanged:: 3.10
The :c:member:`PyConfig.argv` arguments are now only parsed once,
:c:member:`PyConfig.parse_argv` is set to ``2`` after arguments are
parsed, and arguments are only parsed if
:c:member:`PyConfig.parse_argv` equals ``1``.

.. c:function:: void PyConfig_Clear(PyConfig *config)

Release configuration memory.
Expand Down Expand Up @@ -833,7 +867,7 @@ PyConfig

If :c:member:`~PyConfig.orig_argv` list is empty and
:c:member:`~PyConfig.argv` is not a list only containing an empty
string, :c:func:`PyConfig_Read()` copies :c:member:`~PyConfig.argv` into
string, :c:func:`PyConfig_Read` copies :c:member:`~PyConfig.argv` into
:c:member:`~PyConfig.orig_argv` before modifying
:c:member:`~PyConfig.argv` (if :c:member:`~PyConfig.parse_argv` is
non-zero).
Expand All @@ -849,12 +883,22 @@ PyConfig

Parse command line arguments?

If non-zero, parse :c:member:`~PyConfig.argv` the same way the regular
If equals to ``1``, parse :c:member:`~PyConfig.argv` the same way the regular
Python parses :ref:`command line arguments <using-on-cmdline>`, and strip
Python arguments from :c:member:`~PyConfig.argv`.

The :c:func:`PyConfig_Read` function only parses
:c:member:`PyConfig.argv` arguments once: :c:member:`PyConfig.parse_argv`
is set to ``2`` after arguments are parsed. Since Python arguments are
strippped from :c:member:`PyConfig.argv`, parsing arguments twice would
parse the application options as Python options.

Default: ``1`` in Python mode, ``0`` in isolated mode.

.. versionchanged:: 3.10
The :c:member:`PyConfig.argv` arguments are now only parsed if
:c:member:`PyConfig.parse_argv` equals to ``1``.

.. c:member:: int parser_debug

Parser debug mode. If greater than 0, turn on parser debugging output (for expert only, depending
Expand Down Expand Up @@ -1108,7 +1152,7 @@ the :option:`-X` command line option.


Initialization with PyConfig
----------------------------
============================

Function to initialize Python:

Expand All @@ -1123,6 +1167,9 @@ If :c:func:`PyImport_FrozenModules`, :c:func:`PyImport_AppendInittab` or
:c:func:`PyImport_ExtendInittab` are used, they must be set or called after
Python preinitialization and before the Python initialization.

The current configuration (``PyConfig`` type) is stored in
``PyInterpreterState.config``.

Example setting the program name::

void init_python(void)
Expand All @@ -1136,17 +1183,17 @@ Example setting the program name::
status = PyConfig_SetString(&config, &config.program_name,
L"/path/to/my_program");
if (PyStatus_Exception(status)) {
goto fail;
goto exception;
}

status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto fail;
goto exception;
}
PyConfig_Clear(&config);
return;

fail:
exception:
PyConfig_Clear(&config);
Py_ExitStatusException(status);
}
Expand Down Expand Up @@ -1202,7 +1249,7 @@ configuration, and then override some parameters::
.. _init-isolated-conf:

Isolated Configuration
----------------------
======================

:c:func:`PyPreConfig_InitIsolatedConfig` and
:c:func:`PyConfig_InitIsolatedConfig` functions create a configuration to
Expand All @@ -1223,7 +1270,7 @@ configuration.
.. _init-python-config:

Python Configuration
--------------------
====================

:c:func:`PyPreConfig_InitPythonConfig` and :c:func:`PyConfig_InitPythonConfig`
functions create a configuration to build a customized Python which behaves as
Expand All @@ -1237,46 +1284,11 @@ and :ref:`Python UTF-8 Mode <utf8-mode>`
(:pep:`540`) depending on the LC_CTYPE locale, :envvar:`PYTHONUTF8` and
:envvar:`PYTHONCOERCECLOCALE` environment variables.

Example of customized Python always running in isolated mode::

int main(int argc, char **argv)
{
PyStatus status;

PyConfig config;
PyConfig_InitPythonConfig(&config);
config.isolated = 1;

/* Decode command line arguments.
Implicitly preinitialize Python (in isolated mode). */
status = PyConfig_SetBytesArgv(&config, argc, argv);
if (PyStatus_Exception(status)) {
goto fail;
}

status = Py_InitializeFromConfig(&config);
if (PyStatus_Exception(status)) {
goto fail;
}
PyConfig_Clear(&config);

return Py_RunMain();

fail:
PyConfig_Clear(&config);
if (PyStatus_IsExit(status)) {
return status.exitcode;
}
/* Display the error message and exit the process with
non-zero exit code */
Py_ExitStatusException(status);
}


.. _init-path-config:

Path Configuration
------------------
==================

:c:type:`PyConfig` contains multiple fields for the path configuration:

Expand Down Expand Up @@ -1356,7 +1368,7 @@ The ``__PYVENV_LAUNCHER__`` environment variable is used to set


Py_RunMain()
------------
============

.. c:function:: int Py_RunMain(void)

Expand All @@ -1376,7 +1388,7 @@ customized Python always running in isolated mode using


Py_GetArgcArgv()
----------------
================

.. c:function:: void Py_GetArgcArgv(int *argc, wchar_t ***argv)

Expand All @@ -1386,7 +1398,7 @@ Py_GetArgcArgv()


Multi-Phase Initialization Private Provisional API
--------------------------------------------------
==================================================

This section is a private provisional API introducing multi-phase
initialization, the core feature of the :pep:`432`:
Expand Down
8 changes: 7 additions & 1 deletion Lib/test/_test_embed_set_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ def setUp(self):
self.sys_copy = dict(sys.__dict__)

def tearDown(self):
self.set_config(parse_argv=0)
_testinternalcapi.set_config(self.old_config)
sys.__dict__.clear()
sys.__dict__.update(self.sys_copy)

Expand Down Expand Up @@ -234,6 +234,12 @@ def test_argv(self):
self.assertEqual(sys.argv, ['python_program', 'args'])
self.assertEqual(sys.orig_argv, ['orig', 'orig_args'])

self.set_config(parse_argv=0,
argv=[],
orig_argv=[])
self.assertEqual(sys.argv, [''])
self.assertEqual(sys.orig_argv, [])

def test_pycache_prefix(self):
self.check(pycache_prefix=None)
self.check(pycache_prefix="pycache_prefix")
Expand Down
Loading