Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C API] PEP 741: Add PyInitConfig C API to customize the Python initialization #107954

Closed
vstinner opened this issue Aug 14, 2023 · 10 comments
Closed

[C API] PEP 741: Add PyInitConfig C API to customize the Python initialization #107954

vstinner opened this issue Aug 14, 2023 · 10 comments
Labels
topic-C-API type-feature A feature request or enhancement

Comments

@vstinner
Copy link
Member

vstinner commented Aug 14, 2023 •
edited by bedevere-app bot
Loading

Links to previous discussion of this feature:

https://discuss.python.org/t/fr-allow-private-runtime-config-to-enable-extending-without-breaking-the-pyconfig-abi/18004

Summary of proposal:

Add a text-based configuration to Python initialization

Pitch:

An API should be added to the limited C API customize the Python initialization (PyConfig API, PEP 587).

The problem is that the PyConfig C API is excluded from the limited C API, whereas the legacy initialization API is deprecated (ex: Py_VerboseFlag in Python 3.12) and being removed in Python 3.13 (ex: PySys_SetPath()).

Linked PRs
@vstinner vstinner added type-feature A feature request or enhancement topic-C-API labels Aug 14, 2023
@LostTime76
Copy link

LostTime76 commented Aug 15, 2023 •
edited
Loading

As discussed in discourse, I do not think the text file is the best option for embedding. Depending on the embedding scenario, I think we will run into issues of finding the file due to how python is currently resolving paths. I have a very specific scenario that may be solved in easier ways. Even if you by chance get python to find the file in my particular scenario, I don't want the user of my embedded python exe to have to specify package paths that my exe should be able to set itself.

Its not as easy as putting the file in the same directory as the embedded exe or the main script. Take the scenario that the embedding exe is using the global python install or the embedding exe is launched as a child process from an external process where the process path is not the path of the embedded exe.

I think there may be much easier ways to solve my specific scenario. I would like to use as few APIs as possible and not duplicate how python is already parsing the options from the command line.

What about adding additional command line options that can be set from argv? If we could set additional package paths from argv, it would allow us to directly point to where we need to without hassle.

Environment variables do not work nicely in the embedded scenario, even though we have PYTHONPATH, because there is no way to cleanly set an environment variable from an already running process to be accessible from a loaded python dll.

@vstinner
Copy link
Member Author

I'm thinking about a function taking a string, not a file.

@rokm rokm mentioned this issue Aug 16, 2023
Merged
@vstinner vstinner mentioned this issue Sep 4, 2023
Open
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Add _PyConfig_Parse()
fa98a1c
@bedevere-app bedevere-app bot mentioned this issue Sep 30, 2023
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Refactor initconfig.c, add CONFIG_SPEC
01fcc4c 
Add a specification of the PyConfig structure to factorize the code.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC
92d80d6 
Add a specification of the PyConfig structure to factorize the code.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC
c09b9a4 
Add a specification of the PyConfig structure to factorize the code.
@bedevere-app bedevere-app bot mentioned this issue Sep 30, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Add _PyConfig_Parse()
96cb693
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC
7191629 
Add a specification of the PyConfig structure to factorize the code.
vstinner added a commit that referenced this issue Sep 30, 2023
@vstinner
gh-107954: Refactor initconfig.c: add CONFIG_SPEC (#110146)
f3bb00e 
Add a specification of the PyConfig structure to factorize the code.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 30, 2023
@vstinner
pythongh-107954: Add _PyConfig_Parse()
e6e4e5c
vstinner added a commit to vstinner/cpython that referenced this issue Oct 1, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
7ea38bf 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetString(config, key, value)
* PyInitConfig_SetList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Changes:

* Add Include/initconfig.h header.
@bedevere-app bedevere-app bot mentioned this issue Oct 1, 2023
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Oct 1, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
4bee604 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Changes:

* Add Include/initconfig.h header.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 1, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
2b5a76c 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Add these 11 functions to the limited C API.

Changes:

* Add Include/initconfig.h header.
vstinner added a commit to vstinner/cpython that referenced this issue Oct 3, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
c10a404 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Add these 11 functions to the limited C API.

Changes:

* Add Include/initconfig.h header.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 30, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
624cbcb 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

* Add functions to get runtime config:

  * PyConfig_GetInt()
  * PyConfig_GetStr()
  * PyConfig_GetStrList()

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
* Add Modules/_testcapi/config.c
* Add Lib/test/test_capi/test_config.py.
vstinner added a commit to vstinner/cpython that referenced this issue Nov 30, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
e85efd1 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_GetErrorMsg(config)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

* Add functions to get runtime config:

  * PyConfig_GetInt()
  * PyConfig_GetStr()
  * PyConfig_GetStrList()

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
* Add Modules/_testcapi/config.c
* Add Lib/test/test_capi/test_config.py.
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
328fa6d 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_Exception(config)
* PyInitConfig_GetError(config, &err_msg)
* PyInitConfig_GetExitCode(config, &exitcode)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

* Add functions to get runtime config:

  * PyConfig_GetInt()
  * PyConfig_GetStr()
  * PyConfig_GetStrList()

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
* Add Modules/_testcapi/config.c
* Add Lib/test/test_capi/test_config.py.
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyInitConfig C API
47c117a 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_Exception(config)
* PyInitConfig_GetError(config, &err_msg)
* PyInitConfig_GetExitCode(config, &exitcode)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

* Add functions to get runtime config:

  * PyConfig_GetInt()
  * PyConfig_GetStr()
  * PyConfig_GetStrList()

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
* Add Modules/_testcapi/config.c
* Add Lib/test/test_capi/test_config.py.
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyConfig_Get() function
fba1c0f
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyConfig_Get() function
30af159
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyConfig_Get() function
3bc322f 
Add PyConfig_Get() and PyConfig_GetInt() functions to get the current
Python configuration.
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyConfig_Get() function
e8d143e 
Add PyConfig_Get() and PyConfig_GetInt() functions to get the current
Python configuration.

_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.
vstinner added a commit to vstinner/cpython that referenced this issue Dec 1, 2023
@vstinner
pythongh-107954: Add PyConfig_Get() function
c20baf0 
Add PyConfig_Get() and PyConfig_GetInt() functions to get the current
Python configuration.

_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.
vstinner added a commit to vstinner/cpython that referenced this issue Mar 14, 2024
@vstinner
pythongh-107954: Add PyInitConfig C API
65e49ac 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_Exception(config)
* PyInitConfig_GetError(config, &err_msg)
* PyInitConfig_GetExitCode(config, &exitcode)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024
@vstinner @adorilson
pythongh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec (pytho…
b41d07d 
…n#116359)

_PyConfig_AsDict() now returns bool objects for options using the new
PyConfig_MEMBER_BOOL type.

Update tests for these changes.
diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024
@vstinner @diegorusso
pythongh-107954: Add PyConfig_MEMBER_BOOL type to PyConfigSpec (pytho…
86e962a 
…n#116359)

_PyConfig_AsDict() now returns bool objects for options using the new
PyConfig_MEMBER_BOOL type.

Update tests for these changes.
vstinner added a commit to vstinner/cpython that referenced this issue May 23, 2024
@vstinner
pythongh-107954: Add PyInitConfig C API
2022d23 
Add PyInitConfig functions:

* PyInitConfig_Python_New()
* PyInitConfig_Isolated_New()
* PyInitConfig_Free(config)
* PyInitConfig_SetInt(config, key, value)
* PyInitConfig_SetStr(config, key, value)
* PyInitConfig_SetWStr(config, key, value)
* PyInitConfig_SetStrList(config, key, length, items)
* PyInitConfig_SetWStrList(config, key, length, items)
* PyInitConfig_Exception(config)
* PyInitConfig_GetError(config, &err_msg)
* PyInitConfig_GetExitCode(config, &exitcode)

Add also functions using it:

* Py_InitializeFromInitConfig(config)
* Py_ExitWithInitConfig(config)

Add these functions to the limited C API.

Changes:

* Add Doc/c-api/config.rst.
* Add Include/initconfig.h header.
vstinner added a commit to vstinner/cpython that referenced this issue Aug 29, 2024
@vstinner
pythongh-107954, PEP 741: Add PyConfig_Get() function
4c79a52 
Add PyConfig_Get(), PyConfig_GetInt(), PyConfig_Set() and
PyConfig_Names() functions to get and set the current runtime Python
configuration.

_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.
@bedevere-app bedevere-app bot mentioned this issue Aug 29, 2024
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2024
@vstinner
pythongh-107954, PEP 741: Add PyInitConfig C API
0eb3fd0 
Add Doc/c-api/config.rst documentation.
@vstinner vstinner changed the title [C API] PEP 741: Add stable ABI to customize Python initialization [C API] PEP 741: Add PyInitConfig C API to customize the Python initialization Aug 30, 2024
@bedevere-app bedevere-app bot mentioned this issue Aug 30, 2024
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2024
@vstinner
pythongh-107954, PEP 741: Add PyInitConfig C API
c745977 
Add Doc/c-api/config.rst documentation.
Glyphack pushed a commit to Glyphack/cpython that referenced this issue Sep 2, 2024
@vstinner @Glyphack
pythongh-107954: Refactor initconfig.c: add CONFIG_SPEC (python#110146)
5afbfdc 
Add a specification of the PyConfig structure to factorize the code.
vstinner added a commit that referenced this issue Sep 2, 2024
@vstinner @picnixz
gh-107954, PEP 741: Add PyConfig_Get()/Set() functions (#123472)
33b7909 
Add PyConfig_Get(), PyConfig_GetInt(), PyConfig_Set() and
PyConfig_Names() functions to get and set the current runtime Python
configuration.

Add visibility and "sys spec" to config and preconfig specifications.

_PyConfig_AsDict() now converts PyConfig.xoptions as a dictionary.

Co-authored-by: Bénédikt Tran <[email protected]>
vstinner added a commit that referenced this issue Sep 3, 2024
@vstinner
gh-107954, PEP 741: Add PyInitConfig C API (#123502)
ef9d547 
Add Doc/c-api/config.rst documentation.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2024
@vstinner
pythongh-107954, PEP 741: Adjust Python initialization config
b618040 
Setting dev_mode to 1 in an isolated configuration now enables also
faulthandler.

Moreover, setting "module_search_paths" option with
PyInitConfig_SetStrList() now sets "module_search_paths_set" to 1.
@bedevere-app bedevere-app bot mentioned this issue Sep 4, 2024
Merged
vstinner added a commit that referenced this issue Sep 4, 2024
@vstinner
gh-107954, PEP 741: Adjust Python initialization config (#123663)
b423ae6 
Setting dev_mode to 1 in an isolated configuration now enables also
faulthandler.

Moreover, setting "module_search_paths" option with
PyInitConfig_SetStrList() now sets "module_search_paths_set" to 1.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2024
@vstinner
pythongh-107954, PEP 741: Add PyInitConfig_AddModule() function
3a62386
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2024
@vstinner
pythongh-107954, PEP 741: Add PyInitConfig_AddModule() function
1ca74e2
vstinner added a commit to vstinner/cpython that referenced this issue Sep 4, 2024
@vstinner
pythongh-107954, PEP 741: Add PyInitConfig_AddModule() function
5241126
@bedevere-app bedevere-app bot mentioned this issue Sep 4, 2024
Merged
vstinner added a commit that referenced this issue Sep 4, 2024
@vstinner
gh-107954, PEP 741: Add PyInitConfig_AddModule() function (#123668)
0d6b6e3
@vstinner
Copy link
Member Author

vstinner commented Sep 6, 2024

PEP 741 is now fully implemented, I close the issue.

@vstinner vstinner closed this as completed Sep 6, 2024
pablogsal added a commit to pablogsal/cpython that referenced this issue Sep 26, 2024
@pablogsal
pythongh-107954: Fix configuration type for the perf profiler
3fb0be9
@bedevere-app bedevere-app bot mentioned this issue Sep 26, 2024
Merged
pablogsal added a commit to pablogsal/cpython that referenced this issue Sep 26, 2024
@pablogsal
pythongh-107954: Fix configuration type for the perf profiler
6092199
ambv pushed a commit that referenced this issue Sep 27, 2024
@pablogsal
gh-107954: Fix configuration type for the perf profiler (#124636)
02b49c5
vstinner added a commit to vstinner/cpython that referenced this issue Nov 20, 2024
@vstinner
pythongh-107954: Document PEP 741 in What's New 3.14
796b17b
@bedevere-app bedevere-app bot mentioned this issue Nov 20, 2024
Merged
vstinner added a commit that referenced this issue Nov 25, 2024
@vstinner
gh-107954: Document PEP 741 in What's New 3.14 (#127056)
d9331f1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic-C-API type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@milianw @vstinner @LostTime76