-
-
Notifications
You must be signed in to change notification settings - Fork 30.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhance the BOLT build process #101282
Labels
Comments
cc @aaupov |
corona10
added a commit
to corona10/cpython
that referenced
this issue
Feb 4, 2023
corona10
pushed a commit
that referenced
this issue
May 15, 2023
carljm
added a commit
to carljm/cpython
that referenced
this issue
May 15, 2023
* main: (29 commits) pythongh-101819: Fix _io clinic input for unused base class method stubs (python#104418) pythongh-101819: Isolate `_io` (python#101948) Bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic (python#104501) pythongh-104494: Update certain Tkinter pack/place tests for Tk 8.7 errors (python#104495) pythongh-104050: Run mypy on `clinic.py` in CI (python#104421) pythongh-104490: Consistently define phony make targets (python#104491) pythongh-67056: document that registering/unregistering an atexit func from within an atexit func is undefined (python#104473) pythongh-104487: PYTHON_FOR_REGEN must be minimum Python 3.10 (python#104488) pythongh-101282: move BOLT config after PGO (pythongh-104493) pythongh-104469 Convert _testcapi/float.c to use AC (pythongh-104470) pythongh-104456: Fix ref leak in _ctypes.COMError (python#104457) pythongh-98539: Make _SSLTransportProtocol.abort() safe to call when closed (python#104474) pythongh-104337: Clarify random.gammavariate doc entry (python#104410) Minor improvements to typing docs (python#104465) pythongh-87092: avoid gcc warning on uninitialized struct field in assemble.c (python#104460) pythonGH-71383: IDLE - Document testing subsets of modules (python#104463) pythongh-104454: Fix refleak in AttributeError_reduce (python#104455) pythongh-75710: IDLE - add docstrings and comments to editor module (python#104446) pythongh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (python#104424) Add a mention of PYTHONBREAKPOINT to breakpoint() docs (python#104430) ...
carljm
added a commit
to carljm/cpython
that referenced
this issue
May 15, 2023
* main: (204 commits) pythongh-101819: Fix _io clinic input for unused base class method stubs (python#104418) pythongh-101819: Isolate `_io` (python#101948) Bump mypy from 1.2.0 to 1.3.0 in /Tools/clinic (python#104501) pythongh-104494: Update certain Tkinter pack/place tests for Tk 8.7 errors (python#104495) pythongh-104050: Run mypy on `clinic.py` in CI (python#104421) pythongh-104490: Consistently define phony make targets (python#104491) pythongh-67056: document that registering/unregistering an atexit func from within an atexit func is undefined (python#104473) pythongh-104487: PYTHON_FOR_REGEN must be minimum Python 3.10 (python#104488) pythongh-101282: move BOLT config after PGO (pythongh-104493) pythongh-104469 Convert _testcapi/float.c to use AC (pythongh-104470) pythongh-104456: Fix ref leak in _ctypes.COMError (python#104457) pythongh-98539: Make _SSLTransportProtocol.abort() safe to call when closed (python#104474) pythongh-104337: Clarify random.gammavariate doc entry (python#104410) Minor improvements to typing docs (python#104465) pythongh-87092: avoid gcc warning on uninitialized struct field in assemble.c (python#104460) pythonGH-71383: IDLE - Document testing subsets of modules (python#104463) pythongh-104454: Fix refleak in AttributeError_reduce (python#104455) pythongh-75710: IDLE - add docstrings and comments to editor module (python#104446) pythongh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (python#104424) Add a mention of PYTHONBREAKPOINT to breakpoint() docs (python#104430) ...
indygreg
added a commit
to indygreg/cpython
that referenced
this issue
May 20, 2023
(This change is a quick and dirty way to merge some of the build system improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze. I wanted to scope bloat myself to fix some longstanding deficiencies in the build system around profile-guided builds. But I'm getting soft resistance to the reviews so close to the freeze deadline and it is obvious that we need a simpler solution to hit the 3.12 deadline. While this change is quick and dirty, it attempts to not make things worse.) Before this change, we only applied bolt to the main python binary. After this change, we apply bolt to libpython if it is configured. In shared library builds, most of the C code is in libpython so it is critical to apply bolt to libpython to realize bolt benefits. This change also reworks how bolt instrumentation is applied. It effectively removes the readelf based logic added in pythongh-101525 and replaces it with a mechanism that saves a copy of the pre-bolt binary and restores that copy when necessary. This allows us to perform bolt optimizations without having to manually delete the output binary to force a new bolt run. We also add a new make target for purging bolt files and hook it up to `clean` so bolt state is purged when appropriate. `.gitignore` rules have been added to ignore files related to bolt. Before and after this refactor, `make` will no-op after a previous run. Both versions should also share common make DAG deficiencies where targets fail to trigger as often as they need to or can trigger prematurely in certain scenarios. e.g. after this change you may need to `rm profile-bolt-stamp` to force a bolt run because there aren't appropriate non-phony targets for bolt's make target to depend on. Fixing this is a non-trivial amount of work that will likely have to wait until the 3.13 window. To make it easier to iterate on custom BOLT settings, the flags to pass to instrumentation and application are now defined in configure and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and `BOLT_APPLY_FLAGS`.
Thanks for the work, let's ship the feature at CPython 3.12 as the experimental feature. |
indygreg
added a commit
to indygreg/cpython
that referenced
this issue
May 21, 2023
(This change is a quick and dirty way to merge some of the build system improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze. I wanted to scope bloat myself to fix some longstanding deficiencies in the build system around profile-guided builds. But I'm getting soft resistance to the reviews so close to the freeze deadline and it is obvious that we need a simpler solution to hit the 3.12 deadline. While this change is quick and dirty, it attempts to not make things worse.) Before this change, we only applied bolt to the main python binary. After this change, we apply bolt to libpython if it is configured. In shared library builds, most of the C code is in libpython so it is critical to apply bolt to libpython to realize bolt benefits. This change also reworks how bolt instrumentation is applied. It effectively removes the readelf based logic added in pythongh-101525 and replaces it with a mechanism that saves a copy of the pre-bolt binary and restores that copy when necessary. This allows us to perform bolt optimizations without having to manually delete the output binary to force a new bolt run. We also add a new make target for purging bolt files and hook it up to `clean` so bolt state is purged when appropriate. `.gitignore` rules have been added to ignore files related to bolt. Before and after this refactor, `make` will no-op after a previous run. Both versions should also share common make DAG deficiencies where targets fail to trigger as often as they need to or can trigger prematurely in certain scenarios. e.g. after this change you may need to `rm profile-bolt-stamp` to force a bolt run because there aren't appropriate non-phony targets for bolt's make target to depend on. Fixing this is a non-trivial amount of work that will likely have to wait until the 3.13 window. To make it easier to iterate on custom BOLT settings, the flags to pass to instrumentation and application are now defined in configure and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and `BOLT_APPLY_FLAGS`.
erlend-aasland
pushed a commit
that referenced
this issue
May 22, 2023
…104709) Apply BOLT optimizations to libpython for shared builds. Most of the C code is in libpython so it is critical to apply BOLT there fully realize BOLT benefits. This change also reworks how BOLT instrumentation is applied. It effectively removes the readelf based logic added in gh-101525 and replaces it with a mechanism that saves a copy of the pre-bolt binary and restores that copy when necessary. This allows us to perform BOLT optimizations without having to manually delete the output binary to force a new bolt run. Also: - add a clean-bolt target for purging BOLT files and hook that up to the clean target - .gitignore BOLT related files Before and after this refactor, `make` will no-op after a previous run. Both versions should also share common make DAG deficiencies where targets fail to trigger as often as they need to or can trigger prematurely in certain scenarios. e.g. after this change you may need to `rm profile-bolt-stamp` to force a BOLT run because there aren't appropriate non-phony targets for BOLT's make target to depend on. To make it easier to iterate on custom BOLT settings, the flags to pass to instrumentation and application are now defined in configure and can be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
@indygreg, looks like this part can be marked completed. |
corona10
added a commit
to corona10/cpython
that referenced
this issue
May 22, 2023
This was referenced May 22, 2023
corona10
added a commit
to corona10/cpython
that referenced
this issue
May 22, 2023
erlend-aasland
added a commit
to erlend-aasland/cpython
that referenced
this issue
May 23, 2023
Release blocker for CPython 3.12 beta2 candidate PRs |
corona10
added a commit
that referenced
this issue
May 24, 2023
corona10
added a commit
to corona10/cpython
that referenced
this issue
May 24, 2023
…es (pythongh-104752) (cherry picked from commit c437851) Co-authored-by: Dong-hee Na <[email protected]>
corona10
added a commit
that referenced
this issue
May 24, 2023
github-project-automation
bot
moved this from Todo
to Done
in Release and Deferred blockers 🚫
May 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From @indygreg 's #101093
TODO
-fno-reorder-blocks-and-partition
-gdwarf-4
Linked PRs
The text was updated successfully, but these errors were encountered: