GH-40236: [Python][CI] Disable generating C lines in Cython tracebacks #40225
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@github-actions crossbow submit wheel-windows* |
This comment was marked as outdated.
This comment was marked as outdated.
|
Surprisingly, this fixes (at least for the time being) the PyArrow compilation time issues. I'll make a proper issue for this and mark the PR ready for review. @raulcd @jorisvandenbossche |
pitrou
changed the title
|
|
pitrou
force-pushed
the
cython-c-lines-in-traceback
branch
from
1257044
to
d27ff9a
Compare
|
@github-actions crossbow submit -g wheel -g python |
|
Revision: d27ff9a Submitted crossbow builds: ursacomputing/crossbow @ actions-e32362e5a6
|
This was referenced Feb 26, 2024
|
Here's the Cython issue[1] for reference. It looks like we are just removing string data (line number + function name) in traces. The issue states this is good to have in production. Looks like we are just adding too much in once file (hence GH-40166). |
danepitkin
approved these changes
Feb 26, 2024
github-actions
bot
added
awaiting committer review
|
Thanks for looking at this @danepitkin . I'll merge this in the interest of improving PR feedback. |
|
Nice! Do you actually understand what we "lost" with this change? |
|
Like you I never saw any C line numbers in Cython-generated tracebacks. It seems that an additional setting is involved, which is off by default, so Cython was really generating unused information. |
zanmato1984
pushed a commit
to zanmato1984/arrow
that referenced
this pull request
Feb 28, 2024
…cebacks (apache#40225) ### Rationale for this change We're getting timeouts (on AppVeyor) and very long compilation times (on GHA wheel builds) for `lib.cpp`, a Cython-generated C++ file. Examination suggests that `lib.cpp` is more than 300 thousand lines long, and we can hypothesize that this can blow up available memory on some machines and compilers. ### What changes are included in this PR? Disable a not really useful (and undocumented) Cython feature to make C++ code slightly easier to compile. ### Are these changes tested? Yes. This solves, at least temporarily, the timeout issues on AppVeyor and makes the wheel builds much faster (down to ~35 minutes for the wheel build step, instead of 3 hours). ### Are there any user-facing changes? No. * GitHub Issue: apache#40236 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>
thisisnic
pushed a commit
to thisisnic/arrow
that referenced
this pull request
Mar 8, 2024
…cebacks (apache#40225) ### Rationale for this change We're getting timeouts (on AppVeyor) and very long compilation times (on GHA wheel builds) for `lib.cpp`, a Cython-generated C++ file. Examination suggests that `lib.cpp` is more than 300 thousand lines long, and we can hypothesize that this can blow up available memory on some machines and compilers. ### What changes are included in this PR? Disable a not really useful (and undocumented) Cython feature to make C++ code slightly easier to compile. ### Are these changes tested? Yes. This solves, at least temporarily, the timeout issues on AppVeyor and makes the wheel builds much faster (down to ~35 minutes for the wheel build step, instead of 3 hours). ### Are there any user-facing changes? No. * GitHub Issue: apache#40236 Authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Antoine Pitrou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rationale for this change
We're getting timeouts (on AppVeyor) and very long compilation times (on GHA wheel builds) for
lib.cpp, a Cython-generated C++ file. Examination suggests thatlib.cppis more than 300 thousand lines long, and we can hypothesize that this can blow up available memory on some machines and compilers.What changes are included in this PR?
Disable a not really useful (and undocumented) Cython feature to make C++ code slightly easier to compile.
Are these changes tested?
Yes. This solves, at least temporarily, the timeout issues on AppVeyor and makes the wheel builds much faster (down to ~35 minutes for the wheel build step, instead of 3 hours).
Are there any user-facing changes?
No.