Fix bug in create_sysimg_from_object_file for Xcode 15 CLT #935

jayscook · 2024-04-02T18:30:29Z

Fix for issue #738, which conditionally adds the -Wl,-ld_classic flags to o_file_flags in create_sysimg_from_object_file based on Xcode CLT version.

All credit to @PhilReinhold for this code; I'm simply opening this PR based on his latest correspondence on the issue.

Tested locally on gcc 15.0.0 and Julia 1.8.5. @hbe72 confirmed in the issue correspondence that it's working on gcc 15.0.0 and Julia 1.10

Re: CI failures, I'm seeing other recent PRs (1, 2) that are also failing the nightly build tests and buildkite. Please let me know if you see anything in this change that should address these failures.

sjkelly · 2024-04-03T21:52:32Z

src/PackageCompiler.jl

+    if Sys.isapple()
+        cltools_version_cmd = `pkgutil --pkg-info=com.apple.pkg.CLTools_Executables`
+        cltools_version = match(r"version: (.*)\n", readchomp(cltools_version_cmd))[1]
+        if startswith(cltools_version, "15")


I am wondering if we should make this >= 15 ?
I presume this issue will still be around in xcode 16 and we don't want to go back to the flags for <=14?

I'm mildly torn on this. As-is, it forces us to revisit this fix for xcode 16 and investigate whether the issue has been solved or if our mildly hacky solution is still needed. Ultimately, though, that comes at the expense of (most likely) breaking development for anyone who upgrades to Xcode 16. I see no reason to assume that this will not continue to be an issue in xcode 16+, so making it ≥ 15 seems like the best course of action. Thanks!

topolarity · 2024-04-04T11:33:07Z

src/PackageCompiler.jl

-    o_file_flags = Sys.isapple() ? `-Wl,-all_load $object_files` : `-Wl,--whole-archive $object_files -Wl,--no-whole-archive`
+    if Sys.isapple()
+        cltools_version_cmd = `pkgutil --pkg-info=com.apple.pkg.CLTools_Executables`
+        cltools_version = match(r"version: (.*)\n", readchomp(cltools_version_cmd))[1]


This should handle the case that pkgutil is unable to locate com.apple.pkg.CLTools_Executables

Good call. Do you have any thoughts on what version we should assume under this failure case? I don't have a good feel for whether it's more likely that a system has been upgraded to the latest version or not, but given how Apple ties Xcode version to macOS version (and professional dev envs are eventually going to force OS updates) it may be safest to default to 15+ (i.e. to apply the flags).

Unfortunately it looks like there's a syntax ambiguity so that a lot of linkers interpret this flag as -ld_classic i.e. link against libd_classic.so, which will obviously fail.

I'm not sure how far back we have to go before that becomes an issue. It also sounds like we might have to worry about using a different compiler not from XCode CLT.

Could we execute the compiler binary and ask it for its version directly?

While I have a general understanding of the problem you're describing, this is admittedly over my head. Maybe something like the first answer on this SO post could work?

My biggest concern is that this patch assumes that the user is using the XCode CLT to compile on macOS, which is not necessarily true

By "execute the compiler binary directly", I mean a combination of either:

Modifying get_compiler_cmd to inspect the path to the binary and see that it's provided by Xcode, and/or

Execute run_compiler(`` --version``) or similar to determine whether we're running the Xcode-provided compiler and which version.

I've tinkered with this a bit and am still having a bit of trouble.

I've tried getting the path to the compiler with

base_dir = dirname(sysimg_source_path) compiler_source_path = joinpath(base_dir, "compiler", "compiler.jl")

Printing compiler_source_path results in /Users/jaycook/.julia/juliaup/julia-1.8.5+0.aarch64.apple.darwin14/share/julia/base/compiler/compiler.jl. I was hoping more for something like /Library/Developer/CommandLineTools/usr/bin, where I could check for a string like "xcode" or "CommandLineTools".

run_compiler()'s underlying call to run() does not appear to return the result of the command that's run. For example, print(run_compiler(`` --version``)) results in Process(``gcc --version``, ProcessExited(0)).

If you're so inclined, do you care to toss up a commit for the implementation that you have in mind?

sjkelly · 2024-04-04T19:46:31Z

src/PackageCompiler.jl

-        if startswith(cltools_version, "15")
+        try
+            cltools_version_cmd = `pkgutil --pkg-info=com.apple.pkg.CLTools_Executables`
+            cltools_version = match(r"version: (.*)\n", readchomp(cltools_version_cmd))[1]


I think you can check for nothing here, rather than try/catch.

Can you suggest a good way to test this? If I change the --pkg-info to force a failure I get the following error, but I imagine that the "real" failure mode here is simply returning nothing. Just not sure how to go about getting my system into a state that allows me to test this before shipping it.

ufechner7 · 2024-08-01T11:09:19Z

Can this be merged?

jayscook · 2024-08-01T16:15:21Z

Can this be merged?

This comment feels like one that should be addressed, but I don't have the know-how to devise a solution. Hopefully @topolarity can suggest a solution.

The other option would be to merge as-is as a step forward and address the above issue when it becomes an issue for someone.

ufechner7 · 2024-08-01T16:40:56Z

Well, the macOS tests still fail. But with a different (?) error:

 ERROR: LoadError: Cannot locate artifact 'fooifier' for aarch64-apple-darwin-libgfortran5-cxx11-julia_version+1.10.4 in '/private/var/folders/zn/hj183dg15s713b47j2wlhwzw0000gn/T/jl_7YRmdg/MyApp/Artifacts.toml'

I mean, this is easy to fix:

Unfortunately, Julia 1.6 and 1.7 do not have native binaries available for Apple Silicon macOS.
Therefore, it is not possible to install Julia with the current constraints.
For instructions on how to fix this error, please see the following Discourse post: https://discourse.julialang.org/t/how-to-fix-github-actions-ci-failures-with-julia-1-6-or-1-7-on-macos-latest-and-macos-14/117019

But it also fails on Julia 1.10.

codecov · 2024-10-16T04:01:53Z

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Project coverage is 84.51%. Comparing base (85cdb03) to head (8c4fda1).

Files with missing lines	Patch %	Lines
src/PackageCompiler.jl	81.81%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #935      +/-   ##
==========================================
- Coverage   84.56%   84.51%   -0.06%     
==========================================
  Files           3        3              
  Lines         823      833      +10     
==========================================
+ Hits          696      704       +8     
- Misses        127      129       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DilumAluthge · 2024-10-20T00:44:28Z

I'm going to close this PR in favor of #972, because #972 includes some additional changes that are intended to address review comments (specifically, #972 attempts to only add the additional linker flags if we confirm that the active compiler is the Apple Clang from either Xcode.app or the Xcode CLT).

jayscook · 2024-10-21T15:02:28Z

I'm going to close this PR in favor of #972, because #972 includes some additional changes that are intended to address review comments (specifically, #972 attempts to only add the additional linker flags if we confirm that the active compiler is the Apple Clang from either Xcode.app or the Xcode CLT).

Thanks! Marked this as ready just as 972 was going up and forgot to come back and close.

jayscook added 2 commits April 2, 2024 12:36

Fix bug in create_sysimg_from_object_file for Xcode 15 CLT

6d2cdd1

Bump version

1d955aa

jayscook mentioned this pull request Apr 3, 2024

macOS: System image file fails consistency check on binary startup #738

Closed

sjkelly reviewed Apr 3, 2024

View reviewed changes

topolarity reviewed Apr 4, 2024

View reviewed changes

Check for versions ≥ 15, handle pkgutil error

7161e1a

sjkelly reviewed Apr 4, 2024

View reviewed changes

sjkelly mentioned this pull request Jun 25, 2024

use Base.julia_exename(), fixes #664 #947

Merged

ufechner7 mentioned this pull request Aug 1, 2024

Fix CI failures on macOS #955

Closed

Merge branch 'master' into xcode-flag-fix

3d8f36b

DilumAluthge self-requested a review October 15, 2024 21:56

DilumAluthge mentioned this pull request Oct 16, 2024

macOS: If using Xcode CLT >= 15, add -Wl,-ld_classic in create_sysimg_from_object_file() #972

Draft

DilumAluthge removed their request for review October 16, 2024 03:26

DilumAluthge marked this pull request as draft October 16, 2024 03:27

Merge branch 'master' into xcode-flag-fix

e17d3a8

jayscook marked this pull request as ready for review October 16, 2024 15:17

Merge branch 'master' into xcode-flag-fix

8c4fda1

DilumAluthge closed this Oct 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bug in create_sysimg_from_object_file for Xcode 15 CLT #935

Fix bug in create_sysimg_from_object_file for Xcode 15 CLT #935

jayscook commented Apr 2, 2024 •

edited

Loading

sjkelly Apr 3, 2024

jayscook Apr 4, 2024

topolarity Apr 4, 2024

jayscook Apr 4, 2024

topolarity Apr 4, 2024 •

edited

Loading

jayscook Apr 5, 2024

topolarity Apr 5, 2024 •

edited

Loading

jayscook Apr 8, 2024

sjkelly Apr 4, 2024

jayscook Apr 5, 2024

ufechner7 commented Aug 1, 2024

jayscook commented Aug 1, 2024

ufechner7 commented Aug 1, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

DilumAluthge commented Oct 20, 2024

jayscook commented Oct 21, 2024

Fix bug in create_sysimg_from_object_file for Xcode 15 CLT #935

Fix bug in create_sysimg_from_object_file for Xcode 15 CLT #935

Conversation

jayscook commented Apr 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topolarity Apr 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

topolarity Apr 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ufechner7 commented Aug 1, 2024

jayscook commented Aug 1, 2024

ufechner7 commented Aug 1, 2024 • edited Loading

codecov bot commented Oct 16, 2024 • edited Loading

Codecov Report

DilumAluthge commented Oct 20, 2024

jayscook commented Oct 21, 2024

jayscook commented Apr 2, 2024 •

edited

Loading

topolarity Apr 4, 2024 •

edited

Loading

topolarity Apr 5, 2024 •

edited

Loading

ufechner7 commented Aug 1, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading