x86_64-darwin: fix packages that won’t build #180931

reckenrode · 2022-07-10T04:54:14Z

Description of changes

#176661 allows x86_64-darwin to use the 11.0 SDK. This PR fixes most packages that were flagged as broken on x86_64darwin. It also reworks the MoltenVK derivation to build without needing Xcode (and bumps it because that patch has been sitting in my queue along with the Xcode build change). There are two exceptions:

Go packages. These are being done in another PR (either go_1_18: Use apple_sdk_11_0.callPackage #180704 or as part of go,goBuildModules: bump default go version to 1.18 #179622).
Qt. The Qt derivation is pretty big, so I left it alone.
SDL2. This causes a lot of rebuilds. I’m going to submit the fix upstream and reference the patch in a separate PR against staging. Depending on the result of nixpkgs-review, I will probably include fixes to broken packages that depend on SDL2.
MoltenVK. Moved to moltenvk: refactor, update, and fix breakage #180988. That PR will fix broken packages that use Vulkan.

I also updated the SDK-specific callPackage to make it more robust. It also now provides a couple of GCC stdenvs and rustPlatform.

Sandboxing is set to relaxed. I had to add a sandboxProfile to ccl to let it build with sandboxing.

Things done

reckenrode · 2022-07-10T05:16:17Z

I rebased on staging. That’s way more rebuilds than I expected.

reckenrode · 2022-07-10T05:25:35Z

I converted the PR to a draft because it looks like nixpkgs-review is identifying more things that are broken, which should probably be addressed in a “fix the broken things on Darwin” PR.

reckenrode · 2022-07-10T14:11:38Z

Based on feedback from @SuperSandro2000, I’ve split the MoltenVK changes off into a separate PR (#180988) to keep this one focused on fixing packages that were marked broken on Darwin.

toonn

Pretty much all of this LGTM.

toonn · 2022-07-10T19:28:43Z

pkgs/development/interpreters/python/rustpython/default.nix

@@ -31,14 +41,5 @@ rustPlatform.buildRustPackage rec {
    homepage = "https://rustpython.github.io";
    license = licenses.mit;
    maintainers = with maintainers; [ prusnak ];
-


Maybe we can leave a TODO here to switch back away from the SDK when stdenv gets bumped to 10.13, if that is enough?

toonn · 2022-07-10T19:30:29Z

pkgs/tools/backup/bupstash/default.nix

@@ -28,10 +28,6 @@ rustPlatform.buildRustPackage rec {
    homepage = "https://bupstash.io";
    license = licenses.mit;
    platforms = platforms.unix;
-    # = note: Undefined symbols for architecture x86_64:


This looks like another one that might work with the stdenv bump to 10.13. Another TODO?

toonn · 2022-07-10T19:31:06Z

pkgs/tools/filesystems/squashfs-tools-ng/default.nix

@@ -21,17 +21,5 @@ stdenv.mkDerivation rec {
    license = licenses.gpl3Plus;
    maintainers = with maintainers; [ qyliss ];
    platforms = platforms.unix;
-


utimensat will be in the stdenv bump to 10.13, TODO?

toonn · 2022-07-10T19:53:56Z

pkgs/os-specific/darwin/apple-sdk-11.0/default.nix

@@ -46,7 +46,7 @@ let
    inherit MacOSX-SDK;

    Libsystem = callPackage ./libSystem.nix {};
-    LibsystemCross = pkgs.darwin.Libsystem;
+    LibsystemCross = packages.Libsystem;


Are we sure this is fine? I feel like it'd have been defined as an alias before if that was the case.

Not sure I follow. Aren’t aliases for deprecated names?

This change was to correct what I believe was a mistake in the previous patch. The 11.0 SDK originally assumed that pkgs.darwin.Libsystem belongs to the SDK, but that’s not necessarily the case. On x86_64-darwin, that’s the 10.12 SDK.

What I meant with an alias was rec { Libsystem = ...; LibsystemCross = Libsystem; }, not aliases.nix.

I'm really not sure how cross works in Nix. But I do suspect LibsystemCross might have to be different from Libsystem in some way, no?

Ah. It’s not currently different on aarch64-darwin, so I propagated that here. I assume it works because Apple’s SDK is naturally cross-capable.

toonn · 2022-07-10T21:37:43Z

pkgs/os-specific/darwin/apple-sdk-11.0/pkg-compatibility.nix

+      };
+    in
+    if !needsOverrides then stdenv
+    else overridePlatformDarwinVersions (pkgs.overrideCC stdenv clang) "10.12" "11.0";


I'm having some trouble keeping track of all the overrides.
Is this stdenv the standard stdenv, based on the source releases for 10.12, with an overridden compiler that is based on SDK Libsystem? Is that the only reference to system libraries from stdenv? Or is this completely taken care of by the inherit in callPackage?

It’s the 10.12 stdenv with the compiler and SDK Libsystem replaced. This is the same implementation as in current master just moved to a new file and with build/hostPlatform overridden. I couldn’t see a way to build a pure 11.0 SDK without doing surgery on the Darwin stdenv definition.

I originally wrote the overridePlatformDarwinVersions function to use with overriding the GCC stdenv, but it turns out that builds fine from the private nixpkgs. If it would improve clarity, I can inline the function definition, so it’s more like how things are currently on master.

The function is perfectly fine.

What I'm concerned about here is that apple_sdk_11_0.stdenv is still using the 10.12 SDK somehow. Having a bootstrap stage is totally reasonable if that's what it takes. If it's no problem whatsoever then so much the better : )

reckenrode · 2022-07-12T20:50:39Z

I’ve got an updated reflecting the changes requested here plus also discussed on Matrix. I’m waiting for it to build successfully before pushing. It now uses a clang built in the 11.0 SDK stdenv instead of one from the 10.12 SDK with overrides.

reckenrode · 2022-07-12T20:52:03Z

I also added cmake to the callPackage since that was mentioned on Matrix as having an issue with pulling in 10.12 SDK stuff even with an 11.0 SDK environment. I expect other compilers and build tools may need this as well.

reckenrode · 2022-07-13T01:31:29Z

Converting this to draft while I work through the stdenv stuff.

siraben · 2022-07-13T05:16:47Z

Result of nixpkgs-review pr 180931 run on aarch64-darwin 1

4 packages marked as broken and skipped:

llvmPackages_10.clangNoLibc
llvmPackages_7.clangNoLibc
llvmPackages_8.clangNoLibc
llvmPackages_9.clangNoLibc

1 package failed to build:

vc4-newlib

34 packages built:

aspino
bintoolsNoLibc
binutilsNoLibc
bsnes-hd
btop
bullet-roboschool
crossLibcStdenv
llvmPackages.bintoolsNoLibc
llvmPackages.clangNoCompilerRt
llvmPackages.clangNoLibc
llvmPackages.compiler-rt-no-libc
llvmPackages_10.bintoolsNoLibc
llvmPackages_10.clangNoCompilerRt
llvmPackages_12.bintoolsNoLibc
llvmPackages_12.clangNoCompilerRt
llvmPackages_12.clangNoLibc
llvmPackages_13.bintoolsNoLibc
llvmPackages_13.clangNoCompilerRt
llvmPackages_13.clangNoLibc
llvmPackages_14.bintoolsNoLibc
llvmPackages_14.clangNoCompilerRt
llvmPackages_14.clangNoLibc
llvmPackages_7.bintoolsNoLibc
llvmPackages_7.clangNoCompilerRt
llvmPackages_8.bintoolsNoLibc
llvmPackages_8.clangNoCompilerRt
llvmPackages_9.bintoolsNoLibc
llvmPackages_9.clangNoCompilerRt
newlib-nanoCross
newlibCross
or1k-newlib
preLibcCrossHeaders
rustpython
vengi-tools

chasecaleb · 2022-07-13T14:31:03Z

@reckenrode Could you look at Trivy as part of this PR too, please? I suspect that should work now too:

nixpkgs/pkgs/tools/admin/trivy/default.nix

Lines 52 to 54 in 0906692

    
           # Need updated macOS SDK 
        
           # https://github.com/NixOS/nixpkgs/issues/101229 
        
           broken = (stdenv.isDarwin && stdenv.isx86_64);

Edit: Sorry, I made this comment too quickly and just realized you called out Go packages as being worked on in #179622 . So maybe Trivy isn't relevant here in your PR? But I don't see Trivy being unmarked as broken in that PR either so I don't know where it belongs.

reckenrode · 2022-07-14T01:52:58Z

Sorry, I made this comment too quickly and just realized you called out Go packages as being worked on in #179622 . So maybe Trivy isn't relevant here in your PR? But I don't see Trivy being unmarked as broken in that PR either so I don't know where it belongs.

Based on what @SuperSandro2000 said in #180704, I believe the plan is to address those packages in a follow up PR to keep the Go 1.18 update PR from growing out of hand.

prusnak · 2022-12-27T18:15:20Z

This should be broken to individual pull requests, since some of the packages have already been fixed elsewhere.

reckenrode requested review from Mic92, LnL7, zowoq, FRidh and jonringer as code owners July 10, 2022 04:54

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Jul 10, 2022

github-actions bot added 6.topic: python 6.topic: rust labels Jul 10, 2022

reckenrode force-pushed the darwin-sdk-fixes branch from d9e738b to 469bb27 Compare July 10, 2022 05:00

ofborg bot added 8.has: clean-up 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 501+ 10.rebuild-darwin: 501-1000 10.rebuild-linux: 501+ 10.rebuild-linux: 2501-5000 labels Jul 10, 2022

reckenrode marked this pull request as draft July 10, 2022 05:06

reckenrode force-pushed the darwin-sdk-fixes branch from 469bb27 to e349cb4 Compare July 10, 2022 05:14

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: ruby 6.topic: stdenv Standard environment 8.has: module (update) This PR changes an existing module in `nixos/` labels Jul 10, 2022

reckenrode changed the base branch from master to staging July 10, 2022 05:14

reckenrode marked this pull request as ready for review July 10, 2022 05:15

github-actions bot removed 8.has: module (update) This PR changes an existing module in `nixos/` 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: stdenv Standard environment 6.topic: ruby labels Jul 10, 2022

reckenrode marked this pull request as draft July 10, 2022 05:24

reckenrode added 3 commits July 10, 2022 09:57

plan9port: fix build on x86_64-darwin

8127f5e

mkp224o: fix build on x86_64-darwin

aa9b1ac

blackbox: fix build on x86_64-darwin

d5b70be

reckenrode force-pushed the darwin-sdk-fixes branch 2 times, most recently from ea2917e to d5b70be Compare July 10, 2022 14:03

reckenrode mentioned this pull request Jul 10, 2022

moltenvk: refactor, update, and fix breakage #180988

Merged

13 tasks

toonn approved these changes Jul 10, 2022

View reviewed changes

bobby285271 added the 12.approvals: 1 This PR was reviewed and approved by one reputable person label Jul 11, 2022

reckenrode marked this pull request as draft July 13, 2022 01:31

reckenrode mentioned this pull request Jul 14, 2022

Update macOS SDK #101229

Closed

reckenrode mentioned this pull request Jul 15, 2022

make-bootstrap-tools: add pbzx and tbd tools on x86_64-darwin #181550

Merged

13 tasks

zowoq mentioned this pull request Aug 2, 2022

wizer: init at 1.4.0 #184597

Merged

13 tasks

stephank mentioned this pull request Aug 8, 2022

cmake: detect libc location at runtime #181431

Merged

13 tasks

zowoq mentioned this pull request Sep 22, 2022

Revert "rustc: build with jemalloc" #192328

Merged

13 tasks

Strum355 mentioned this pull request Nov 16, 2022

Cross compiling to x86_64-darwin from aarch64-darwin seems broken #180771

Closed

wegank mentioned this pull request Jan 4, 2023

bupstash, httm, rustpython: unbreak on x86_64-darwin #209065

Closed

13 tasks

wegank mentioned this pull request Jan 28, 2023

cargo-watch: fix darwin build #213072

Merged

13 tasks

reckenrode mentioned this pull request Mar 28, 2023

Tracking issue: Update Darwin bootstrap files #222717

Closed

wegank added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 19, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x86_64-darwin: fix packages that won’t build #180931

x86_64-darwin: fix packages that won’t build #180931

reckenrode commented Jul 10, 2022 •

edited

Loading

reckenrode commented Jul 10, 2022

reckenrode commented Jul 10, 2022

reckenrode commented Jul 10, 2022

toonn left a comment

toonn Jul 10, 2022

toonn Jul 10, 2022

toonn Jul 10, 2022

toonn Jul 10, 2022

reckenrode Jul 12, 2022

toonn Jul 12, 2022

reckenrode Jul 12, 2022

toonn Jul 10, 2022

reckenrode Jul 12, 2022 •

edited

Loading

toonn Jul 12, 2022

reckenrode commented Jul 12, 2022

reckenrode commented Jul 12, 2022

reckenrode commented Jul 13, 2022

siraben commented Jul 13, 2022

chasecaleb commented Jul 13, 2022 •

edited

Loading

reckenrode commented Jul 14, 2022

prusnak commented Dec 27, 2022 •

edited

Loading

x86_64-darwin: fix packages that won’t build #180931

Are you sure you want to change the base?

x86_64-darwin: fix packages that won’t build #180931

Conversation

reckenrode commented Jul 10, 2022 • edited Loading

Description of changes

Things done

reckenrode commented Jul 10, 2022

reckenrode commented Jul 10, 2022

reckenrode commented Jul 10, 2022

toonn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reckenrode Jul 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reckenrode commented Jul 12, 2022

reckenrode commented Jul 12, 2022

reckenrode commented Jul 13, 2022

siraben commented Jul 13, 2022

chasecaleb commented Jul 13, 2022 • edited Loading

reckenrode commented Jul 14, 2022

prusnak commented Dec 27, 2022 • edited Loading

reckenrode commented Jul 10, 2022 •

edited

Loading

reckenrode Jul 12, 2022 •

edited

Loading

chasecaleb commented Jul 13, 2022 •

edited

Loading

prusnak commented Dec 27, 2022 •

edited

Loading