macOS Sierra and Haskell "malformed mach-o: load commands size" #22810

Dridus · 2017-02-14T23:48:15Z

Issue description

The (at this point infamous) load commands size problem still occurs on nixpkgs even after the GHC 8.0.2 / LTS 8.0 update.

I'll try to summarize the underlying issue, but please be aware I'm not an expert and am just stitching together things from other issues. Please correct me if the details are wrong.

This issue is due to a change in macOS's linker in Sierra which limits the mach-O linker load commands to 32768 bytes. Those load commands are dominated by RPATH entries and nix exacerbates the issue greatly by having a ton of long named directories, around one per dependency.

In GHC (https://ghc.haskell.org/trac/ghc/ticket/12479), Stack (commercialhaskell/stack#2577), and Cabal (haskell/cabal#3955, haskell/cabal#3982, et al) it looks like it was resolved by carefully managing the location of dylibs to all be located in the same place so that only one RPATH needed emitting (https://ghc.haskell.org/trac/ghc/ticket/12479#comment:42, probably others).

I could imagine a similar solution could be cooked up by generic-builder and with-package-wrapper to create a lib directory for each haskell derivation to build that has symlinks for each transient dependency, but I'm really quite new to all this so I haven't attempted it yet.

Steps to reproduce

nix-build stack-a5b8d468.nix with stack-a5b8d468.nix:

let
  bootstrap = import <nixpkgs> {};
  pkgsSrc = bootstrap.fetchFromGitHub {
    owner = "NixOS";
    repo = "nixpkgs";
    rev = "a5b8d468a504e0eedcda71de1694201806fb921d";
    sha256 = "0alh9v12nyxsdlssg72k892zdzzc15wr8fqy8qpq9rc282bpx7xm";
  };
  pkgs = import pkgsSrc {};
in
  pkgs.stack

On my system yields:

…
[ 10 of 121] Compiling System.Process.Read ( src/System/Process/Read.hs, dist/build/System/Process/Read.o )

<no location info>: error:
    ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.2 for x86_64-apple-darwin):
	Loading temp shared object failed: dlopen(/private/var/folders/4b/7smbp2kj7m770r24jxxfwr700000gn/T/nix-build-stack-1.3.2.drv-0/ghc46406_0/libghc_135.dylib, 5): no suitable image found.  Did find:
	/private/var/folders/4b/7smbp2kj7m770r24jxxfwr700000gn/T/nix-build-stack-1.3.2.drv-0/ghc46406_0/libghc_135.dylib: malformed mach-o: load commands size (41792) > 32768

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
…

stack-a5b8d468.txt

Technical details

System: macOS Sierra 10.12.3
Nix version: nix-env (Nix) 1.11.4
Nixpkgs version: a5b8d46

The text was updated successfully, but these errors were encountered:

aljce · 2017-03-03T15:52:32Z

This is crushing. Basically none of my haskell projects are executable.

Dridus · 2017-03-03T17:45:13Z

It is. @peti @LnL7, anybody else involved in the haskell and/or darwin infrastructure maybe have any opinions, tips, or concerns to contribute? I intend to try to fix it at some point soon if someone more experienced doesn't come along, but I'm definitely pretty new to the whole nix infrastructure.

aljce · 2017-03-03T20:07:13Z

^ same for me

peti · 2017-03-03T20:58:11Z

I don't see a reasonable solution to this issue other than to disable dynamic linking of Haskell libraries on Darwin.

robinbb · 2017-03-04T03:08:39Z

I don't see a reasonable solution to this issue other than to disable dynamic linking of Haskell libraries on Darwin.

How can one accomplish this?

peti · 2017-03-04T09:41:32Z

Like http://nixos.org/nixpkgs/manual/#how-to-build-with-profiling-enabled, but instead of enableLibraryProfiling = true set

enableSharedExecutables = false;
enableSharedLibraries = false;

Dridus · 2017-03-06T15:46:02Z

as a preliminary report, I tried this with our project that was bombing by using:

    haskellPackageOverrides = self: super: {
      mkDerivation = args: super.mkDerivation (args // {
        enableSharedExecutables = false;
        enableSharedLibraries = false;
      });
    };

and it failed due to a build failure with file-embed:

building path(s) ‘/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10’
setupCompilerEnvironmentPhase
Build with /nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2.
unpacking sources
unpacking source archive /nix/store/qh39vhl5920wgaxwj7jy69njjy548y3y-file-embed-0.0.10.tar.gz
source root is file-embed-0.0.10
setting SOURCE_DATE_EPOCH to timestamp 1461252791 of file file-embed-0.0.10/test/sample/foo
patching sources
compileBuildDriverPhase
setupCompileFlags: -package-db=/private/var/folders/4b/7smbp2kj7m770r24jxxfwr700000gn/T/nix-build-file-embed-0.0.10.drv-0/package.conf.d -j8 -threaded
[1 of 1] Compiling Main             ( Setup.lhs, /private/var/folders/4b/7smbp2kj7m770r24jxxfwr700000gn/T/nix-build-file-embed-0.0.10.drv-0/Main.o )
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --with-gcc=clang --package-db=/private/var/folders/4b/7smbp2kj7m770r24jxxfwr700000gn/T/nix-build-file-embed-0.0.10.drv-0/package.conf.d --ghc-option=-j8 --disable-split-objs --disable-library-profiling --disable-profiling --disable-shared --disable-coverage --enable-library-vanilla --disable-executable-dynamic --enable-tests
Configuring file-embed-0.0.10...
Dependency base ==4.*: using base-4.9.1.0
Dependency bytestring >=0.9.1.4: using bytestring-0.10.8.1
Dependency directory >=1.0.0.3: using directory-1.3.0.0
Dependency file-embed -any: using file-embed-0.0.10
Dependency filepath -any: using filepath-1.4.1.1
Dependency template-haskell -any: using template-haskell-2.11.1.0
Using Cabal-1.24.2.0 compiled by ghc-8.0
Using compiler: ghc-8.0.2
Using install prefix:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10
Binaries installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/bin
Libraries installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/lib/ghc-8.0.2/file-embed-0.0.10
Dynamic libraries installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/lib/ghc-8.0.2/x86_64-osx-ghc-8.0.2
Private binaries installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/libexec
Data files installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/share/x86_64-osx-ghc-8.0.2/file-embed-0.0.10
Documentation installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/share/doc/x86_64-osx-ghc-8.0.2/file-embed-0.0.10
Configuration files installed in:
/nix/store/ypy48rz2k6b8vm6mfw4qi9409bfrwc3y-file-embed-0.0.10/etc
No alex found
Using ar found on system at:
/nix/store/51bin2x0i1v9jff0sd98b51hmzv00g0v-cctools-binutils-darwin/bin/ar
No c2hs found
No cpphs found
Using gcc version 4.2.1 given by user at:
/nix/store/pr2jbdyfagqarc0pb023c64r5ny4gn87-clang-wrapper-3.7.1/bin/clang
Using ghc version 8.0.2 found on system at:
/nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2/bin/ghc
Using ghc-pkg version 8.0.2 found on system at:
/nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2/bin/ghc-pkg
No ghcjs found
No ghcjs-pkg found
No greencard found
Using haddock version 2.17.3 found on system at:
/nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2/bin/haddock
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at:
/nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2/bin/hpc
Using hsc2hs version 0.68.1 found on system at:
/nix/store/7nl6dii88xd761nnz3xyh11qcnrqvqri-ghc-8.0.2/bin/hsc2hs
Using hscolour version 1.24 found on system at:
/nix/store/fkwgiwswps6akvh0l5wga450b8hgkzxw-hscolour-1.24.1/bin/HsColour
No jhc found
Using ld found on system at:
/nix/store/pr2jbdyfagqarc0pb023c64r5ny4gn87-clang-wrapper-3.7.1/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using strip found on system at:
/nix/store/51bin2x0i1v9jff0sd98b51hmzv00g0v-cctools-binutils-darwin/bin/strip
Using tar found on system at:
/nix/store/ybaqdf0hlkkjrlsx5dazlp0arz6hbraq-gnutar-1.29/bin/tar
No uhc found
building
Building file-embed-0.0.10...
Preprocessing library file-embed-0.0.10...
[1 of 1] Compiling Data.FileEmbed   ( Data/FileEmbed.hs, dist/build/Data/FileEmbed.o )
Preprocessing test suite 'test' for file-embed-0.0.10...
[1 of 1] Compiling Main             ( test/main.hs, dist/build/test/test-tmp/Main.o )

<no location info>: error:
    <command line>: can't load .so/.DLL for: libHSfile-embed-0.0.10-KWMmUDaNSRP7AxWSEHlrrK.dylib (dlopen(libHSfile-embed-0.0.10-KWMmUDaNSRP7AxWSEHlrrK.dylib, 5): image not found)
builder for ‘/nix/store/n9b72x3vfpaxl8f9dbkws759cgn7fsl1-file-embed-0.0.10.drv’ failed with exit code 1

I haven't tracked down yet for sure why this fails. It doesn't seem like file-embed has any odd test suite configuration either nixpkgs or its cabal file. It's not the first package it tried to build with a test suite either - code-page built prior with test suites. file-embed may be the first test suite that uses TH though and that could be the reason.

ghost · 2017-03-10T10:13:29Z

Here's a hacky way to workaround this:

pkgs.haskell.packages.ghc802.override {
  overrides = self: super: {
    mkDerivation = args: super.mkDerivation (args // { 
      postCompileBuildDriver = ''
        echo "Patching dynamic library dependencies"
        # 1. Link all dylibs from 'dynamic-library-dirs's in package confs to $out/lib/links
        mkdir -p $out/lib/links
        for d in $(grep dynamic-library-dirs $packageConfDir/*|awk '{print $2}'); do
          ln -s $d/*.dylib $out/lib/links
        done
            
        # 2. Patch 'dynamic-library-dirs' in package confs to point to the symlink dir
        for f in $packageConfDir/*.conf; do
          sed -i "s,dynamic-library-dirs: .*,dynamic-library-dirs: $out/lib/links," $f
        done

        # 3. Recache package database
        ghc-pkg --package-db="$packageConfDir" recache
     '';
    });
  };
}

@peti would you be ok with something like this for Darwin only? Would be happy to prepare a PR.

peti · 2017-03-10T10:37:42Z

@dajmaki, yeah, that would be a viable workaround, I suppose.

LnL7 · 2017-03-13T19:57:00Z

@dajmaki I was thinking about something like this with rpath before. This will cause conflicts when using multiple haskell packages in a buildEnv unless you put the symlinks in a separate drv or output.

3noch · 2017-03-13T20:41:52Z

I just got hit with this. Still no path forward?

k0001 · 2017-03-13T22:18:25Z

Just got bitten by this as well.

joamaki · 2017-03-14T10:23:43Z

(whoops commented from my work account 'dajmaki', switching to my private one). @LnL7 could you give an example of a conflict? I'm not quite following.

EDIT: Another problem with my workaround is that if one is building a statically linked executable this will include the dynamic libraries in the closure for no reason.

domenkozar · 2017-03-22T13:04:32Z

I get this on Sierra once I pass dead_code to linker when building purescript. See #21200 (comment)

That makes no sense to me, why is it suddenly using longer path?

cstrahan · 2017-04-18T23:55:11Z

Assigning myself to keep this on my radar, but no guarantee I'll be able to get too it in a timely manner.

The Sierra linker added a limit on the number of paths that any one dynamic library (`*.dylib`) can reference. This causes problems when a Haskell library has many immediate dependencies (NixOS#22810). We follow a similar fix as GHC/Cabal/Stack: for each derivation, create a new directory with symlinks to all the dylibs of its immediate dependencies, and patch its package DB to reference that directory using the new `dynamic-library-dirs` field. Note that this change is a no-op for older versions of GHC, i.e., they will continue to fail on some packages as before. Also note that this change causes the bootstrapped versions of GHC to be recompiled, since they depend on `hscolour` which is built by `generic-builder.nix`. Tested by building the `stack` binary as described in NixOS#22810.

The Sierra linker added a limit on the number of paths that any one dynamic library (`*.dylib`) can reference. This causes problems when a Haskell library has many immediate dependencies (#22810). We follow a similar fix as GHC/Cabal/Stack: for each derivation, create a new directory with symlinks to all the dylibs of its immediate dependencies, and patch its package DB to reference that directory using the new `dynamic-library-dirs` field. Note that this change is a no-op for older versions of GHC, i.e., they will continue to fail on some packages as before. Also note that this change causes the bootstrapped versions of GHC to be recompiled, since they depend on `hscolour` which is built by `generic-builder.nix`. Tested by building the `stack` binary as described in #22810. (cherry picked from commit 7131e06)

Dridus · 2017-05-08T16:07:07Z

marking this issue closed on account of #25537 being merged and fixing my case! thanks very much @judah @shlevy @peti @LnL7 et al!

copumpkin · 2017-07-12T15:19:22Z

I had an idea for a workaround here... not sure it belongs in Nix though.

Mach-O dylibs have the option to re-export other dylibs. We can create "intermediate" dylibs that re-export all the other libraries, as deep as we want. So if the first re-exporting dylib fills up, we create another one. And if needed, we do it recursively. I hope we don't get that big though 😄

domenkozar · 2017-07-20T13:02:13Z

I don't know how we could solve this for stack in Nix itself.

We have about 300 dependencies, each taking on average about 100 chars for the rpath path, resulting into ~33000 chars, just above the limit.

I think stack will need to handle this similarly as we do in Nix :)

domenkozar · 2017-07-20T13:08:22Z

What if we package dyld https://opensource.apple.com/tarballs/dyld/dyld-421.1.tar.gz and patch #define MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE (32*1024)? Nix wins 🗡️

But this probably means we have to bump our SDK to 10.12

copumpkin · 2017-07-20T13:40:24Z

Pre-10.10 we might have been able to use a custom dyld but this happened

shlevy · 2017-07-20T15:35:18Z

Holy crap could Darwin be more hostile to developers? 😠

Dridus · 2017-07-20T16:19:53Z

sure! they could require everything to be signed

cstrahan · 2017-07-20T16:28:33Z

Has anyone opened a ticket with Apple (rdar or Radar or whatever they call it) regarding MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE? Might be best to fix the problem at its source... assuming they actually listen to their developers...

Ericson2314 · 2017-07-21T05:43:51Z

I can only hope that since Apple is so against static linking they will be slightly more sympathetic.

shlevy · 2017-11-11T23:44:00Z

Marking closed since #27536 is in, reopen if this is still broken.

Previously the package conf files were handled without paying attention to the fact that it's pretty-printed output. One problem was discovered with GHC 8.8.1 on Darwin, where the dynamic-library-dirs first field seems to have increased in length, meaning while before it was dynamic-library-dirs: some-small-directory-name some-more-directories Now it is dynamic-library-dirs: some-larger-directory-name some-more-directories Which breaks the code installed for NixOS#25537, because that assumed the former format, resulting in the reoccurence of the bug in NixOS#22810, see infinisil/all-hies#43 This commit fixes this by "unprettyfying" the package conf files before processing them.

Previously the package conf files were handled without paying attention to the fact that it's pretty-printed output. One problem was discovered with GHC 8.8.1 on Darwin, where the dynamic-library-dirs first field seems to have increased in length, meaning while before it was dynamic-library-dirs: some-small-directory-name some-more-directories Now it is dynamic-library-dirs: some-larger-directory-name some-more-directories Which breaks the code installed for #25537, because that assumed the former format, resulting in the reoccurence of the bug in #22810, see infinisil/all-hies#43 This commit fixes this by "unprettyfying" the package conf files before processing them. Closes #78738.

Previously the package conf files were handled without paying attention to the fact that it's pretty-printed output. One problem was discovered with GHC 8.8.1 on Darwin, where the dynamic-library-dirs first field seems to have increased in length, meaning while before it was dynamic-library-dirs: some-small-directory-name some-more-directories Now it is dynamic-library-dirs: some-larger-directory-name some-more-directories Which breaks the code installed for NixOS#25537, because that assumed the former format, resulting in the reoccurence of the bug in NixOS#22810, see infinisil/all-hies#43 This commit fixes this by "unprettyfying" the package conf files before processing them. Closes NixOS#78738.

The workaround that this change deletes was initially contributed in NixOS#25537 to mitigate NixOS#22810 until a more permanent solution could be devised. That more permanent solution was eventually contributed in NixOS#27536, which now obviates the initial workaround, so this change removes it.

The Sierra linker added a limit on the number of paths that any one dynamic library (`*.dylib`) can reference. This causes problems when a Haskell library has many immediate dependencies (NixOS#22810). We follow a similar fix as GHC/Cabal/Stack: for each derivation, create a new directory with symlinks to all the dylibs of its immediate dependencies, and patch its package DB to reference that directory using the new `dynamic-library-dirs` field. Note that this change is a no-op for older versions of GHC, i.e., they will continue to fail on some packages as before. Also note that this change causes the bootstrapped versions of GHC to be recompiled, since they depend on `hscolour` which is built by `generic-builder.nix`. Tested by building the `stack` binary as described in NixOS#22810. (cherry picked from commit 7131e06)

LnL7 added 6.topic: darwin Running or building packages on Darwin 6.topic: haskell labels Feb 15, 2017

LnL7 mentioned this issue Mar 7, 2017

Haskell/just static executables #23607

Merged

LnL7 mentioned this issue Mar 20, 2017

Illegal Instruction: 4 on macOS 10.12 Sierra #17372

Closed

domenkozar added the 0.kind: bug Something is broken label Mar 22, 2017

cstrahan self-assigned this Apr 18, 2017

judah mentioned this issue May 5, 2017

haskell: work around linker limits on Mac OS X Sierra. #25537

Merged

4 tasks

Dridus closed this as completed May 8, 2017

henrytill mentioned this issue May 10, 2017

Linker limit workaround on Darwin causes buildEnv environments to conflict #25627

Open

Ericson2314 mentioned this issue Jul 21, 2017

Linking hack for macOS Sierra #27536

Merged

8 tasks

3noch mentioned this issue Jul 22, 2017

Darwin ghc build is broken reflex-frp/reflex-platform#133

Closed

shlevy closed this as completed Nov 11, 2017

avieth mentioned this issue May 1, 2018

[CBR-51] use stack lts-11.2 (#2735) input-output-hk/cardano-sl#2745

Merged

chronal-xnor mentioned this issue Aug 16, 2019

cc_tests with a large dependency tree eventually exceed the Mac dyld load commands size limit bazelbuild/bazel#9190

Closed

infinisil mentioned this issue Jan 28, 2020

hie-8.8.1 from unstable errors out at launch infinisil/all-hies#43

Closed

infinisil mentioned this issue Jan 29, 2020

haskell generic-builder: Fix package conf handling #78738

Closed

1 task

veprbl mentioned this issue Feb 17, 2020

haskell generic-builder: fix Darwin regression for lmdb #80191

Merged

10 tasks

typesanitizer mentioned this issue Mar 28, 2022

The Grinch stole all the vowels: Odd naming in .cabal/store (macOS) haskell/cabal#7209

Open

Gabriella439 mentioned this issue Dec 1, 2022

haskellPackages.mkDerivation: Remove Darwin dynamic libraries workaround #203963

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS Sierra and Haskell "malformed mach-o: load commands size" #22810

macOS Sierra and Haskell "malformed mach-o: load commands size" #22810

Dridus commented Feb 14, 2017 •

edited

Loading

aljce commented Mar 3, 2017

Dridus commented Mar 3, 2017

aljce commented Mar 3, 2017

peti commented Mar 3, 2017

robinbb commented Mar 4, 2017

peti commented Mar 4, 2017

Dridus commented Mar 6, 2017

ghost commented Mar 10, 2017 •

edited by ghost

Loading

peti commented Mar 10, 2017

LnL7 commented Mar 13, 2017

3noch commented Mar 13, 2017

k0001 commented Mar 13, 2017

joamaki commented Mar 14, 2017 •

edited

Loading

domenkozar commented Mar 22, 2017

cstrahan commented Apr 18, 2017

Dridus commented May 8, 2017

copumpkin commented Jul 12, 2017

domenkozar commented Jul 20, 2017

domenkozar commented Jul 20, 2017 •

edited

Loading

copumpkin commented Jul 20, 2017

shlevy commented Jul 20, 2017

Dridus commented Jul 20, 2017

cstrahan commented Jul 20, 2017

Ericson2314 commented Jul 21, 2017 •

edited

Loading

shlevy commented Nov 11, 2017

macOS Sierra and Haskell "malformed mach-o: load commands size" #22810

macOS Sierra and Haskell "malformed mach-o: load commands size" #22810

Comments

Dridus commented Feb 14, 2017 • edited Loading

Issue description

Steps to reproduce

Technical details

aljce commented Mar 3, 2017

Dridus commented Mar 3, 2017

aljce commented Mar 3, 2017

peti commented Mar 3, 2017

robinbb commented Mar 4, 2017

peti commented Mar 4, 2017

Dridus commented Mar 6, 2017

ghost commented Mar 10, 2017 • edited by ghost Loading

peti commented Mar 10, 2017

LnL7 commented Mar 13, 2017

3noch commented Mar 13, 2017

k0001 commented Mar 13, 2017

joamaki commented Mar 14, 2017 • edited Loading

domenkozar commented Mar 22, 2017

cstrahan commented Apr 18, 2017

Dridus commented May 8, 2017

copumpkin commented Jul 12, 2017

domenkozar commented Jul 20, 2017

domenkozar commented Jul 20, 2017 • edited Loading

copumpkin commented Jul 20, 2017

shlevy commented Jul 20, 2017

Dridus commented Jul 20, 2017

cstrahan commented Jul 20, 2017

Ericson2314 commented Jul 21, 2017 • edited Loading

shlevy commented Nov 11, 2017

Dridus commented Feb 14, 2017 •

edited

Loading

ghost commented Mar 10, 2017 •

edited by ghost

Loading

joamaki commented Mar 14, 2017 •

edited

Loading

domenkozar commented Jul 20, 2017 •

edited

Loading

Ericson2314 commented Jul 21, 2017 •

edited

Loading