haskell: add the linkWithGold function to link packages with ld.gold

[basvandijk]

This was inspired by https://www.reddit.com/r/haskell/comments/63shj7/link_with_the_gold_linker_for_faster_build_times/.

[mention-bot]

@basvandijk, thanks for your PR! By analyzing the history of the files in this pull request, we identified @peti, @domenkozar and @pikajude to be potential reviewers.

[basvandijk]

@peti it would be interesting turning on linkWithGold by default (for GHC >= 7.8) and let hydra build the package set and see how much will break. I would expect that we need to add a few --ghc-option=-optl-lm here and there.

[nh2]

I would expect that we need to add a few --ghc-option=-optl-lm here and there.

@basvandijk I think those -lms should then be integrated into the upstream cabal packages, as that is the clean way to do it.

[basvandijk]

@nh2 agreed.

I forced-pushed a commit to this PR that adds a reference to a PR for double-conversion to explicitly link with libm.

[basvandijk]

BTW I initially used the following implementation but abandoned it because the current implementation is simpler and doesn't require rebuilding all Haskell packages. But it might be interesting to compare the two...

[basvandijk]

@nh2 it would be great to also have your linkWithLLD available. Are you working on a PR for that BTW?

[nh2]

it would be great to also have your linkWithLLD available. Are you working on a PR for that BTW?

@basvandijk do you mean to add this example as a nixpkgs package?

[basvandijk]

@nh2 no, I mean I would like to have a function like my linkWithGold but then for linking with lld. It could be called linkWithLLD and work something like:

linkWithLLD = drv : appendConfigureFlag drv
    "--with-gcc=clang --with-ld=clang";

What is probably missing is that clang should be brought in scope.

[nh2]

@basvandijk Yes; and pkgs.lld would also be needed, and flags to build to pass ld-option, cc-option, ghc-option etc.

I'm not currently working on that, so if you'd like to do it please go ahead, it would be awesome!

Also, if you come across a problem involving linking functions from libstc++, ping me.

[basvandijk]

Ok this sounds like a nice weekend project. I'll see if I have some time for it. Thanks for the info!

[ljli]

Great! Any thoughts on making it the default where possible? We pass --split-sections by default for GHC > 8.0.1, which apparently requires the gold linker to take effect and thus I assume doesn't do anything currently. Wrong info, still desirable though to reduce compile time.

[nh2]

@basvandijk I found that Haskell executables built with lld in nix that also link -l system shared libraries don't link correctly, with entries in ldd in resulting executbles being not found (rpaths are missing). That's because ld and gold have wrappers that do the following:

nixpkgs/pkgs/build-support/cc-wrapper/ld-wrapper.sh

Line 133 in 593b46f

# Second, for each directory in the library search path (-L...),

 # First, find all -L... switches.

    # Second, for each directory in the library search path (-L...),
    # see if it contains a dynamic library used by a -l... flag.  If
    # so, add the directory to the rpath.

lld currently doesn't have such a wrapper.

[nh2]

Also this type of patch or setting to --enable-new-dtags will likely be needed.

[basvandijk]

@aristidb thanks for the merge. Could this also be cherry-picked on release-17.03? I would like to use this in production.

[nh2]

I've created a separate issue #24744 for the problem that lld doesn't work correctly with nix.

[domenkozar]

I wonder if linkWithGold should also pass --ghc-option=-optl-lm to the package it's compiling. Maybe also --ghc-option=-optl-pthread.

[nh2]

I wonder if linkWithGold should also pass --ghc-option=-optl-lm to the package it's compiling. Maybe also --ghc-option=-optl-pthread.

@domenkozar Personally I think it shouldn't, unless it's a temporary hack. It should be fixed upstream, as ld happily accepts these flags and that's better than implicitly linking those anyway.

[domenkozar]

Every package that uses Prelude.sqrt would need that flag added to cabal file. That sounds unfeasible to me :)

[nh2]

Every package that uses Prelude.sqrt would need that flag added to cabal file.

@domenkozar With "upstream" I mean the package that declares the function, not the one that uses it (that would be infeasible, users shouldn't have to rely on the implementation detail with what C function a Haskell function is implemented).

So in that case I think base should have that extra-libraries: m.

Do you think this will make any problem?

[domenkozar]

If we can get base fixed, I'm all for it.

[domenkozar]

There's a bug when you depend on a library that is linked with gold: https://ghc.haskell.org/trac/ghc/ticket/13810#comment:3

[nh2]

I would expect that we need to add a few --ghc-option=-optl-lm here and there.

@basvandijk I think those -lms should then be integrated into the upstream cabal packages, as that is the clean way to do it.

I have now confirmed that --ghc-option=-optl-lm should not be used.

It wil pass it also to those linker invocations that create the individual Haskell module .o files, which will then fail with e.g.

/nix/store/iz98xp5p48jayn034wq89y30plpx5xns-binutils-2.27/bin/ld.gold: fatal error: cannot mix -r with dynamic object /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/libm.so.6

in some cases.

Putting extra-libraries: m into the executable section of the .cabal file is the only working solution (or more precisely, any library that uses functions from libm, but note that is hard to determine because nothing in the toolchain will complain when you forget it on a library that uses function from libm, so you'll only get missing symbols at the link of the final executable).

[domenkozar]

Anyone remembers why we didn't enable gold by default?

[nh2]

@domenkozar No, I don't know. I know why we aren't using lld as default: #24744

[adamse]

@domenkozar, @nh2: I think #80466 is the missing piece for gold as the default linker.