Clang link time optimization for darwin

[dipinhora]

Motivation for this change

Clang link time optimization was not working on OSX.
Resolves #19098.

Trying to follow CONTRIBUTING.md (hopefully I didn't make too many mistakes):
cc @copumpkin (for cctool/darwin stdenv changes)
cc @lovek323 @raskin @viric (for clang 3.6/llvm 3.7/clang 3.7/clang 3.8/clang 3.9 changes)
cc @fpletz @globin (for cc-wrapper changes)

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • OS X
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

NOTE: Not all tests were successful but I don't believe the issues are caused by any changes in this PR but I am not sure because I didn't do a baseline of the tests without the changes first.

The following is are the results of the testing combinations as represented by checkboxes above:

Technical details

• System: Mac OSX 10.10.5; Output of uname -a:

Darwin Dipins-MBP.home 14.5.0 Darwin Kernel Version 14.5.0: Wed Jul 29 02:26:53 PDT 2015; root:xnu-2782.40.9~1/RELEASE_X86_64 x86_64

• Nix version: nix-env (Nix) 1.11.4
• Nixpkgs version: "17.03pre93030.548bcd0"

mac:

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed
works fine

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox true
error:

building path(s) ‘/nix/store/y9vg140f50a2zq4qsdpdavdmlrlhjjpf-clang-wrapper-9.9.9’
error: derivation ‘/nix/store/5jplsyd68iaxw6d1vrvfl0n2xqpvzh8k-clang-wrapper-9.9.9.drv’ specifies a sandbox profile, but this is only allowed when ‘build-use-sandbox’ is ‘relaxed’

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox relaxed
error: (known issue: NixOS/nix#759)

building path(s) ‘/nix/store/y9vg140f50a2zq4qsdpdavdmlrlhjjpf-clang-wrapper-9.9.9’
killing process 26411
killing process 26411: Operation not permitted
note: keeping build directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-clang-wrapper-9.9.9.drv-1’
error: while setting up the build environment: changing into ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-clang-wrapper-9.9.9.drv-1’: No such file or directory

NIX_PATH=nixpkgs=/Users/dipinhora/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed
error:

note: keeping build directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-python3.5-pytest-2.9.2.drv-0’
error: opening directory ‘/private/var/folders/tw/867wzcgs2gg1zwkwn0ws8_k80000gn/T/nix-build-python3.5-pytest-2.9.2.drv-0/pytest-of-dipinhora/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
/Users/dipinhora/.nix-profile/bin/nix-shell: failed to build all dependencies

When run a second time:

Traceback (most recent call last):
  File "/nix/store/q9n9wsafqp93rcv31wy1hd0l6dpg45gi-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/bkq3sl3kyz1rki2dv6n1wm7xs824x2s2-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 64, in _verify_python3_env
    stderr=subprocess.PIPE).communicate()[0]
  File "/nix/store/wb7zc6hicj05kgncqkrwiv7z9r9mj21d-python3-3.5.2/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/nix/store/wb7zc6hicj05kgncqkrwiv7z9r9mj21d-python3-3.5.2/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'locale'

nixos: (without and with nix.useSandbox = true) (on aws: ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20160907.1 (ami-2ef48339))

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8
works fine

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8 --option build-use-sandbox true
works fine

NIX_PATH=nixpkgs=/root/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed --max-jobs 8 --cores 8
error:

Traceback (most recent call last):
  File "/nix/store/2gaxi7l0z59gq7zya7xqi1gkynp9yy3i-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.

Additional information: on this system no suitable UTF-8
locales were discovered.  This most likely requires resolving
by reconfiguring the locale system.

ubuntu: (on aws: nixos-16.09.666.3738950-x86_64-hvm-ebs (ami-93347684))

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --max-jobs 8 --cores 8
works fine

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox true
error:

building path(s) ‘/nix/store/g7a2yvmgs3ffy91dwppsaifrvgwz41xr-bootstrap-gcc-wrapper’
note: keeping build directory ‘/tmp/nix-build-bootstrap-gcc-wrapper.drv-3’
error: cannot change ownership of ‘/nix/store/vjirhdirwq2k84jxsds0473ik8gn1n00-bootstrap-gcc-wrapper.drv.chroot’: Operation not permitted

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p llvmPackages_38.clang --pure --keep-failed --option build-use-sandbox relaxed
error:

building path(s) ‘/nix/store/g7a2yvmgs3ffy91dwppsaifrvgwz41xr-bootstrap-gcc-wrapper’
note: keeping build directory ‘/tmp/nix-build-bootstrap-gcc-wrapper.drv-2’
error: cannot change ownership of ‘/nix/store/vjirhdirwq2k84jxsds0473ik8gn1n00-bootstrap-gcc-wrapper.drv.chroot’: Operation not permitted

NIX_PATH=nixpkgs=/home/ubuntu/nixpkgs/ nix-shell -p nox --run "nox-review wip" --pure --keep-failed --max-jobs 8 --cores 8
error:

note: keeping build directory ‘/tmp/nix-build-python3.5-pytest-2.9.2.drv-0’
error: opening directory ‘/tmp/nix-build-python3.5-pytest-2.9.2.drv-0/pytest-of-ubuntu/pytest-0/testdir/test_cache_failure_warns0/.cache’: Permission denied
/home/ubuntu/.nix-profile/bin/nix-shell: failed to build all dependencies

When run a second time:

Traceback (most recent call last):
  File "/nix/store/2gaxi7l0z59gq7zya7xqi1gkynp9yy3i-nox-0.0.4/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/nix/store/pqb3555sq1anvfp1ax293xk9dw29f2vm-python3.5-click-6.6/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.

Additional information: on this system no suitable UTF-8
locales were discovered.  This most likely requires resolving
by reconfiguring the locale system.

---

[mention-bot]

@dipinhora, thanks for your PR! By analyzing the history of the files in this pull request, we identified @vcunat, @copumpkin and @edolstra to be potential reviewers.

[dipinhora]

Please let me know if you have any questions regarding this change or if anything needs to be altered before it can be merged.

[dipinhora]

@copumpkin Seeing as this changes cctools and impacts the darwin stdenv, do you have any questions or concerns regarding any of the changes?

If not, what is the process for getting this merged? Does this need to go into staging first?

[copumpkin]

Yeah sorry for not being in touch. My plan was to merge this into staging this weekend with some other changes I was going to make, but staging is currently broken on Darwin so now I'm looking into that. I think it's generally fine though, thanks!

On Oct 16, 2016, at 15:04, Dipin Hora [email protected] wrote:

@copumpkin Seeing as this changes cctools and impacts the darwin stdenv, do you have any questions or concerns regarding any of the changes?

If not, what is the process for getting this merged? Does this need to go into staging first?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[copumpkin]

So I think that in principle it's fine, but in practice it complicates the stdenv bootstrap a bit. I know you made the changes needed to the bootstrap, but it leads to 3 complete builds of LLVM per stdenv build instead of 1 before this change. I'm hoping to make some changes to make it easier to tie together bootstraps, but for now I'd rather not merge at least the LLVM 3.7 changes because it'll make it a nightmare for me to rebuild the stdenv for the other changes I'm making (since LLVM builds are by far the majority of the stdenv build time this makes the whole thing a lot slower)

[dipinhora]

@copumpkin Thanks for reviewing the changes. Much appreciated.

I had noticed that 3 complete builds of llvm/clang were done but didn't do a before/after comparison so didn't realize it was due to my changes.

The LLVM 3.7 changes at https://github.com/NixOS/nixpkgs/pull/19312/files#diff-cf6aa7f087ebdb6ee5061fdbfad9375aR31 and https://github.com/NixOS/nixpkgs/pull/19312/files#diff-cf6aa7f087ebdb6ee5061fdbfad9375aR41 that set the compatibility version are unfortunately required to make it all work because it's not possible to change the compatibility version that ld looks for using install_name_tool. And without the change ld will only load the libLTO.dylib files with a compatibility version of 3.7. This wouldn't be an issue if the stdenv was using clang 3.8 because llvm 3.8 correctly sets the compatibility version for libLTO.dylib to 1.

One way to avoid the llvm 3.7 changes would be if there was some way to change the compatibility version for libLTO.dylib directly in the ld binary generated (similar to how install_name_tool is used to rename the library that ld references from libLTO.3.7.dylib to libLTO.dylib). Is there some way to accomplish that?

[copumpkin]

Interesting. One thing I'd done a few weeks ago was upgrade the darwin stdenv to LLVM 3.8 but some strange things started happening and I didn't have time to figure them out so I undid it and reverted to 3.7.

I don't really know what the best solution is here. My inclination would be to ask you to wait until we get the stdenv to 3.8 and/or the stdenv bootstrapping machinery rework (so we can minimize the LLVM rebuilds) but I don't know how urgently you need this. It doesn't seem like the LLVM stuff changes all that often so it doesn't feel like your PR will bitrot very quickly.

How about in the next few days I start a Hydra build (against one of my custom branches) of a 3.8-based stdenv, and then we can see how that affects this? I'm fearful of putting 3.8 straight into staging because I want to have a full Hydra build to compare to see what we're breaking by upgrading.

[dipinhora]

No rush. I'm currently using a fork of nixpkgs with the changes I need so it's not a blocker for me. I just wanted it in upstream so there's less reason for me to keep a fork and have to build pretty much everything from source instead of being able to rely on the Hydra caches.

I think your plan to try a build of a 3.8 based stdenv and to switch over to it is a good one. Let me know if I can help in converting over to a 3.8 based stdenv.

[vcunat]

I'd assume that LTO is also broken on Linux and that it would need a (DY)LD_LIBRARY_PATH change in this PR.

[dipinhora]

@vcunat No, LTO works on Linux. See: PR #19457 for a derivation that uses it.

[FRidh]

What is the status of this PR?

[LnL7]

I'll let some things build using this and see how it looks.

[LnL7]

The stdenvBootstrapTools.x86_64-darwin.test fails with this change.

http://hydra.nixos.org/eval/1320568?compare=trunk

dyld: Library not loaded: @rpath/libLTO.dylib
  Referenced from: /nix/store/c2rh6b2f28d0r4kvsfpc8sbi6b80wgm0-bootstrap-tools/bin/ld
  Reason: image not found
/nix/store/zj9i54l7cyz676y5h4fds10ngwipf3r0-unpack-bootstrap-tools.sh: line 45: 54128 Trace/BPT trap: 5       ld -macosx_version_min 10.7 -arch x86_64 -dylib -o $out/lib/libSystem.B.dylib -compatibility_version 1.0 -current_version 1226.10.1 -reexport_library $out/lib/system/libsystem_c.dylib -reexport_library $out/lib/system/libsystem_kernel.dylib $args
builder for ‘/nix/store/w8g3w6v3m4599xafa0z5vjv54bkiz66g-bootstrap-tools.drv’ failed with exit code 133

[copumpkin]

I've now merged an upgrade of the stdenv to 4.0, and plan to get LTO into that soon. Sorry it's taken me so long!

[dtzWill]

Beep boop!

[mmahut]

Are there any updates on this pull request, please?

[lilyball]

I'm hitting what I think is this problem. I'd love to see an update on this PR.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-in-distress/3604/7

[stale]

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the
   related code and @ mention them in a comment.
2. Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
   irc.freenode.net.

[shlevy]

Still important to me, see 31579c6

[Luflosi]

There is a different PR (#46730), now merged, which claims to make LTO work on Darwin. There is even a comment pinging the original author of this PR. This suggests, that it's a replacement of this PR. But now, two years later, LTO is broken again and has been for at least one and a half years, see for example the workaround in #56740.

[stale]

I marked this as stale due to inactivity. → More info

[AndersonTorres]

Closing as dead