Skip to content
This repository has been archived by the owner on Jan 16, 2025. It is now read-only.

gcc: sandbox within series, not minor release #34303

Closed
wants to merge 3 commits into from

Conversation

mistydemeo
Copy link
Member

Further to the discussion in #33948 - this removes --enable-version-specific-runtime-libs from GCC, and instead sandboxes libraries and headers into locations versioned by series instead of minor release.

@mistydemeo mistydemeo force-pushed the gcc branch 2 times, most recently from 365e607 to 55fb1dc Compare November 19, 2014 00:40
@mistydemeo mistydemeo changed the title gcc: sandbox within libexec gcc: sandbox within series, not minor release Nov 19, 2014
@mistydemeo
Copy link
Member Author

It ignored the passed --includedir, but the headers that are installed to include are privateish and sandboxed per verison anyway:

/usr/local/Cellar/gcc/4.9.2_1/include/c++/4.9.2/

Full paths here: https://gist.github.com/mistydemeo/8a66cc983f3086ac8a80

The libraries are installed to stable locations, e.g. /usr/local/lib/gcc/4.9/libgfortran.dylib, which ensures that they won't break on point releases, and that the libraries can be provided by gcc49 after the gcc formula moves to a new major series.

cc @tdsmith, @staticfloat, @tkelman for feedback

@@ -25,6 +25,7 @@ def osmajor
url "http://ftpmirror.gnu.org/gcc/gcc-4.9.2/gcc-4.9.2.tar.bz2"
mirror "ftp://gcc.gnu.org/pub/gcc/releases/gcc-4.9.2/gcc-4.9.2.tar.bz2"
sha1 "79dbcb09f44232822460d80b033c962c0237c6d8"
revision 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to revision everything built against gfortran too I guess.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep - and make sure to give enough notice to outside taps that they can rebuild/revision their stuff before this goes in.

@MikeMcQuaid
Copy link
Member

Looks good to me 👍

@mistydemeo
Copy link
Member Author

Building Fortran code with gfortran outside of Homebrew produces these install names:

$ otool -L ./test
./test:
    /usr/local/lib/gcc/4.9/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/lib/gcc/4.9/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

@ianml
Copy link
Contributor

ianml commented Nov 19, 2014

Looks good. Would this be spun out to the versioned gcc formulae? Here's the linkage with R:

/usr/local/lib/libR.dylib:
    /usr/local/opt/r/R.framework/Versions/3.1/Resources/lib/libR.dylib (compatibility version 3.1.0, current version 3.1.2)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    /usr/local/lib/gcc/4.9/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/local/lib/gcc/4.9/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.2.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1151.16.0)
    /usr/local/opt/readline/lib/libreadline.6.dylib (compatibility version 6.0.0, current version 6.3.0)
    /usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 53.1.0)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)

@mistydemeo
Copy link
Member Author

I don't anticipate new releases in any of the pre-4.9 GCC series, so I'd rather err on the side of leaving them as is - that way we don't break any software users have already built just in the name of consistency. I'll apply the same change to gcc49, but will probably only apply the changes to 4.8 and earlier if they get new releases.

@tkelman
Copy link
Contributor

tkelman commented Nov 19, 2014

Thanks for working on this (what's that about Santa Claus?), but I'll have to defer to @staticfloat, if he has time to test this out, to say whether this improves matters for JuliaLang bottling. Definitely looks promising.

Is gcc_s in your Fortran test linking to /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/libgcc_s.1.dylib a potential manifestation of the same problem?

@mistydemeo
Copy link
Member Author

Is gcc_s in your Fortran test linking to /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/libgcc_s.1.dylib a potential manifestation of the same problem?

That's normal for software built outside of Homebrew; the compiler picks the full path rather than the symlinked path. Inside Homebrew the install names are rewritten post-build, so that would point to /usr/local/lib/gcc/4.9/libgcc_s.1.dylib.

@tkelman
Copy link
Contributor

tkelman commented Nov 19, 2014

Ah, gotcha, missed that distinction. So this should fix the bottle problem, but doesn't address the need to rebuild Fortran sources after gcc updates for users who compile things (outside Homebrew) from source with Homebrew's gfortran? That's probably the safe thing to do anyway, but it's a persistent cause of issue filings that we can try to mitigate with better docs.

@jacknagel
Copy link
Contributor

@mistydemeo this troubles me:

Building Fortran code with gfortran outside of Homebrew produces these install names:

$ otool -L ./test
./test:
    /usr/local/lib/gcc/4.9/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/lib/gcc/4.9/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

Compare the linkage to libgfortran.3.dylib and libquadmath.0.dylib vs. the linkage to libgcc_s.1.dylib. I wonder why that one still has a linkage into the Cellar?

@mistydemeo
Copy link
Member Author

I'm confused too - I'm not sure why that happens. I noticed it occurs with both gfortran and g++.

@mistydemeo
Copy link
Member Author

Here's the verbose output from gfortran -o test test.o -v:

Driving: gfortran -mmacosx-version-min=10.9.4 -o test test.o -v -l gfortran -shared-libgcc
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/4.9.2_1/libexec/gcc/x86_64-apple-darwin13.4.0/4.9.2/lto-wrapper
Target: x86_64-apple-darwin13.4.0
Configured with: ../configure --build=x86_64-apple-darwin13.4.0 --prefix=/usr/local/Cellar/gcc/4.9.2_1 --bindir=/usr/local/Cellar/gcc/4.9.2_1/bin --datarootdir=/usr/local/Cellar/gcc/4.9.2_1/share --libdir=/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9 --includedir=/usr/local/Cellar/gcc/4.9.2_1/include/gcc/4.9 --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-4.9 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-cloog=/usr/local/opt/cloog --with-isl=/usr/local/opt/isl --with-system-zlib --enable-libstdcxx-time=yes --enable-stage1-checking --enable-checking=release --enable-lto --disable-werror --with-pkgversion='Homebrew gcc 4.9.2_1' --with-bugurl=https://github.com/Homebrew/homebrew/issues --enable-plugin --disable-nls --enable-multilib
Thread model: posix
gcc version 4.9.2 (Homebrew gcc 4.9.2_1)
Reading specs from /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/../../../libgfortran.spec
rename spec lib to liborig
COLLECT_GCC_OPTIONS='-mmacosx-version-min=10.9.4' '-o' 'test' '-v' '-shared-libgcc' '-mtune=core2'
COMPILER_PATH=/usr/local/Cellar/gcc/4.9.2_1/libexec/gcc/x86_64-apple-darwin13.4.0/4.9.2/:/usr/local/Cellar/gcc/4.9.2_1/libexec/gcc/x86_64-apple-darwin13.4.0/4.9.2/:/usr/local/Cellar/gcc/4.9.2_1/libexec/gcc/x86_64-apple-darwin13.4.0/:/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/:/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/
LIBRARY_PATH=/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/:/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/../../../:/usr/lib/
COLLECT_GCC_OPTIONS='-mmacosx-version-min=10.9.4' '-o' 'test' '-v' '-shared-libgcc' '-mtune=core2'
 /usr/local/Cellar/gcc/4.9.2_1/libexec/gcc/x86_64-apple-darwin13.4.0/4.9.2/collect2 -dynamic -arch x86_64 -macosx_version_min 10.9.4 -weak_reference_mismatches non-weak -o test -L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2 -L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/../../.. test.o -lgfortran -no_compact_unwind -lSystem -lgcc_ext.10.5 -lgcc -lquadmath -lm -lgcc_ext.10.5 -lgcc -lSystem -v
collect2 version 4.9.2
/usr/bin/ld -dynamic -arch x86_64 -macosx_version_min 10.9.4 -weak_reference_mismatches non-weak -o test -L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2 -L/usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2/../../.. test.o -lgfortran -no_compact_unwind -lSystem -lgcc_ext.10.5 -lgcc -lquadmath -lm -lgcc_ext.10.5 -lgcc -lSystem -v
@(#)PROGRAM:ld  PROJECT:ld64-241.9
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7m armv7em
Library search paths:
    /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9/gcc/x86_64-apple-darwin13.4.0/4.9.2
    /usr/local/Cellar/gcc/4.9.2_1/lib/gcc/4.9
    /usr/lib
    /usr/local/lib
Framework search paths:
    /Library/Frameworks/
    /System/Library/Frameworks/
 /usr/local/bin/gnm -n test.o

@mistydemeo
Copy link
Member Author

The current GCC formula has more or less the same behaviour:

# after rebuilding test
$ otool -L ./test
./test:
    /usr/local/lib/gcc/x86_64-apple-darwin13.4.0/4.9.2/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin13.4.0/4.9.2/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/lib/gcc/x86_64-apple-darwin13.4.0/4.9.2/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

@jacknagel
Copy link
Contributor

$ otool -L /usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib
/usr/local/Cellar/gcc/4.9.2/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib:
    /usr/local/lib/gcc/x86_64-apple-darwin14.0.0/4.9.2/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

The dylib ID is set correctly (second line of the above output), so I'm stumped.

@jacknagel
Copy link
Contributor

I think I figured it out, I need to compile the whole thing with the install name machinery turned off though. Will report back.

@staticfloat
Copy link
Contributor

I'm super busy right now so I don't have time to test in any detail. But
this looks like it would solve our bottling issues.
On Nov 19, 2014 3:54 PM, "Jack Nagel" [email protected] wrote:

I think I figured it out, I need to compile the whole thing with the
install name machinery turned off though. Will report back.


Reply to this email directly or view it on GitHub
#34303 (comment).

@jacknagel
Copy link
Contributor

With this PR + the following patch:

diff --git a/Library/Formula/gcc.rb b/Library/Formula/gcc.rb
index ae1f08d..05866b8 100644
--- a/Library/Formula/gcc.rb
+++ b/Library/Formula/gcc.rb
@@ -133,6 +133,8 @@ def install
       args << "--enable-multilib"
     end

+    inreplace "libgcc/config/t-slibgcc-darwin", "@shlib_slibdir@", "#{HOMEBREW_PREFIX}/lib/gcc/#{version_suffix}"
+
     mkdir "build" do
       unless MacOS::CLT.installed?
         # For Xcode-only systems, we need to tell the sysroot path.

I get:

$ otool -L a.out
a.out:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /usr/local/lib/gcc/4.9/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

@mistydemeo
Copy link
Member Author

Added and updated configure args.

@mistydemeo
Copy link
Member Author

Can confirm it works now:

$ otool -L ./test
./test:
    /usr/local/lib/gcc/4.9/libgfortran.3.dylib (compatibility version 4.0.0, current version 4.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    /usr/local/lib/gcc/4.9/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/lib/gcc/4.9/libquadmath.0.dylib (compatibility version 1.0.0, current version 1.0.0)

@@ -132,6 +130,10 @@ def install
args << "--enable-multilib"
end

# Ensure correct install names when linking against libgcc_s;
# see discussion in https://github.com/Homebrew/homebrew/pull/34303
inreplace "libgcc/config/t-slibgcc-darwin", "@shlib_slibdir@", "#{HOMEBREW_PREFIX}/lib/gcc/#{version_suffix}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to submit this upstream too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jacknagel can answer that, but I think this is an edge case in how we package it / our symlink structure rather than an upstream issue per se.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh, I think it would still be handy to let upstream at least know that this is something we'd like to configure.

@staticfloat
Copy link
Contributor

Misty's terminal output above is making me giggle like a preteen. This looks great.

@mistydemeo
Copy link
Member Author

All right, if this looks good let's set a schedule for getting this in. I'd like all the external taps to have advance notice to revision their formulae and rebuild bottles so they're ready to push at the same time we merge this. How long do you think you need?

@staticfloat
Copy link
Contributor

So the proposed plan is that I merge this branch locally, build my bottles and revision my formulae, and push alongside this merge into mainline?

@mistydemeo
Copy link
Member Author

Yep! You can run cd $(brew --prefix) && brew pull 34303 --bottle to get access to this GCC bottle for yourself to build bottles with, then git reset --hard origin/master in /usr/local after you're done.

@staticfloat
Copy link
Contributor

Great. I totally forgot about brew pull! I'll try to get to this tonight, and will let you know when I've got things built.

@mistydemeo
Copy link
Member Author

@tdsmith How much time do you think homebrew-science and homebrew-python will need to get ready? There are no bottles, but there are formula revisions to do.

@MikeMcQuaid Is it safe to do formula revisions for things that need fortran in this PR? Will they be rebuilt using the new GCC if their commits come afterwards?

@mistydemeo
Copy link
Member Author

I know it's a pain, but can you build fftw from source for this? (brew install fftw -s) I'm currently blocked on getting the bottles built due to one of the deps of something in this branch.

@MikeMcQuaid
Copy link
Member

so I think this is proximally a build bot issue; Homebrew's python should be installed by the build bot sooner.

It may well be. I'm trying to fix bot issues but I'll need help from you folks to do so. Remember that anything the bot does can be tested locally with brew test-bot so I'll need at least a (relatively) minimal test case to sort stuff out but ideally a patch to the bot itself (as it doesn't really scale for me to fix all issues with it myself).

@mistydemeo
Copy link
Member Author

I can repro with pyside as the only argument to brew test-bot.

@jacknagel
Copy link
Contributor

I can't reproduce it using pyside, but I can with binwalk. I don't think it's a bot failure, as I can also reproduce it with brew install binwalk. Most likely a dependency resolution bug.

@mistydemeo
Copy link
Member Author

Python dependency is ending up in the list of binwalk's recursive requirements twice, once as optional and once as required.

@mistydemeo
Copy link
Member Author

Jack and I were discussing the binwalk issue, and it looks like it's happening here in expand_requirements.

The :recommended Python dep makes it into expand_requirements, but is then pruned out because build.without?(req) evaluates to true - since binwalk itself does not have a :python requirement, only its dep pyside. I'm... quite surprised we haven't run into this before, actually.

Jack's planning to take a look at this over the next few days.

@mistydemeo
Copy link
Member Author

@tdsmith Out of curiosity - given that binwalk uses pyside, which does have a :python dep, does it make sense to you for binwalk to not have one and use only the system python instead? Should they be matching?

@tdsmith
Copy link
Contributor

tdsmith commented Dec 12, 2014

Definitely maybe; our Python and Apple's oughta be binary compatible so as long as libraries don't explicitly link to libpython (they shouldn't) it shouldn't be a problem. Any problems should manifest as segfaults on import. If this has been working it might be a clue that we don't need a :python dep on pyside.

We shouldn't need to resource numpy and scipy on OS X >= 10.9 if we use system Python since Apple ships 'em. (Doesn't solve this problem on 10.8 and earlier, alas.)

@mistydemeo
Copy link
Member Author

I'm going to hold off on merging this until binwalk can be revisioned.

(Doesn't solve this problem on 10.8 and earlier, alas.)

Yep, we still have to build the 10.8 bottle... :(

@staticfloat
Copy link
Contributor

@mistydemeo When I try to do the brew pull 34303 --bottle on my OSX 10.8 mac mini, I get a merge error, is this normal?

$ git log -1
commit ddf1387eda3dd7e9fefffa3820d044ef83689bc8
Author: Jack Nagel <[email protected]>
Date:   Fri Dec 12 22:27:34 2014 -0500

    Formula#install has public visibility

    It is called with an explicit receiver in build.rb, so the base class
    method should also be public.
$ git status
On branch master
nothing to commit, working directory clean
$ brew pull 34303 --bottle
######################################################################## 100.0%
==> Applying patch
Applying: gcc: sandbox within series release number only
Applying: Revision formulae that use gfortran for GCC bump
Using index info to reconstruct a base tree...
A       Library/Formula/itsol.rb
A       Library/Formula/salt.rb
Falling back to patching base and 3-way merge...
CONFLICT (modify/delete): Library/Formula/salt.rb deleted in HEAD and modified in Revisio$
 formulae that use gfortran for GCC bump. Version Revision formulae that use gfortran for
GCC bump of Library/Formula/salt.rb left in tree.
CONFLICT (modify/delete): Library/Formula/itsol.rb deleted in HEAD and modified in Revisi$
n formulae that use gfortran for GCC bump. Version Revision formulae that use gfortran fo$
 GCC bump of Library/Formula/itsol.rb left in tree.
Failed to merge in the changes.
Patch failed at 0002 Revision formulae that use gfortran for GCC bump
The copy of the patch that failed is found in:
   /usr/local/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Unstaged changes after reset:
M       Library/Formula/binwalk.rb
M       Library/Formula/fftw.rb
M       Library/Formula/grib-api.rb
M       Library/Formula/jags.rb
M       Library/Formula/libgetdata.rb
M       Library/Formula/libxc.rb
M       Library/Formula/mpich2.rb
M       Library/Formula/pgplot.rb
M       Library/Formula/qd.rb
Error: Patch failed to apply: aborted.

@tdsmith
Copy link
Contributor

tdsmith commented Dec 13, 2014

Think we can cheat on fixing dependency resolution right now by pulling #34945, which removes the :python dependency from pyside and shiboken (and oughta be worth doing independent of this issue).

@tdsmith
Copy link
Contributor

tdsmith commented Dec 13, 2014

@BrewTestBot test this please

staticfloat referenced this pull request in mistydemeo/homebrew Dec 13, 2014
@mistydemeo
Copy link
Member Author

This needs a rebase, which I'd done locally but I've held off on merging to avoid starting the bot on a build. Pushing now that @tdsmith has pushed a workaround for binwalk.

@mistydemeo
Copy link
Member Author

@staticfloat Once the bottles are built, I'll wait on merging this until you're ready - sorry for the issues.

@staticfloat
Copy link
Contributor

Yep, no worries. Ping me when the bottles are built, and I'll try building the Julia bottle once more. It should only take me ~20 minutes to build the Julia bottle.

@mistydemeo
Copy link
Member Author

Bottle build is finished. Let me know when you're ready!

@staticfloat
Copy link
Contributor

Great! FFTW bottle is working and Julia bottle is good to go!

Roger Houston, we are ready for liftoff.

@staticfloat
Copy link
Contributor

Had to come in here again and just reiterate how great it's going to be to have this new functionality. It's going to cut down on OSX user issues by a measurable fraction. Thank you so much for this everybody!

@mistydemeo
Copy link
Member Author

Great! I'll merge this when I wake up tomorrow.

@mistydemeo
Copy link
Member Author

Merged!

@mistydemeo mistydemeo deleted the gcc branch December 14, 2014 19:19
@Homebrew Homebrew locked and limited conversation to collaborators Feb 17, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants