Syntax crates are broken on OSX Homebrew builds

[sfackler]

A minimal-ish example:

#[feature(macro_registrar)];
#[crate_id = "procmacro"];
#[crate_type = "rlib"];
#[crate_type = "dylib"];

extern mod syntax;

use syntax::ast;
use syntax::ast::Name;
use syntax::codemap::{Span};
use syntax::ext::base;
use syntax::ext::base::{SyntaxExtension, BasicMacroExpander, NormalTT, ExtCtxt};
use syntax::parse::token;

#[macro_registrar]
pub fn macro_registrar(register: |Name, SyntaxExtension|) {
    register(token::intern("mymac"),
    NormalTT(~BasicMacroExpander {
        expander: expand_syntax_ext,
        span: None,
    },
    None));
}

pub fn expand_syntax_ext(_: &mut ExtCtxt,
                        sp: Span,
                        _: &[ast::TokenTree]) -> base::MacResult {
  println!("{:?}", sp);
  base::MacResult::dummy_expr()
}

#[feature(phase)];

#[phase(syntax)]
extern mod procmacro;

fn main() {
  mymac!(1 + 1);
}

try.rs:10:3: 10:8 error: macro undefined: 'mymac'
try.rs:10   mymac!(1 + 1);
            ^~~~~
error: aborting due to previous error
fish: Job 1, 'rustc try.rs -L .' terminated by signal SIGSEGV (Address boundary error)

Backtrace:

thread #2: tid = 0x1a5ec3, 0x00000001000a2243 libstd-966edb7e-0.10-pre.dylib`rt::task::__extensions__::run::anon::anon::expr_fn::aq + 211, stop reason = EXC_BAD_ACCESS (code=1, address=0x10e762470)
    frame #0: 0x00000001000a2243 libstd-966edb7e-0.10-pre.dylib`rt::task::__extensions__::run::anon::anon::expr_fn::aq + 211
    frame #1: 0x0000000103402620
    frame #2: 0x00000001000a215f libstd-966edb7e-0.10-pre.dylib`rt::task::__extensions__::run::anon::expr_fn::ap + 79
    frame #3: 0x00000001000a987c libstd-966edb7e-0.10-pre.dylib`rust_try + 12
    frame #4: 0x00000001000a20b1 libstd-966edb7e-0.10-pre.dylib`rt::task::Task::run::hdd180d0b0d7e46390gam::v0.10.pre + 81
    frame #5: 0x00000001005b9b5e libgreen-80d9e76a-0.10-pre.dylib`task::__extensions__::build_start_wrapper::anon::expr_fn::aP + 302

There are a couple of things that are weird here:

• mymac isn't being registered properly. Maybe it's using a different interner than the rest of rustc?
• The segfault is happening way outside of the expansion infrastructure. One possibility is that it's trying to call into the external crate somehow, but that's already been unloaded. Maybe the @-box annihilator?
• This has cropped up in a couple of places, all on OSX, and all via rust installs built by Homebrew. Manually building and installing seems to fix the issue. The only nonstandard thing the Homebrew build does is symlinking all of the libraries and executables in place.

[lilyball]

It seems that macro undefined errors trigger before can't find crate errors, even when the crate is marked with #[phase(syntax)], so I expect that the crate itself is not being found.

[zr40]

I've found that with a Homebrew build, the segfault always happens if this is present:

#[phase(syntax)]
extern mod procmacro;

The segfault happens even when mymac! isn't used at all.

[jakerr]

Probably ignore this as I think it was misguided but I'll leave it here for posterity. Better analysis in my more recent comment.
I tried copying all of the rust symlinked bins and libs from the homebrew Cellar into $prefix/bin and $prefix/lib
It gave me what looks like a better stack trace:

#0  0x000000010d6470f7 in rt::util::abort::h65fe94bb28dd0358aT::v0.10.pre ()
#1  0x0000000102da8000 in ?? ()
#2  0x000000010d6a3c18 in rt::local_ptr::compiled::unsafe_borrow::h93e89e50cebf3c9aak::v0.10.pre ()
#3  0x000000010d64d2c8 in local_data::get_local_map::hc5911f93163a621aaT::v0.10.pre ()
#4  0x000000010ddebd36 in parse::token::get_ident_interner::h9ea3b62d777c8600aN::v0.10.pre ()
#5  0x00000001039c7986 in parse::token::intern::h748d0a32f10e1d16aa::v0.0 ()
#6  0x00000001039c78de in macro_registrar::h416c685e5b4ec32fap::v0.0 ()
#7  0x0000000100b57b3e in ext::expand::expand_view_item::hdfeeaa09d714c000ae::v0.10.pre ()
#8  0x0000000100b4b920 in fold::noop_fold_mod::anon::expr_fn::avaB ()
#9  0x0000000100b4b696 in iter::Iterator::collect::h82a97f15963aa9c5aw::v0.10.pre ()
#10 0x0000000100b4b1f1 in ext::expand::expand_mod_items::h65efd0ff56d77e7eag::v0.10.pre ()
#11 0x0000000100b7217e in ext::expand::expand_crate::h96c9ce886b6bc095aa::v0.10.pre ()
#12 0x00000001015b0902 in driver::driver::phase_2_configure_and_expand::anon::expr_fn::aj ()
#13 0x00000001015b04aa in util::common::time::h9007c3cbc81d0ab1ab::v0.10.pre ()
#14 0x00000001015ae4b9 in driver::driver::phase_2_configure_and_expand::h5c89b9c9305776d8aL::v0.10.pre ()
#15 0x00000001015b80d2 in driver::driver::compile_input::h89f98c15dfe1272caK::v0.10.pre ()
#16 0x00000001015ddb72 in run_compiler::hc5ad2afd27047b94aK::v0.10.pre ()
#17 0x00000001015ea58d in main_args::anon::expr_fn::aP ()
#18 0x00000001015e8bbe in monitor::anon::expr_fn::aE ()
#19 0x00000001015e609e in task::TaskBuilder::try::anon::expr_fn::y6aKag ()
#20 0x00000001005be0d0 in task::__extensions__::build_start_wrapper::anon::anon::expr_fn::aX ()
#21 0x00000001000a6468 in rt::task::__extensions__::run::anon::expr_fn::ap ()
#22 0x00000001000ad95c in rust_try ()
#23 0x00000001005bddee in task::__extensions__::build_start_wrapper::anon::expr_fn::aP ()

[jakerr]

I did some more investigation and found that the issue is with not using @rpath in the LC_LOAD_DYLIB when built with the homebrew version. I'm pretty sure it has to do with how the @rpaths are made with respect to CFG_PREFIX in rpath.rs but couldn't quite nail it down.

Running otool -l libprocmacro-1bc91ebe-0.0.dylib once against the lib built with manually-built rustc and once with homebrew-built rust c gives the following difference:

Manually built:

          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libstd-966edb7e-0.10-pre.dylib (offset 24)
   time stamp 2 Thu Jan  1 09:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libsync-7bf3a0fc-0.10-pre.dylib (offset 24)
   time stamp 2 Thu Jan  1 09:00:02 1970
      current version 0.0.0
compatibility version 0.0.0

# etc ...

Load command 19
          cmd LC_RPATH
      cmdsize 144
         path @loader_path/../../../../../../usr/local/rust-manual-build/0.10-pre-8ef25/lib/rustlib/x86_64-apple-darwin/lib (offset 12)
Load command 20
          cmd LC_RPATH
      cmdsize 112
         path /usr/local/rust-manual-build/0.10-pre-8ef25/lib/rustlib/x86_64-apple-darwin/lib (offset 12)

Homebrew built:

Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 112
         name /usr/local/homebrew/lib/rustlib/x86_64-apple-darwin/lib/libstd-966edb7e-0.10-pre.dylib (offset 24)
   time stamp 2 Thu Jan  1 09:00:02 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 112
         name /usr/local/homebrew/lib/rustlib/x86_64-apple-darwin/lib/libsync-7bf3a0fc-0.10-pre.dylib (offset 24)
   time stamp 2 Thu Jan  1 09:00:02 1970
      current version 0.0.0
compatibility version 0.0.0

#etc...

Load command 19
          cmd LC_RPATH
      cmdsize 120
         path @loader_path/../../../../../../usr/local/homebrew/Cellar/rust/HEAD/lib/rustlib/x86_64-apple-darwin/lib (offset 12)
Load command 20
          cmd LC_RPATH
      cmdsize 88
         path /usr/local/homebrew/Cellar/rust/HEAD/lib/rustlib/x86_64-apple-darwin/lib (offset 12)

Notice how the manual built one uses @rpath/ in the LC_LOAD_DYLIB commands while the homebrew-built one uses a absolute path (even though it has set a proper rpath that could be used. See the LC_RPATH block at the bottom.)

I verified that this was indeed the issue by using the library built with homebrew-rustc fixing it up with this command:

install_name_tool -change /usr/local/homebrew/lib/rustlib/x86_64-apple-darwin/lib/libstd-966edb7e-0.10-pre.dylib @rpath/libstd-966edb7e-0.10-pre.dylib libprocmacro-1bc91ebe-0.0.dylib
install_name_tool -change /usr/local/homebrew/lib/rustlib/x86_64-apple-darwin/lib/libsync-7bf3a0fc-0.10-pre.dylib @rpath/libsync-7bf3a0fc-0.10-pre.dylib libprocmacro-1bc91ebe-0.0.dylib

#etc ...

Doing that lets compilation of try.rs succeed using the homebrew version of rustc.

[sfackler]

rpaths strike again! I wonder why the rpaths differ between the homebrew build and the normal one. In any case, the behavior of the unmodified Homebrew build is still bogus. I'll try to figure out what's going on there.

[sfackler]

Adding --disable-rpath to the recipe's configure flags seems to fix the issue. @alexcrichton is that a reasonable thing to push upstream?

[alexcrichton]

Due to favoring static binaries by default, I think it'd be reasonable to upstream it. /usr/local/lib should already be in DYLD_LIBRARY_PATH so a dynamically linked rustc will continue to work automatically. A dynamically linked binary will require some DYLD_LIBRARY_PATH trickery, but that's not too commont (rustdoc builds dynamic exes when testing).

[sfackler]

Homebrew/legacy-homebrew#26686

[thestinger]

The same problems will occur for other projects, so I don't think the build system should override the rustc default. It should be changed for all Rust code if it's going to change for Rust itself.

[jakerr]

I agree with thestinger. Can't we fix the bug in rpath so that it behaves consistently regardless of the build environment? My comment above shows that the homebrew-rustc-built library is fine after manually fixing up the rpaths.

[thestinger]

Rust is the only programming language that I know of setting this kind of thing by default. I think it's just plain wrong to do it at all. It doesn't work at all on Windows so there's an inconsistent experience by default across platforms, and it opens up hidden security holes by default elsewhere.

[jakerr]

I see, I misunderstood you. I think that's fair.
It would be nice to continue to allow some rpath control via flags to rustc to make building .App bundles etc easier.
Then again that concern can probably be handled by a cross-platform packaging utility as long as each platform allows for editing the rpath or equivalent after the fact like OSX does with install_name_tool.

[sfackler]

The Homebrew install should be fixed now.