Forwarding DYLD_LIBRARY_PATH

[bitemyapp]

https://www.reddit.com/r/haskell/comments/3ooxu4/library_not_loaded_libmariadb2dylib_os_x/

Ah, I got the solution myself. It seems that I needed to do a

export DYLD_LIBRARY_PATH=/usr/local/lib/mariadb/

to add it to the search paths for dylds and it seems stack isn't forwarding env variables like DYLD_LIBRARY_PATH. I noticed because I tried to run the hsc2hs generated executable manually and it worked as expected.

Is this intended behaviour for stack?

[snoyberg]

Just saw this issue after responding on Reddit. I'm not aware of anything Stack is doing to make DYLD_LIBRARY_PATH be handled specially, perhaps it's the OS itself doing something funny?

[alexbiehl]

Spawning children processes of processes restricted by System Integrity Protection, such as by
launching a helper process in a bundle with NSTask or calling the exec(2) command, resets the Mach
special ports of that child process. Any dynamic linker (dyld) environment variables, such as
DYLD_LIBRARY_PATH, are purged when launching protected processes.

Source: https://forums.developer.apple.com/thread/9233

[snoyberg]

Well, that explains that. If someone has an idea for how to resolve this issue, please send a PR, but my lack of OS X experience will prevent me from getting involved.

[borsboom]

Is this specific to Stack, or does the same thing happen with cabal-install? I just wonder if a workaround is even possible, because it would presumably require getting the subprocess to set DYLD_LIBRARY_PATH itself. But if cabal-install doesn't have this problem then it should be fixable in Stack. Could it be related to the fact that ghc is actually a shell script wrapper rather than a binary?

I'm not a Mac developer, just a Un*x developer that happens run OS X on the desktop, so I don't have much understanding of System Integrity Protection at this point. I'm avoiding upgrading to El Capitan because I don't want to break my own development environment, but probably I should bite the bullet so I can look at these issues...

[DemiMarie]

Sounds like the only solution is to avoid going through a shell script wrapper, or distributing a custom shell.

[alexbiehl]

I can have a look at it when I'm back home. The idea is:

I was able to compile with regular libmysqlclient (without setting DYLD_LIBRARY_PATH) maybe it was in PATH or some other variable so that hsc2hs were able to find it nonetheless. The same principle should apply to libmariadb. Maybe then I can find what differed there.

[borsboom]

@alexbiehl Have you had a chance to look into this? I'm also still curious whether this is unique to Stack or whether cabal-install has the same problem.

[YPares]

@borsboom After doing some tests on OSX, I can say it's not specific to stack. cabal-install has the same problem.
When trying to build https://github.com/tweag/HaskellR/tree/master/inline-r for instance:

$ cabal configure --extra-lib-dirs=/nix/store/hjr3jryjq1kgk1lgc7ikwnqh0xwlc5yx-R-3.2.2/lib/R/lib
$ cabal build

it causes the error (during the hsc2hs phase since it says dist/build/Foreign/R/Type_hsc_make failed), whereas (after cleaning)

$ export DYLD_LIBRARY_PATH=/nix/store/hjr3jryjq1kgk1lgc7ikwnqh0xwlc5yx-R-3.2.2/lib/R/lib
$ cabal configure
$ cabal build

works. So it's not stack-specific, cabal does the same.

[mgsloan]

Sounds like we should close this one out and file a bug with Cabal (if one doesn't already exist).

[borsboom]

@YPares: unless I am misunderstanding the issue, it actually looks like this problem is Stack-specific based on your results with cabal-install. As I understand it, Stack doesn't even work if you do set DYLD_LIBRARY_PATH, since El Capitan's System Integrity Protection prevents the environment variable from being passed through. Did you perform your test on El Capitan?

[YPares]

@borsboom Yes, I'm on El Capitan.
And for my part it worked when I set DYLD_LIBRARY_PATH. Maybe I misunderstood the issue then, as @bitemyapp first post seemed to indicate that setting DYLD_LIBRARY_PATH fixed that. (I'm also pretty new to OSX development myself)

[borsboom]

I've upgraded a system to El Capitan and was able to reproduce @bitemyapp's issue. Here's a quick example:

On El Capitan (note the first command outputs an empty line):

$ DYLD_LIBRARY_PATH=/usr/local/lib/mariadb/ stack exec -- sh -c 'echo $DYLD_LIBRARY_PATH'

$ FOO=/usr/local/lib/mariadb/ stack exec -- sh -c 'echo $FOO'
/usr/local/lib/mariadb/

vs. Yosemite (both commands output the path):

$ DYLD_LIBRARY_PATH=/usr/local/lib/mariadb/ stack exec -- sh -c 'echo $DYLD_LIBRARY_PATH'
/usr/local/lib/mariadb/
$ FOO=/usr/local/lib/mariadb/ stack exec -- sh -c 'echo $FOO'
/usr/local/lib/mariadb/

@YPares: can you try both commands on your El Capitan setup? And can you confirm that you didn't disable System Integrity Protection?

[borsboom]

aside: leaving out Stack entirely, just the following exhibits the same behaviour:

DYLD_LIBRARY_PATH=/usr/local/lib/mariadb/ sh -c 'echo $DYLD_LIBRARY_PATH'

[borsboom]

After I disabled System Integrity Protection, El Capitan behaves like Yosemite with respect to DYLD_LIBRARY_PATH. So I think that is going to have to be our recommended course of action to work around this issue.

[borsboom]

I've added a FAQ entry for this and linked to it from the OS X installation instructions.

[kevin-lee]

@borsboom Isn't there any better way than disabling System Integrity Protection? I mean like the one mentioned here.

[sam2000]

Hi, I am a newbie so sorry if this is a silly question. Disabling a security feature like this should be fine as a workaround I think. But would since the issue is now 'closed' does this mean this will not be 'fixed' in a way that would make it work on OSX without having to mess with OS security? Because I am just getting started with haskell and yesod but I am hoping to use a Mac mini as a production server for Yesod eventually and and I am worried I may have to do something like this on the production server to get Yesod to run?

[cbarrett]

Telling users to disable SEP to use Stack is really disappointing. Indeed, as @kevin-lee's link shows, it's only binaries in system locations that aren't allowed to be affected by DYLD_* environment variables. Hopefully Stack can resolve this.

[mgsloan]

Note that this isn't directly a problem with stack, but something that appears to affect all uses of cabal / ghc. While maybe there's a workaround that stack could attempt, the "proper" solution is likely a change outside of stack itself.

[sam2000]

Thanks mgsloan, In that case I guess someone has to report this to the cabal/ghc people I guess.

I have sent the following message to the Haskell libraries mailing list. Please let me know if there is a better place to report this issue.

Hi,
There was a discussion in github about stack (see link below) where the conclusion seems to be pointing towards a bug in Cabal on Mac OSX EL Capitan related to forwarding DYLD_LIBRARY_PATH

#1161 (comment)

At the moment the suggested workaround is to disable System Integrity Protection which sounds a bit scary to me.

So I thought this might be a good idea to reported to someone in a Cabal team.

At the moment I am using an Ubuntu Virtual Machine to avoid having to mess with SIP on my mac. Hope some clever/kind person from the Haskell community would be able to fix this soon.

Thanks

Sam

[cbarrett]

@mgsloan ah, good point. Thank you, I'll keep investigating there.

[sam2000]

@cbarrett Thanks so much for investigating this. Please could you post a link if a bug is registered against cabal or GHC so I can keep track of any developments. I am hoping I don't have to use the VM for too long

[mgsloan]

There is now another discussion about this (see #1799). Mac folks please try to help us come up with a hack that allows things to work without disabling SIP.

@sam2000 Also, you will be able to run binaries produced by haskell. This is just an issue with passing environment variables when invoking a process during the build.

[borsboom]

(copied from #1799 (comment))

I think this may be happening at the GHC level as well. ghc is actually a shell script wrapper, and since /bin/sh is a protected binary DYLD_LIBRARY_PATH will get stripped from the environment at as soon as the compiler is invoked. It may be possible to work around by copying /bin/sh elsewhere and re-writing the shell script wrapper to use the new path. Note: this shell script wrapper is included with GHC, so this would more properly be fixed upstream.

[borsboom]

FYI: I've been doing some more testing, and I can now reproduce the issue with stack, cabal-install, and directly using runghc Setup.lhs.

[borsboom]

And now I have workaround that worked in my case (more-or-less what I mentioned in #1161 (comment)). First I copied /bin/sh to an another location in my home directory, then I edited <GHC-PATH>/bin/hsc2hs (which is a shell script) and replaced the path in #!/bin/sh with the new path. This fixed it for all three cases (stack, cabal-install, and directly using runghc Setup.lhs). I suspect that some of the other wrappers (such as ghc and ghci at least) would also need to be modified to handle loading dynamic libraries in Template Haskell and the REPL, so probably best to do this to all of them.

Potentially Stack could apply this sort of workaround itself. stack setup could copy /bin/sh somewhere into ~/.stack/programs and scan the GHC bin directory for any shell scripts and replace their shebangs to the new PATH. To work around it with a system-installed GHC would be a bit more involved -- it would have to copy the shell scripts elsewhere before rewriting the shebangs and then modify PATH so that the rewritten scripts are preferred over those installed in the system.

[borsboom]

Opened a GHC issue to discuss solutions: https://ghc.haskell.org/trac/ghc/ticket/11617

[bitemyapp]

Thank you @borsboom! All the work to get this sorted is much appreciated 😄

[sam2000]

What wonderful news! I recently wanted to give yesod a try but when I realised that just setting it up required stack which means messing with OSX security stuff I gave up on it! Now what you have done gives me hope! Thank you so much Emanuel Borsboom! You are a star!

Sent from my iPhone

On 20 Feb 2016, at 22:06, Emanuel Borsboom [email protected] wrote:

And now I have workaround that worked in my case (more-or-less what I mentioned in #1161 (comment)). First I copied /bin/sh to an another location in my home directory, then I edited GHC-PATH/bin/hsc2hs (which is a shell script) and replaced the path in #!/bin/sh with the new path. This fixed it for all three cases (stack, cabal-install, and directly using runghc Setup.lhs).

Potentially Stack could apply this sort of workaround itself. stack setup could copy /bin/sh somewhere into ~/.stack/programs and scan the GHC bin directory for any shell scripts and replace their shebangs to the new PATH. To work around it with a system-installed GHC would be a bit more involved -- it would have to copy the shell scripts elsewhere before rewriting the shebangs and then modify PATH so that the rewritten scripts are preferred over those installed in the system.

This all seems pretty doable

—
Reply to this email directly or view it on GitHub.

[borsboom]

@sam2000 Chances are Yesod will work out of the box, without making any changes to SIP, anyway. This is only necessary for the rare case when you have to set DYLD_LIBRARY_PATH, and I had to manually move a some libraries out of the location where Homebrew installs them in order to reproduce this.

[sam2000]

@EmanuelBorsboom in that case I will try this on my Mac. Thanks again!

Sent from my iPhone

On 20 Feb 2016, at 23:40, Emanuel Borsboom [email protected] wrote:

@sam2000 Chances are Yesod will work out of the box, without making any changes to SIP, anyway. This is only necessary for the rare case when you have to set DYLD_LIBRARY_PATH, and I had to manually move a some libraries out of the location where Homebrew installs them in order to reproduce this.

—
Reply to this email directly or view it on GitHub.

[borsboom]

Gershom recently made this comment on the GHC issue:

Having done a little investigating, it seems to me that the real answer is "you shouldn't need to set DYLD_LIBRARY_PATH" . Indeed I don't think there are any normal workflows on OS X that should require this. The issue would only arise when linking to an external lib that inserts a path on DYLD_LIBRARY_PATH, and such libs are now effectively basically broken throughout the OS X ecosystem.

Here are a few articles that recommend against setting it (ever) as taken from a thread discussing this issue on Oracle bindings for node:

​https://blogs.oracle.com/ali/entry/avoiding_ld_library_path_the
​http://linuxmafia.com/faq/Admin/ld-lib-path.html
We may want to add a FAQ page for manual workarounds, but my gut says "don't fix, and recommend to those who encounter this that they should reconsider why they need DYLD_LIBRARY_PATH to begin with".

That said, I'm not one of the people "feeling the pain" here, so opinions may vary.

I've never been bitten by this problem (I've always installed libraries using Homebrew, which puts them in /usr/local/lib where they belong), so I can't really offer any particular opinion there. If you do have this problem, and you disagree with Gershom's assessment, please comment on the GHC issue.

Any ideas for workarounds that don't require DYLD_LIBRARY_PATH also appreciated. For example, maybe we can document an easy way to specify the location of libraries via the command-line rather than via the environment variable.

[borsboom]

Since the discussion here seems to have stalled without any feedback from anyone still having problems, I'm closing this issue. We can re-open later if necessary.

[dtaskoff]

Is it possible for stack to copy the sh from which sh and modify the scripts in stack path | grep compiler-bin | cut -d' ' -f2 to use the copied executable?
Doing that manually solves the issue.

[dtaskoff]

To save some people that manual work, I'm providing the script I'm using.

NOTE: the script must be ran from a stack project or you can pass --stack-yaml <path-to-stack-yaml> to it (so that it can find the appropriate directory containing the GHC scripts for that project).

If you're okay with that behaviour, I can try to implement it in the stack setup step for macOSes.

[mboes]

FTR this breaks inline-java. The reason is: the jni package uses hsc2hs. Cabal seems to think hsc2hs should link against libjvm.so if the jni library will, but it doesn't set an -rpath. So the intermediate binary produced by hsc2hs fails to run because it can't find libjvm.so. The reason this hits us now and did not do so previously is because Homebrew now installs all JVM's in /Library/Java/JavaVirtualMachines, which is a directory protected by SIP, whereas /usr/local/opt, which is where things previously were, is not.

DYLD_LIBRARY_PATH would be a workaround to the -rpath issue, except that with SIP that env var becomes pretty useless. :-/

[ocramz]

I can reproduce this behaviour with stack 2.9.1 on MacOS Big Sur (11.5.2).

1. build with stack build --ghc-options "-L<dir> -l<library-name>" (dynlib file is in dir)
2. stack exec ... crashes with "Image not found" ❌
3. Running the built program manually with ./.stack-work/... and the DYLD_LIBRARY_PATH variable set to the absolute path of the directory where the dynlib is successful ✅

Thank you to @alexbiehl for finding the root cause and @bitemyapp for showing a fix.