Relocatable packages

[christiaanb]

This patch partially implements Relocatable Packages as discussed in #462. This entails:

• RPATH included in libraries and executables are $ORIGIN (@loader_path on OS X) relative.
• data-files are located relative to the executable based on System.Environment.getExecutablePath.
• package.conf files now contain directory specifications that are ${pkgroot} relative.

It enables the installation of packages that are prefix-relative with regards to dependencies in the same package database. For example, assume a relocatable ghc installation (e.g. http://ghcformacosx.github.io/ for OS X). You can add packages to this relocatable ghc installation by: cabal install --global --enable-relocatable <package>. You can then make a new tar-ball of this installation and deploy it on someone else's machine, and the added libraries and executable will work out-of-the-box.

Another example is the .cabal-sandbox directory created by cabal sandboxes. When you do a cabal install --dependencies-only --enable-relocatable you will get a .cabal-sandbox directory which is relocatable anywhere on the developer machine. That is, all libraries and executables in the .cabal-sandbox directory are prefix-relative amongst eachother, however, they will contain hardcoded paths to the dependencies in the global package database (e.g. the base package). That is what is meant by: prefix-relative with regards to dependencies in the same package database.

data-files are located based on the location returned by System.Environment.getExecutablePath. I have confirmed that this works on my OS X and Linux machines. The current implementation of relocatable packages will not work on operating systems where System.Environment.getExecutablePath does not function correctly.

This patch also puts the handling of RPATH in the hands of Cabal on those systems on which the patch has been tested (OS X and Linux). When --enable-relocatable is specified, the RPATHs will be prefix-relative. Coincidentally, this also fixes #1568 for the tested platforms, even when --disable-relocatable.

This patch has been tested on OS X 10.8, and Linux (CentOS 6.4), using GHC 7.8.3. Relocatable packages are only supported on those platforms, and installation will fail (Cabal will inform the user) if --enable-relocatable is specified for an unsupported platform.

As said, this partially implements #462. To fully implement that issue, we would additionally need a deploy command. Which you would call like: cabal deploy --deploy-dir=<deploy_dir> <package> which would collect the binaries, libraries of the dependencies (in case of a dynamically linked executable), and data-files, and turn them into a relocatable bundle at the specified directory.

[christiaanb]

One (major) flaw in my current implementation is that in handles libraries with data-files poorly. One such example would be pandoc. The reason is that data-files are found relative to the bin-dir. This poses two problems:

1. data-files cannot be located from within ghc binary (ghci, runhaskell, etc.) when the library is installed in:
   • The global database: The ghc binary (which System.Environment.getExecutablePath will resolve to) is located in lib/ghc-<version>/bin, not bin.
   • Any other database: The ghc binary is definitely not prefix relative to any package database other than the global database.
2. data-files cannot be located by a binary installed in a location that is not prefix-relative to the library.

Problem 2. could be solved by wrapper schell scripts, such as the ghc script in ghc's bin directory. But I wouldn't know how to solve problem 1 other than embedding the data files in the library (http://hackage.haskell.org/package/file-embed)

[23Skidoo]

Thanks, will review soon. Can you please use rebase instead of merge to remove spurious merge commits?

[christiaanb]

It seems I have already totally messed up my git history with an incorrect rebase earlier.... as you can see, some messages appear double. Should I just delete my current github repo? and push the changes as one big patch agains master?

[23Skidoo]

Try rebasing against master first. You can combine/remove/reorder patches with rebase -i.

I'd prefer to have several smaller patches instead of a single giant patch, e.g. whitespace changes should be separate from code changes.

[tibbe]

Is this rebased and ready to be merged?

[christiaanb]

It is properly rebased. Whether it is ready to be merged depends on your thoughts regarding my first comment on the problem of data-files in libraries.

[tibbe]

@christiaanb does it make the data files situation worse for users who don't use --enable-relocatable? If not I'm fine with merging it.

[christiaanb]

@tibbe Users who do not use --enable-relocatable are unaffected.

[luite]

data-files locations for libraries are now tracked in the InstalledPackageInfo with the data-dir field (was the first of the GHCJS patches, went in a few weeks ago), that should probably be used to prepare a bundle for deployment. Is this being done yet (commit date looks like it's too old, but it was pushed/rebased more recently)?

[ttuegel]

Somehow the new RPATH handling breaks test suites when building with --enable-executable-dynamic --enable-shared. Test executables cannot find their shared libraries, as seen in this Travis build. @christiaanb would you mind taking a look?

[ttuegel]

I should mention, this RPATH/test suite bug only appears using GHC 7.8.

EDIT: Actually, this probably isn't true, sorry!

[christiaanb]

Indeed, I incorrectly assumed we only needed to setup (DY)LD_LIBRARY_PATH for relocatable packages. I've created a fix at: #2289

[domenkozar]

@christiaanb I'm working if it's possible to disable this behavior? I was hoping --disable-relocatable would do the job.

We're having problems in Nix due to this behavior for two reasons:

1. DYLD_LIBRARY_PATH is not respected on OSX subshells due to security restrictions

2. C dependency library has RPATH as relative:

$ otool -L /private/var/folders/v0/ydgx93x96375ws14_16p93gr0000gq/T/nix-build-rocksdb-haskell-0.1.0.drv-0/rocksdb-haskell-0.1.0/dist/build/Database/RocksDB/C_hsc_make tabase/Roc
/private/var/folders/v0/ydgx93x96375ws14_16p93gr0000gq/T/nix-build-rocksdb-haskell-0.1.0.drv-0/rocksdb-haskell-0.1.0/dist/build/Database/RocksDB/C_hsc_make:
        librocksdb.4.13.dylib (compatibility version 0.0.0, current version 0.0.0)
        /nix/store/wngzdrk0b53q24b5infy1g2j4d5mh1mx-libiconv-osx-10.11.6/lib/libiconv.dylib (compatibility version 7.0.0, current version 7.0.0)
        /nix/store/f0m22bm6x14fq68jxnzw2w9dca83jfk7-gmp-6.1.1/lib/libgmp.10.dylib (compatibility version 14.0.0, current version 14.1.0)
        /nix/store/qsq0qzfi0dib713a83adz70scf5m3z1a-Libsystem-osx-10.11.6/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

So in Nix itself, we use the following ugly workaround NixOS/nixpkgs@19e2e72, but I'm hoping to also fix Stack+Nix.

[domenkozar]

See #4183 for further discussion.