Don't statically link binaries against glibc/system libraries.

[wz1000]

Currently static HLS binaries for linux are built using --enable-executable-static. This makes cabal build fully static binaries, but statically linking glibc is not a supported configuration by glibc. In addition to this, glibc mismatches between the CI images and user environments can lead to many problems, including:

• Adding one empty import causes "Unexpected usage error" #1687
• HLS failing with Assertion freesize < size failed on a fairly minimal example #2328
• assertion failure from hls and ghc 8.10: `freesize < size' failed #429 as well as all the other similar issues (https://github.com/haskell/haskell-language-server/search?p=1&q=allocatestack&type=issues)

I also believe many of the segfaults reported on TH code have this as the underlying cause rather than bugs with the RTS linker (which do exist, but they aren't responsible for all of the pain).

Instead, we should be using the following configuration for building releases:

executable-dynamic: False

This uses the (vanilla, static) way to build and link Haskell code, but system libraries will still be linked in dynamically.

The difference between the binaries can be observed like so:

$ ldd hls-static-full # built with --enable-execuatable-static
	not a dynamic executable
$ ldd hls-static # built with --disable-executable-dynamic
	linux-vdso.so.1 (0x00007fff50bcf000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f408a70d000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f408a6f3000)
	libtinfo.so.5 => /usr/lib/libtinfo.so.5 (0x00007f408a68f000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f408a684000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f408a67f000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f408a678000)
	libgmp.so.10 => /usr/lib/libgmp.so.10 (0x00007f408a5d6000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f408a40a000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f408a2c6000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f408a768000)

By contrast, a fully dynamic build will link against a bunch of haskell libraries in ~/.cabal/store and dist-newstyle/

Another option is to statically link against musl on an alpine image, but this might still run into problems with loading user code (for TH and plugins) linked against mismatching system libraries. If we do take this route, we will have to bundle a whole GHC distribution (including libraries) with HLS releases, and ensure that hie-bios/cabal/stack uses the bundled GHC distribution to compile all the dependencies of the user's project.

This is made more complicated given that cabal will share the package stores of different compilers as long as they have the same ghc --version.

[jneira]

thanks for the proposal, not an expert in linux but it sounds very reasonable
what could be the drawbacks? those executables would be equally (or at least not worse than fully static ones) compatibles with different linux versions with different glibc/system libraries?

[wz1000]

They will not work on NixOS and similar linux distributions which use non-standard locations for system libraries. But I don't think the static builds worked on them anyway.

Other than that we should usually be able to rely on ABI compatibility for glibc which tends to be decent.

[jneira]

ok, so one plan could be prepare an extra binary for linux for a fixed ghc version (8.10.7?) in ci using those flags and dogfood it/offer it to people hitting the relevant bugs

[wz1000]

Like I said, statically linking glibc is not a configuration that is supported (see https://stackoverflow.com/questions/57476533/why-is-statically-linking-glibc-discouraged for instance), so we shouldn't be distributing any binaries statically linked against glibc.

[jneira]

maybe i am being too cautious here for wanting to do it progresively

[amesgen]

Another option is to statically link against musl on an alpine image, but this might still run into problems with loading user code (for TH and plugins) linked against mismatching system libraries. If we do take this route, we will have to bundle a whole GHC distribution (including libraries) with HLS releases, and ensure that hie-bios/cabal/stack uses the bundled GHC distribution to compile all the dependencies of the user's project.

FTR: I made a naive attempt to build haskell-language-server against musl in #429 (comment) (second part), but it did not work as dynamic loading is not supported.

[bgamari]

maybe i am being too cautious here for wanting to do it progresively

I think so, yes. glibc simply does not support static linking. Trying to statically link against it will result in breakage. Moreover, it will be the worst kind of breakage: breakage of a subtle nature.

If you want to statically link against libc then you need to use an implementation which supports static linking. musl is one such implementation. GHC has been improving its support for musl in 9.2. However, musl does not support dynamic loading in its static configuration. Consequently, HLS will need to revert back to relying on GHC's RTS linker for loading.

Another alternative is to accept dynamic linking against glibc (building against a sufficiently old version to ensure compatibility) but statically link against the remaining libraries. It is technically possible to link statically against gmp, zlib, and libtinfo. Sadly, doing so is not entirely trivial. We currently don't have a great interface for requesting this behavior from GHC or Cabal. Indeed, even the pkg-config and linker support for this is rather half-baked. Sadly, despite how common static linking is today, the tools for achieving it with fine granularity are far from polished.

If I were you, I would just do what the rest of the world does and build on a distribution like Alpine that uses musl. This will greatly simplify things.

[pepeiborra]

Not if it requires relying on the GHC linker for dynamic linking. We need to leave that behind.

[jneira]

ok the plan then will be build Linux binaries with the flags suggested by @wz1000 against the older Ubuntu provided by GitHub vms

[pepeiborra]

Another way to link against musl and use the system linker would be to rely on the external interpreter. Unfortunately that has its own set of issues, as discussed in https://gitlab.haskell.org/ghc/ghc/-/issues/18852

[pepeiborra]

Let's follow Zubin and Ben's recommendation and switch to musl, paired with #2391 to suggest a local build with dynamic linking when TH is detected

[pepeiborra]

Closed via #2463