Improve template haskell support (upstream fix and workaround for compiler crashes)

[jneira]

Description of the problem (from #469 (comment)):

GHC has its own dynamic linker, in the rts, and it gets used in certain cases:

• when GHC the binary is linked statically, by GHCI and TH
• when ghc the library is linked statically, by TH and the ghc interactive api
  Implementing a dynamic linker with support for >=3 platforms is a risky and complex exercise. Fortunately there is an easy opt-out (-dynamic) but unfortunately that opt-out makes distribution much harder, and the precompiled HLS binaries are statically linked.

---

Hot to check if you hls binary is using the ghc dynamic linker (from #1982 (comment)):

/path/to/haskell-language-server  +RTS --info | grep "RTS way" | grep dyn && echo "Dynamic" || echo "Static

---

• We have plenty of serious bugs involving the use of template haskell: https://github.com/haskell/haskell-language-server/issues?q=is%3Aissue+is%3Aopen++label%3A%22type%3A+template+haskell+related%22+
• Surely there would be some more that are not diagnosed
• Surely we cant tacke all of them in a unified way
• But i hope there would be something we can do to improve the handling in a global way
• Maybe we could take advantage of gsoc 2021 to make progress on it although it could not be the more attractive work

@pepeiborra @wz1000 could we sketch what could be the needed changes (mainly in ghcide i suppose) to improve it? is it even a good idea?

[pepeiborra]

I don't use TH so I cannot really help much. My understanding is that the main issue is a bug in the GHC static linker for Mac OS, which causes segfaults. The best way to address this would be to enable using the system linker even when the ghc package is statically linked, lifting the restriction described here. But this is a GHC change and it would need some speccing out and a qualified mentor.

[jneira]

So maybe @wz1000?, what do you think about this possible line of work?

[jneira]

I am gonna remove the label for now, as this dont have an actionable outcome

[jneira]

All issues seems to be related with the ghc dynamic linker so i would label this as blocked upstream on https://gitlab.haskell.org/ghc/ghc/-/issues/19021 (opened in #469) and with the generic workaround of compile hls from source with this config in cabal.project or cabal.project.local

package haskell-language-server
  ghc-options: -dynamic

[jneira]

It seems a way to fix template haskell handling in ghc itself has been opened: see #2000 (comment)
The link in the ghc issue tracker would be https://gitlab.haskell.org/ghc/ghc/-/issues/20628

[jneira]

For completeness, there is an issue in the ghcup about just that workaround: https://gitlab.haskell.org/haskell/ghcup-hs/-/issues/245
Dont think it will have direct support, as it can be done as we commented above relatively easy with --patch or --cabal-project-local

[jneira]

About make hls prebuilt executables dinamically linked:

It is true that hls built dynamically linked provides a good workaround for a bad hls/ghc interaction, but it is a workaround after all. The proper fix should be done in ghc, to make possible evaluate th code in a reliable way independently on how the executable requesting that evaluation (hls in our case) is linked.

The relevant upstream issues where imho the definitive fix should be done are:

• https://gitlab.haskell.org/ghc/ghc/-/issues/20628
• https://gitlab.haskell.org/ghc/ghc/-/issues/19021

So i am a little bit reluctant to change our ci to make it produce dynamically linked binaries, having those ci changes and the executables built that way its own problems and issues and being the goal provide a temporary workaround.

Related with documentation we have added recently an entry in the troubleshooting section: https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#problems-with-template-haskell and https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#problems-with-dynamic-linking

Feel free to suggest more docs changes to make easier indentify the problem and find the workaround.

[hasufell]

(the first is more straightforward and i suppose they are equivalent)

They are not. The first applies -dynamic to all packages (and will likely rebuild the entire world), the second only to haskell-language-server.

[jneira]

@adamse is working in a ghc version which can improve th support.

• details here: Document why hls binaries are statically linked (and track ghc changes to allow loading dynamic libs within a static executable) #2000
• relevant ghc docs link about: https://gitlab.haskell.org/ghc/ghc/-/wikis/dynamic-linking-debate#do-we-need-the-static-linker-anyway
• relevant ghc issue: https://gitlab.haskell.org/ghc/ghc/-/issues/20628

I've collected some reproducer cases for th issues, posting them here as well:

Revising the issues labeled with template haskell related i've found some:

• Segfault when running hls with TH and libpq #1982 (comment)
• Template Haskell + HLS = Unexpected usage error ld-linux-x86-64.so.2: cannot open shared object file #1841 (point to a repro repo: https://github.com/FinleyMcIlwaine/hls-issue)
• Crash Loop in VSCode Extension using Template Haskell #1297 (points to https://github.com/FinleyMcIlwaine/hls-issue)
• SIGSEGV (address boundary error) in Mac OS when encountering Template Haskell splice #469 (this one seemed to be macos specific, points to https://github.com/patrickt/possession)
• Segmentation fault when encountering a TemplateHaskell splice #800 (comment) (points to https://github.com/afcowie/hello/tree/ghcide-issue-444)
• GHC runtime linker: fatal error: I found a duplicate definition for symbol #1737 (points to https://github.com/serokell/edna)

There will be some others in others th related issues. Sorry i cant verify all those reproducers are up to date and being correct.

[nh2]

Also:

• Linking issue with Template Haskell that depends on C code #365

[nh2]

I tried the workaround #1431 (comment) in my nix environment setting this line:

https://github.com/NixOS/nixpkgs/blob/f366af7a1b3891d9370091ab03150d3a6ee138fa/pkgs/development/tools/haskell/haskell-language-server/withWrapper.nix#L14

to haskell.lib.enableSharedExecutables.

However, in my nix-shell, I got this error:

Consulting the cradle to get project GHC version...
Project GHC version: 8.10.7
haskell-language-server exe candidates: ["haskell-language-server-8.10.7","haskell-language-server"]
Launching haskell-language-server exe at:/nix/store/ri354r8xy2xrlagrj5aljrgawg23dcyb-haskell-language-server-1.4.0.0/bin/haskell-language-server-8.10.7
/nix/store/ri354r8xy2xrlagrj5aljrgawg23dcyb-haskell-language-server-1.4.0.0/bin/haskell-language-server-8.10.7: error while loading shared libraries: libHSCabal-3.2.1.0-ghc8.10.7.so: cannot open shared object file: No such file or directory
haskell-language-server-wrapper: callProcess: /nix/store/ri354r8xy2xrlagrj5aljrgawg23dcyb-haskell-language-server-1.4.0.0/bin/haskell-language-server-8.10.7 (exit 127): failed

So the error is libHSCabal-3.2.1.0-ghc8.10.7.so: cannot open shared object file: No such file or directory.

Reason it's in ldd twice, and the first reference is not found:

% ldd /nix/store/ri354r8xy2xrlagrj5aljrgawg23dcyb-haskell-language-server-1.4.0.0/bin/haskell-language-server-8.10.7 | grep HSCabal
	libHSCabal-3.2.1.0-ghc8.10.7.so => not found
	libHSCabal-3.2.1.0-ghc8.10.7.so => /nix/store/673ic4dsw35wmqwq59z821l67fjfix8w-ghc-8.10.7/lib/ghc-8.10.7/Cabal-3.2.1.0/libHSCabal-3.2.1.0-ghc8.10.7.so (0x00007f1c98094000)

Anybody happen to know what's up here?

---

Very ugly workaround:

Given that all these .so are there twice (once not found, once found), I just grepped for all of the not found one, took the paths from their found equivalents, and put them in an LD_LIBRARY_PATH in my shell.

This, together with #365 (comment), apparently makes HLS work completely for my larger project that uses a lot of inline-c.

[jneira]

I am gonna close all th issues about compiler crashes but this and #2000 as all of them seems to have the same root cause

[jfraudeau]

It looks like this was fixed in release 1.7.0.0 by #2675 and #2431