GHC 9.4.5 x86_64, linker error during static build with stack

[paulcadman]

Firstly, thanks for maintaining this project. I wondered if you'd be able to help debug a build issue when using the GHC 9.4.5 container, or suggest something we could try.

We're trying to use ghc-musl to produce static builds for the juvix project but are experiencing a linker issue with the build.

We're using ghc-musl:9.4.5 on x86_64.

The build works with a standard (i.e distributed by GHC) distribution of GHC 9.4.5.

Reproduction steps

• I used the following script to do the build build.sh.gz
  • The stack command is stack install --allow-different-user --system-ghc --ghc-options='-static -split-sections -optl-static'
  • We link against a static runtime library that we build using clang
• I launched the container using docker run -v $PWD/build.sh:/build.sh --rm -ti glcr.b-data.ch/ghc/ghc-musl:9.4.5 /bin/bash
• The resulting transcript of the build session: typescript.gz (after gzip -d this is best viewed using less because of recorded ANSI escapes).

Linker error during build

/usr/lib/gcc/x86_64-alpine-linux-musl/12.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/lib/gcc/x86_64-alpine-linux-musl/12.2.1/crtbe
ginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/usr/lib/gcc/x86_64-alpine-linux-musl/12.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status

<no location info>: error:
    `gcc' failed in phase `Linker'. (Exit code: 1)

Upstream issue?

I found this issue which is related https://gitlab.haskell.org/ghc/ghc/-/issues/20168. The solution given there is to configure GHC to use its internal linker (used by ghci). However this does not seem like an ideal solution, we should use the system linker for this (as noted in the GHC documentation, the internal linker has known bugs, and they'd like to remove it).

[benz0li]

Cross reference: utdemir/ghc-musl#28

[benz0li]

Maybe --ghc-options='-optl-fuse-ld=gold -static -split-sections -optl-static' helps.

[benz0li]

@paulcadman See also commercialhaskell/stack#3420

[paulcadman]

Maybe --ghc-options='-optl-fuse-ld=gold -static -split-sections -optl-static' helps.

Thanks for the suggestion, I get the same linker error with these options.

[benz0li]

Do you have all the required options set according to commercialhaskell/stack#3420 (comment)?

[paulcadman]

Do you have all the required options set according to commercialhaskell/stack#3420 (comment)?

Not exactly these options, but we have set of options to produce the static builds successfully with GHC 9.2.7 on alpine: https://github.com/anoma/juvix/blob/e0aadef2988d96804b3a65de741c5cbcccc5ab9a/.github/workflows/linux-static-binary.yaml#L38

[benz0li]

Do you have all the required options set according to commercialhaskell/stack#3420 (comment)?

Not exactly these options, but we have set of options to produce the static builds successfully with GHC 9.2.7 on alpine: https://github.com/anoma/juvix/blob/e0aadef2988d96804b3a65de741c5cbcccc5ab9a/.github/workflows/linux-static-binary.yaml#L38

Cross references:

• https://stackoverflow.com/questions/41419102/haskell-stack-static-binary-relocation-r-x86-64-32-against-tmc-end-can-not/41427067#41427067
• https://www.fpcomplete.com/blog/2016/10/static-compilation-with-stack/

(I assume the latter link is where the crtbeginT hack is coming from)

Unfortunately, I can not help any further.

[paulcadman]

Unfortunately, I can not help any further.

Thanks very much for your help, much appreciated! 🥇

[benz0li]

@paulcadman Please try with the following:

build.sh

#!/usr/bin/env sh

set -x

# required to build the juvix runtime
apk add --update clang14

# install stack
curl https://github.com/commercialhaskell/stack/releases/download/v2.11.1/stack-2.11.1-linux-$(uname -m)-static.tar.gz -OL
tar xf stack-2.11.1-linux-$(uname -m)-static.tar.gz
cp stack-2.11.1-linux-$(uname -m)-static/stack /usr/local/bin

# stack permissions bug workaround
chown -R $(id -un):$(id -gn) ~

# clone and build Juvix
git clone --recursive --branch stack-lts-21.6 https://github.com/anoma/juvix.git
cd juvix
make runtime LIBTOOL=llvm14-ar
echo 'ld-options: -static -pthread' >> package.yaml
stack install --allow-different-user --system-ghc --ghc-options='-split-sections'

[benz0li]

For the Linux AArch64 release you need to build a statically linked stack first. See

• Binary for 64-bit (aarch64) Alpine Linux commercialhaskell/stack#6141
  • Provide statically linked binary for Linux/AArch64 commercialhaskell/stack#6142

Cross reference: https://gitlab.haskell.org/ghc/ghc/-/issues/23482

[benz0li]

More references about building a statically linked binary:

1. https://www.reddit.com/r/haskell/comments/9on8gi/is_there_an_easy_way_to_compile_static_binaries/
2. https://vadosware.io/post/static-binaries-for-haskell-a-convoluted-approach/
3. https://www.reddit.com/r/haskell/comments/6xuiy7/comment/dmjqode/

ld-options: -static is required for sure. I am not sure about everything else.

[paulcadman]

Thank you - the combination of options you suggest works! 🎉

[benz0li]

Thank you - the combination of options you suggest works! 🎉

Please have this reviewed by a GHC expert – I am not one.

Regarding package.yml:

1. Use ld-options: -static -pthread instead of ld-options: -static?
   • Reference: Link Options (Using the GNU Compiler Collection (GCC))
   • I am surprised that --ghc-options '-optl-static -optl-pthread' does not work.
     ℹ️ This should be equivalent to the ld-options in package.yml
2. Use cc-options: -static?
   • Reference: ? (https://github.com/utdemir/ghc-musl#stack)
   • Equivalent to --ghc-options '-static'?

Regarding --ghc-options:

1. Append -fPIC?
   • Reference: 5.6. Flag reference — Glasgow Haskell Compiler latest User's Guide > 5.6.20. Code generation options
2. Append -static -threaded?
   • Reference: 5.6. Flag reference — Glasgow Haskell Compiler latest User's Guide > 5.6.21. Linking options

(You could append -fllvm in addition to compile using the LLVM code generator.)

Or should all GHC options go into package.yml?

Ping @TravisCardwell (utdemir/ghc-musl) @mpilgrem (commercialhaskell/stack)

[TravisCardwell]

The options can be pretty confusing, IMHO.

Regarding package.yml:

1. Use ld-options: -static -pthread instead of ld-options: -static?

This is what I do.

• I am surprised that --ghc-options '-optl-static -optl-pthread' does not work. This should be equivalent to the ld-options in package.yml

These should indeed be equivalent.

2. Use cc-options: -static?

   • Reference: ? (https://github.com/utdemir/ghc-musl#stack)
   • Equivalent to --ghc-options '-static'?

I do not think that these are equivalent. The GHC -static flag is used to avoid shared Haskell libraries. It is enabled by default, but it does not hurt to set it explicitly. cc-options passes command-line arguments to the C compiler, and I do not think that -static is a recognized flag for GCC compilation. I have never tried setting it.

Regarding --ghc-options:

1. Append -fPIC?

The -fPIC documentation says "generate position-independent code (code that can be put into shared libraries)," which sounds like it is not applicable to static compilation.

2. Append -static -threaded?

The -static option is enabled by default, but it does not hurt to set it explicitly. The -threaded option should be set if a threaded runtime is needed.

Or should all GHC options go into package.yml?

I think that it is often fine to put such configuration in your .cabal file (or package.yml when using Hpack) when you only need to support building with a specific set of build tools. In other cases, you can follow the system-dependent parameters documentation, dynamically configuring flags using Autoconf (or a script).

BTW, I usually put static build configuration behind a Cabal flag, allowing me to develop easily and then confirm that static builds still work when appropriate (after big changes and before committing). Cabal syntax example:

flag static
  description: Build static executable
  default: False

executable example
  ...
  if flag(static)
    ld-options: -static -pthread

[paulcadman]

Thank you @TravisCardwell. I tried setting -optl-static in the ghc-options flag again and got the same error. There are few bugs open in stack about the handling of ghc-options, so it looks like I've hit one of those.

I've used your flags suggestion, the equivalent stack/hpack configuration (in package.yaml) is:

flags:
  static:
    description: Build static executable
    default: false
    manual: true

executables:
  example:
    ...
    when:
      - condition: flag(static)
        ld-options:
          - -pthread
          - -static

You then run stack build --flag example:static to set the flag.