Build 2.11.1 from source fails on macOS with linker error to symbol iconv

[essandess]

Building version 2.11.1 from source fails on macOS with linker error to symbol iconv. I've tried with and without setting LDFLAGS=-L/usr/lib, and with both Xcode and CLT.

Tested on

macOS 13.4.1 22F82 x86_64
Xcode 14.3.1 14E300c

:info:build Linking .stack-work/dist/x86_64-osx/Cabal-3.6.3.0/build/stack/stack ...
:info:build Undefined symbols for architecture x86_64:
:info:build   "_iconv", referenced from:
:info:build       _hs_iconv in libHSbase-4.16.4.0.a(iconv.o)
:info:build      (maybe you meant: _hs_iconv_close, _base_GHCziIOziEncodingziIconv_iconvEncoding15_bytes , _base_GHCziIOziEncodingziIconv_iconvEncoding7_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding3_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding13_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding5_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding1_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding4_info , _base_GHCziIOziEncodingziIconv_iconvEncoding7_info , _base_GHCziIOziEncodingziIconv_iconvEncoding14_bytes , _base_GHCziIOziEncodingziIconv_iconvEncoding9_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding8_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding10_bytes , _base_GHCziIOziEncodingziIconv_iconvEncoding_info , _base_GHCziIOziEncodingziIconv_iconvEncoding9_info , _hs_iconv , _base_GHCziIOziEncodingziIconv_iconvEncoding12_info , _base_GHCziIOziEncodingziIconv_iconvEncoding13_info , _base_GHCziIOziEncodingziIconv_iconvEncoding1_info , _base_GHCziIOziEncodingziIconv_iconvEncoding8_info , _base_GHCziIOziEncodingziIconv_iconvEncoding11_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding2_info , _base_GHCziIOziEncodingziIconv_iconvEncoding11_info , _base_GHCziIOziEncodingziIconv_iconvEncoding16_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding6_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding2_closure , _hs_iconv_open , _base_GHCziIOziEncodingziIconv_iconvEncoding6_info , _base_GHCziIOziEncodingziIconv_iconvEncoding16_info , _base_GHCziIOziEncodingziIconv_iconvEncoding12_closure , _base_GHCziIOziEncodingziIconv_iconvEncoding4_closure )
:info:build   "_iconv_close", referenced from:
:info:build       _hs_iconv_close in libHSbase-4.16.4.0.a(iconv.o)
:info:build      (maybe you meant: _hs_iconv_close)
:info:build   "_iconv_open", referenced from:
:info:build       _hs_iconv_open in libHSbase-4.16.4.0.a(iconv.o)
:info:build      (maybe you meant: _hs_iconv_open)
:info:build ld: symbol(s) not found for architecture x86_64
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)
:info:build `clang' failed in phase `Linker'. (Exit code: 1)

Full log:
main.log.gz

[mpilgrem]

@essandess, thanks for reporting. That is a puzzle, because Stack 2.11.1 was built, and the master branch version of Stack is built, on the GitHub hosted runner for macOS 12/x86_64 (macos-latest) and I am pretty sure (I will check again) that it builds on macOS 13/Aarch64 - I don't have local access to macOS 13/x86_64. I understand your build command is stack build --with-gcc /usr/bin/clang --allow-different-user.

On the symbol _iconv on macOS, I found this (old) item on Stack Overflow (no relation): https://stackoverflow.com/questions/57734434/libiconv-or-iconv-undefined-symbol-on-mac-osx - which may give some hints.

EDIT: Your problem is on x86_64, but I have no problem on macOS 13.3.1(a)/Aarch64 with:

git checkout v2.11.1
stack build

EDIT2: There is a GitHub hosted runner for macos-13 in beta. I will try building with that.

[mpilgrem]

@essandess, I tried building with the GitHub hosted macOS 13.4/x86_64 runner at #6182 and had no problems - so I think we can rule out 'macOS 13 versus macOS 12'. My hypothesis is that there is something up with your build environment, perhaps along the lines of the Stack Overflow post that I identified.

[essandess]

This is an issue with stack version 2.11.1, not a local build environment issue, because:

• This is the standard MacPorts environment that has worked to build stack for years—I am trying to update MacPorts to the latest stack version.
• I confirm that building stack version 2.9.1 works using the exact same local MacPorts environment for which version 2.11.1 fails to build.

Therefore, this is a stack issue because something has changed with stack version 2.11.1 that prevents linking with macOS system iconv symbols.

Would you please post a link to the exact commands and environment variables that work to build stack version 2.11.1 on macOS?

[mpilgrem]

@essandess, you mention Stack 2.9.1 to Stack 2.11.1 - what about Stack 2.9.3 (to try to narrow down possibly relevant changes )?

Locally, I can build Stack 2.11.1 on macOS 13.4.1/AArch64 with just command stack build. I think my macOS set up is 'plain vanilla' - it is, essentially, a 'fresh out of the box' Mac mini which I don't use much except to test Stack locally on macOS.

EDIT: In case it is relevant, if I command which -a iconv on my local machine it returns /usr/bin/iconv and iconv --version yields:

iconv (GNU libiconv 1.11)
Copyright (C) 2000-2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Bruno Haible.

EDIT 2: The GitHub runner environment for macos-13 yields the same information about the location and version of iconv.

In terms of what the Stack CI does, it is set out in .github/workflows/integration-tests.yaml in the job integration-tests. That, in turn, uses the Haskell script etc/scripts/release.hs, with command stack etc/scripts/release.hs build in whatever environment the GitHub macos-latest runner provides (the job step named Build bindist).

I am pretty sure that stack release.hs build, on macOS, just executes:

stack --arch=<arch> --interleaved-output  --local-bin-path=<output_dir> install --flag stack:-developer-mode --flag=stack:integration-tests --flag=stack:hide-dependency-versions --flag=stack:supported-build --pedantic stack

where <arch> is derived from Distribution.System.buildPlatform. See about line 275 and releaseBinDir </> binaryName </> stackExeFileName %> \out -> do ....

That command boils down to stack install stack - as none of those flags do anything interesting.

There is no manipulation of environment variables before executing those commands.

I've looked at git diff v2.9.3 v2.11.1 .github/workflows/integration-tests.yml and git diff v2.9.3 v2.11.1 -- etc/scripts/release.hs. Neither have any substantive changes between Stack 2.9.3 and Stack 2.11.1. The latter moved from --resolver nightly-2022-11-14 (GHC 9.2.5) to --resolver lts-20.13 (GHC 9.2.7).

I also looked at git diff v2.9.3 v2.11.1 stack.yaml and I can't see anything untoward there:

diff --git a/stack.yaml b/stack.yaml
index b90b30f4e..66e9b79b7 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -1,14 +1,27 @@
-# GHC 9.2.5
-resolver: lts-20.0
+# GHC 9.2.7
+resolver: lts-20.21

 packages:
 - .

 extra-deps:
-- hpack-0.35.1@sha256:ef816234cbc7b52b0a6c55f7e904b6bc5292b8dd8f2d81ffcbcbc69ab80d75e5,4762
-- pantry-0.8.1@sha256:196111414d2489499fda6213deebcb865bc12285023d5af9bd273bf27cdb185d,4099
-- rio-prettyprint-0.1.4.0@sha256:1f8eb3ead0ef33d3736d53e1de5e9b2c91a0c207cdca23321bd74c401e85f23a,1301
+- ansi-terminal-1.0@sha256:640ffecfd95471388d939fcacb57bdc0cef15f0457746c234a12cdd5a6c6d1e8,2706
+- ansi-terminal-types-0.11.5@sha256:f78440dfd95c4509e88855ac7cc2d9566ddf956a92c1290404cac93ad1a1b00a,1482
 - fsnotify-0.4.1.0@sha256:44540beabea36aeeef930aa4d5f28091d431904bc9923b6ac4d358831c651235,2854
+- pantry-0.8.2.2@sha256:579aa8538c0fde65f9c08fb97d1d5aee8f59e5cc44e5f8feb350ec54bd2b14a6,6026
+- persistent-2.14.5.0@sha256:c3c7a6a200930f956b2a6bb15b9d2cd512980692f6a2d95368a6db335c34c916,7199
+- rio-prettyprint-0.1.4.0@sha256:1f8eb3ead0ef33d3736d53e1de5e9b2c91a0c207cdca23321bd74c401e85f23a,1301
+# lts-20.21 specifies Cabal-3.6.3.0
+- Cabal-3.8.1.0@sha256:155d64beeecbae2b19e5d67844532494af88bc8795d4db4146a0c29296f59967,12220
+# lts-20.21 specifies Cabal-syntax-3.6.0.0
+- Cabal-syntax-3.8.1.0@sha256:4936765e9a7a8ecbf8fdbe9067f6d972bc0299220063abb2632a9950af64b966,7619
+
+allow-newer: true
+
+allow-newer-deps:
+# Required because ansi-wl-pprint-0.6.9 specifies ansi-terminal < 0.12. See:
+# https://github.com/ekmett/ansi-wl-pprint/issues/29
+- ansi-wl-pprint

 drop-packages:
 # See https://github.com/commercialhaskell/stack/pull/4712
@@ -16,7 +29,7 @@ drop-packages:

 docker:
   enable: false
-  repo: fpco/alpine-haskell-stack:9.2.5
+  repo: fpco/alpine-haskell-stack:9.2.7

 nix:
   # --nix on the command-line to enable.
@@ -25,6 +38,10 @@ nix:
   - unzip

 flags:
+  hackage-security:
+    Cabal-syntax: true
+  mintty:
+    win32-2-13-1: false
   stack:
     developer-mode: true

[mpilgrem]

Does this 2015 issue discussion provide any hints? #825

[turboMaCk]

I'm not using Mac myself but colleague of mine run into similar issues when building package that depends on base 4.17.2.0.

My two cents (no solution yet).

The problem with the libiconv supplied by Apple is that its symbols do not start with _lib. So instead of _libiconv_open you get _iconv_open.

So if linker on your system will see any libiconv other than the one from apple it will break because symbols it will be looking for won't exist in the dylib.

So the problem you see is likely because there is probably a newer libiconv somewhere on your system which linker tries to link to. On a clean install this won't be an issue because there will be just the one from apple.

In order to find solution for this problem I'm still missing few pieces. I have no idea where apple's supplied dylib is. Some older resources point to standard /usr/lib but sampling machines of my colleagues neither of them has it there on newer MacOS even those who don't experience these issues. Second is to analyze environment in which stack builds the code to see what does it actually pick up. Sadly I'm not macos user and have very limited experience with stack as well.

when it comes to stack this or that I think the problem is just indirect based on what version of base which version of stack depends? At least I think that might be the case.

edit: I also now found this which might be relevant and supports my claims above https://stackoverflow.com/questions/57734434/libiconv-or-iconv-undefined-symbol-on-mac-osx

UPDATE: We resolved it. In our specific case it was anaconda installation putting different version of libiconv into library path. It seems that apple version of the iconv comes from XCODE. So it's really just a question of making sure env is configured so that linker will use the version from xcode.

[kencu]

The build issue happens when pkgconfig asks for the parameters for zlib, and a non-system zlib and a non-system libiconv are installed in the same prefix.

The zlib pkg-config file winds up giving a library search path that looks like this:

-L/my/prefix/lib

which is just fine for zlib, but that line makes it's way onto the final link line, and the non-system libiconv.dylib is found instead of the (intended) system libiconv.

And because of the way libiconv is managed upstream, they have different symbol names (even if they might be ABI compatible) and so the link fails.

The solutions would be:

1. ghc could (possibly?) statically link libiconv.a during it's build, so the downloaded bootstrap ghc compiler has no external references to any system-installed libiconv.dylb to mess things up.
2. the build instructions could say that if the build system tries to use an external zlib installation, and if that zlib installation also has a libiconv.dylib in the same prefix and library directory, then the build will fail and there is no solution to this issue.

As a workaround, I hacked a pkgconfig zlib.pc file that redirected the build to use /usr as the prefix, and forced that to the front of the pkgconfig search path. That prevented the library path containing libiconv.dylib from being on the link line, and so the build succeeds.

libiconv seems unlikely to ever change their symbol name plan -- it's been like this for years and years.

Is it possible to link libiconv.a statically into ghc? Apple has made that kind of thing harder with their new obfuscated library plan, but hopefully there is still a way...