Some cross-compiled armv7l programs can't get stdout and stderr, causing segfaults

[ju1m]

Describe the bug

On a cross-compiled armv7l NixOS system (eg. running on a Cubieboard2), most programs work fine (eg. date, ls, htop, ...), but some crash with a SIGSEGV (my current list is: gcc, zpool, zfs, zed) apparently because they're given NULL for stdout and stderr.
I've no idea what is causing that, neither whether it's specific to ARM nor if cross-compiling or more generally NixOS is at fault there.

Steps To Reproduce

$ uname -a
Linux carotte 5.14.14 #1-NixOS SMP Wed Oct 20 09:57:59 UTC 2021 armv7l GNU/Linux

$ date
Sat Oct 30 04:10:52 CEST 2021
$ gcc
Segmentation fault (core dumped)
$ zpool
Segmentation fault (core dumped)

Expected behavior

A clear and concise description of what you expected to happen.
gcc and zpool should output some messages and quit normally like date does.

Additional context

date

$ file $(readlink -e $(which date))
/nix/store/y4k95bkj8xd6x75w3n3h7kx52q0hxxjd-coreutils-armv7l-unknown-linux-gnueabihf-8.32/bin/coreutils: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknod-linux-armhf.so.3, for GNU/Linux 2.6.32, not stripped

$ gdb $(which date)
Reading symbols from /run/current-system/sw/bin/date...
(No debugging symbols found in /run/current-system/sw/bin/date)
(gdb) b _start
Breakpoint 1 at 0x16c20
(gdb) b main
Breakpoint 2 at 0x16958
(gdb) r
Starting program: /nix/store/ycj2nw41y2182c6pj61764zmw6lrqz3b-system-path/bin/date 

Breakpoint 1, 0xb6fc8900 in _start () from /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3
(gdb) p stdin
No symbol "stdin" in current context.
(gdb) p stdout
No symbol "stdout" in current context.
(gdb) p stderr
No symbol "stderr" in current context.
(gdb) c
Continuing.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libthread_db.so.1".

Breakpoint 1, 0x00016c20 in _start ()
(gdb) p stdin
$1 = (FILE *) 0xb6ca2460 <_IO_2_1_stdin_>
(gdb) p stdout
$2 = (FILE *) 0xb6ca2b48 <_IO_2_1_stdout_>
(gdb) p stderr
$3 = (FILE *) 0xb6ca2aa8 <_IO_2_1_stderr_>
(gdb) c
Continuing.

Breakpoint 2, 0x00016958 in main ()
(gdb) p stdin
$4 = (FILE *) 0xb6ca2460 <_IO_2_1_stdin_>
(gdb) p stdout
$5 = (FILE *) 0xb6ca2b48 <_IO_2_1_stdout_>
(gdb) p stderr
$6 = (FILE *) 0xb6ca2aa8 <_IO_2_1_stderr_>

$ lddtree $(which date)
    date => /run/current-system/sw/bin/date (interpreter => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3)
        librt.so.1 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/librt.so.1
        libacl.so.1 => /nix/store/rihlmncnr7ndbv2iqymhphybwlakqf1m-acl-2.3.1-armv7l-unknown-linux-gnueabihf/lib/libacl.so.1
        libattr.so.1 => /nix/store/j9vmxw3p9ddvqara9nr8qx2ayg3lhyln-attr-armv7l-unknown-linux-gnueabihf-2.5.1/lib/libattr.so.1
        libgmp.so.10 => /nix/store/lpzkdjhdpxbcl5qc5xaildisd1w6gy05-gmp-6.2.1-armv7l-unknown-linux-gnueabihf/lib/libgmp.so.10
        libcrypto.so.1.1 => /nix/store/r09pkk21s5x7ps09wijwcgqvq1f0nxxk-openssl-armv7l-unknown-linux-gnueabihf-1.1.1l/lib/libcrypto.so.1.1
            libdl.so.2 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libdl.so.2
        libpthread.so.0 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libpthread.so.0
        libc.so.6 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libc.so.6
        ld-linux-armhf.so.3 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3

gcc

$ file /nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnueabihf-10.3.0/bin/gcc
/nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnueabihf-10.3.0/bin/gcc: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknld-linux-armhf.so.3, for GNU/Linux 2.6.32, with debug_info, not stripped

Core was generated by `/nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnue'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __GI___fileno (fp=0x0) at fileno.c:35
35      fileno.c: No such file or directory.
(gdb) bt
#0  __GI___fileno (fp=0x0) at fileno.c:35
#1  0x00039cbc in diagnostic_set_caret_max_width (context=context@entry=0x1371d0 <global_diagnostic_context>, value=<optimized out>) at ../../gcc-10.3.0/gcc/diagnostic.c:138
#2  0x00039d64 in diagnostic_initialize (context=0x1371d0 <global_diagnostic_context>, n_opts=n_opts@entry=0) at ../../gcc-10.3.0/gcc/diagnostic.c:187
#3  0x0001f60c in driver::global_initializations (this=this@entry=0xbef00da4) at ../../gcc-10.3.0/gcc/gcc.c:7472
#4  0x00013418 in driver::main (this=0xbef00da4, this@entry=0xbef00d9c, argc=<optimized out>, argc@entry=10, argv=<optimized out>, argv@entry=0xbef00f24) at ../../gcc-10.3.0/gcc/gcc.c:7391
#5  0x000138c0 in main (argc=10, argv=0xbef00f24) at ../../gcc-10.3.0/gcc/gcc-main.c:47
(gdb) p stdin
$1 = (FILE *) 0xb6f3d460 <_IO_2_1_stdin_>
(gdb) p stdout
$2 = (FILE *) 0x0
(gdb) p stderr
$3 = (FILE *) 0x0

(gdb) p _IO_2_1_stdin_ 
$1 = {file = {_flags = -72507256, _IO_read_ptr = 0x0, _IO_read_end = 0x0, _IO_read_base = 0x0, _IO_write_base = 0x0, _IO_write_ptr = 0x0, _IO_write_end = 0x0, _IO_buf_base = 0x0, _IO_buf_end = 0x0, _IO_save_base = 0x0, _IO_backup_base = 0x0, _IO_save_end 
    _chain = 0x0, _fileno = 0, _flags2 = 0, _old_offset = -1, _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "", _lock = 0xb6f3e4a4 <_IO_stdfile_0_lock>, _offset = -1, _codecvt = 0x0, _wide_data = 0xb6f3d500 <_IO_wide_data_0>, _freeres_list = 0x0
    __pad5 = 0, _mode = 0, _unused2 = '\000' <repeats 39 times>}, vtable = 0xb6f3bad0 <__GI__IO_file_jumps>}
(gdb) p _IO_2_1_stdout_
$2 = {file = {_flags = -72540028, _IO_read_ptr = 0x0, _IO_read_end = 0x0, _IO_read_base = 0x0, _IO_write_base = 0x0, _IO_write_ptr = 0x0, _IO_write_end = 0x0, _IO_buf_base = 0x0, _IO_buf_end = 0x0, _IO_save_base = 0x0, _IO_backup_base = 0x0, _IO_save_end 
    _chain = 0xb6f3d460 <_IO_2_1_stdin_>, _fileno = 1, _flags2 = 0, _old_offset = -1, _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "", _lock = 0xb6f3e498 <_IO_stdfile_1_lock>, _offset = -1, _codecvt = 0x0, _wide_data = 0xb6f3d3d4 <_IO_wide_data
    _freeres_buf = 0x0, __pad5 = 0, _mode = 0, _unused2 = '\000' <repeats 39 times>}, vtable = 0xb6f3bad0 <__GI__IO_file_jumps>}
(gdb) p _IO_2_1_stderr_
$3 = {file = {_flags = -72540026, _IO_read_ptr = 0x0, _IO_read_end = 0x0, _IO_read_base = 0x0, _IO_write_base = 0x0, _IO_write_ptr = 0x0, _IO_write_end = 0x0, _IO_buf_base = 0x0, _IO_buf_end = 0x0, _IO_save_base = 0x0, _IO_backup_base = 0x0, _IO_save_end 
    _chain = 0xb6f3db48 <_IO_2_1_stdout_>, _fileno = 2, _flags2 = 0, _old_offset = -1, _cur_column = 0, _vtable_offset = 0 '\000', _shortbuf = "", _lock = 0xb6f3e48c <_IO_stdfile_2_lock>, _offset = -1, _codecvt = 0x0, _wide_data = 0xb6f3d348 <_IO_wide_dat
    _freeres_buf = 0x0, __pad5 = 0, _mode = 0, _unused2 = '\000' <repeats 39 times>}, vtable = 0xb6f3bad0 <__GI__IO_file_jumps>}

(gdb) b _start
Breakpoint 1 at 0x13998: _start. (2 locations)
(gdb) b main
Breakpoint 2 at 0x133c0: main. (2 locations)
(gdb) r
Starting program: /nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnueabihf-10.3.0/bin/gcc 

Breakpoint 1, 0xb6fc8900 in _start () from /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3
(gdb) p stdin
Missing ELF symbol "stdin".
(gdb) p stdout
$1 = (FILE *) 0x0
(gdb) p stderr
$2 = (FILE *) 0x0
(gdb) c
Continuing.

Breakpoint 1, _start () at ../sysdeps/arm/start.S:79
79      ../sysdeps/arm/start.S: No such file or directory.
(gdb) p stdin
$3 = (FILE *) 0xb6f54460 <_IO_2_1_stdin_>
(gdb) p stdout
$4 = (FILE *) 0x0
(gdb) p stderr
$5 = (FILE *) 0x0
(gdb) c
Continuing.

Breakpoint 2, main (argc=1, argv=0xbeffe904) at ../../gcc-10.3.0/gcc/gcc-main.c:43
43      ../../gcc-10.3.0/gcc/gcc-main.c: No such file or directory.
(gdb) p stdin
$6 = (FILE *) 0xb6f54460 <_IO_2_1_stdin_>
(gdb) p stdout
$7 = (FILE *) 0x0
(gdb) p stderr
$8 = (FILE *) 0x0

$ lddtree /nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnueabihf-10.3.0/bin/gcc
    gcc => /nix/store/xy166xz18727nwp7mi4dyh6n97798i9x-gcc-debug-armv7l-unknown-linux-gnueabihf-10.3.0/bin/gcc (interpreter => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3)
        libm.so.6 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libm.so.6
        libc.so.6 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/libc.so.6
        ld-linux-armhf.so.3 => /nix/store/0mp8a94ka57bf99sdyigpgx3zfhdnw2j-glibc-armv7l-unknown-linux-gnueabihf-2.33-50/lib/ld-linux-armhf.so.3 

zpool

Core was generated by `zpool'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb6a08854 in __vfprintf_internal (s=0x0, format=0x378e8 "missing command\n", format@entry=0x0, ap=..., ap@entry=..., mode_flags=3064966684, mode_flags@entry=0) at vfprintf-internal.c:1359
1359    vfprintf-internal.c: No such file or directory.
(gdb) bt
#0  0xb6a08854 in __vfprintf_internal (s=0x0, format=0x378e8 "missing command\n", format@entry=0x0, ap=..., ap@entry=..., mode_flags=3064966684, mode_flags@entry=0) at vfprintf-internal.c:1359
#1  0xb69f9da4 in __fprintf (stream=<optimized out>, format=0x378e8 "missing command\n") at fprintf.c:32
#2  0x00015ee8 in main ()
(gdb) p stdin
$1 = (FILE *) 0xb6afb460 <_IO_2_1_stdin_>
(gdb) p stdout
$2 = (FILE *) 0x0
(gdb) p stderr
$3 = (FILE *) 0x0

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `0`
 - host os: `Linux 5.14.14, NixOS, 21.11 (Porcupine)`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.5pre20211007_844dd90`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[ju1m]

Note that systemd's coredumpctl may not work on cross-compiled systems without patching the kernel because the cross-compiled Nix store path of systemd-coredump put into kernel.core_pattern overflows CORENAME_MAX_SIZE which allows only 128 chars in kernel.core_pattern.
Note also that lddtree (from pax-utils) is missing a fix because it wants /bin/bash.

[Artturin]

Note also that lddtree (from pax-utils) is missing a fix because it wants /bin/bash.

fixed in c44c82d (#142602)

[stale]

I marked this as stale due to inactivity. → More info

[Majiir]

This is not stale.

I'm still having this issue with zfs. I ran a (somewhat arduous) bisect and found that b3640e0 from #103183 introduced the issue. Maybe @expipiplus1 or @bgamari can help us.

---

To verify the bisect and build at b3640e0 (broken) or its parent 15b3bc3 (working), you will need to fix a few build errors:

git checkout 77c177ebe197a9de3d2a397997d70fa13d12aab4 -- pkgs/development/libraries/fribidi/default.nix
git cherry-pick --no-commit 1a65c5df5f98a63d103977384cdbe12a80379906

The cherry-pick will merge conflict in pkgs/development/interpreters/python/cpython/default.nix. Resolve it by removing this expression:

optionals (isPy37 || isPy38) [
  # Backport a fix for ctypes.util.find_library.
  ./3.7/find_library.patch
]

You can build pkgsCross.armv7l-hf-multiplatform.zfs and run zfs --version (or seemingly any other zfs command) to test. You can add armv7l-linux to boot.binfmt.emulatedSystems to run the binary on another system (like x86).

While the error was introduced with zfs at 0.8.5, the issue persists with zfs at 2.1.6 today.

[Majiir]

Removing the R_ARM_COPY patch fixes cross-compiled zfs on a recent master build. Cross-compiling haskellPackages.hello works, too. I don't know how to more thoroughly test the problem that prompted the patch in the first place.

The bug thread mentioned in the patch has a recent update with a new patch. I tried applying it and rebuilding zfs, but a dependency fails to build with unresolvable R_ARM_ABS32 relocation against symbol.

[Majiir]

It looks like the patch was actually reverted recently (#194253). As of that commit, zfs builds and runs. (zfs builds fail on more recent master commits because of an unrelated samba build issue.)

@ju1m Are there any other binaries still failing for you with nixpkgs at 0deec20?

[abathur]

Closing since I interpret ju1m's 🎉 reaction to the Majiir's last comment as indicating that they are not seeing any more failures here. Happy to reopen if I'm misreading, though!