stdenv: bootstrap riscv64-linux (Step 1) #115406

oxalica · 2021-03-08T12:01:10Z

Motivation for this change

Fix bootstrap packages and bootstrap stdenv on native riscv64-linux platform.

UPDATE 2021.11.27

Rebased and fixed. Currently it doesn't contain bootstrap binary hashes. As suggested by @r-burns #115406 (comment), we should build it on Hydra for trusted binaries.

UPDATE 2021.10.23
Rebased and re-bootstrapped with QEMU VM. Thanks @zhaofengli.

Previous attempt

Files ready to be uploaded to tarballs.nixos.org: (Currently on my cachix https://riscv64.cachix.org)

/nix/store/yffnjz4sny3ygbm59wk69yk56ja0lyk8-busybox
Nar hash: sha256:d67bc5ef5c550bc1e2b86b27fe20b9ffadca42b9d104ae0f27957794be62ac5f
/nix/store/gaa783s4y4w14wyc94lh5pvbpyszbvhc-bootstrap-tools.tar.xz
File hash: sha256:541b51c0dcc4e9fb04652664ff03581c73c16e24198e4030c8ff205fc1114c3d

Steps I did

Cross build bootstrap tools for riscv64-linux, with build platform x86_64-linux.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools-cross.nix riscv64.build -o result-rv64-bootstrap-tools-stage0

Get hashes to fill pkgs/stdenv/linux/bootstrap-files/riscv64.nix.
Then add fixed-output paths into store since our bootstrap file URLs are fake.

nix hash path --type sha256 --base16 ./result-rv64-bootstrap-tools-stage0/on-server/busybox
nix hash file --type sha256 --base16 ./result-rv64-bootstrap-tools-stage0/on-server/bootstrap-tools.tar.xz
# ... hashes to fill

nix store add-path ./result-rv64-bootstrap-tools-stage0/on-server/busybox
nix store add-file ./result-rv64-bootstrap-tools-stage0/on-server/bootstrap-tools.tar.xz

Setup a native riscv64-linux platform.
For me, HiFive-Unleashed is used.
Cross build nix for riscv64-linux, copy closures to the riscv64 platform, and setup /nix and PATH correctly.
I currently don't have any native machine, so I setup an QEMU VM for the bootstrap build.
Note that some packages like coreutils must be built on native machine or QEMU VM, instead of binfmt (QEMU user mode), to pass all tests . See Notes 3 below.
Remote build native bootstrap tools on riscv64-linux with previous cross bootstrap tools.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools.nix build --arg localSystem '{ system = "riscv64-linux"; }' -L --keep-going --builders 'ssh://rv64_machine_or_vm riscv64-linux - 4' -j0 -o result-rv64-bootstrap-tools-stage1

Update bootstrap file hashes to native ones in pkgs/stdenv/linux/bootstrap-files/riscv64.nix.

nix hash path --type sha256 --base16 ./result-rv64-bootstrap-tools-stage1/on-server/busybox
nix hash file --type sha256 --base16 ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz
# ... hashes to fill

nix store add-path ./result-rv64-bootstrap-tools-stage1/on-server/busybox
nix store add-file ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz

Bootstrap again to check if it works. But not update hashes this time.

nix build -f pkgs/stdenv/linux/make-bootstrap-tools.nix build --arg localSystem '{ system = "riscv64-linux"; }' -L --keep-going --builders 'ssh://rv64_machine_or_vm riscv64-linux - 4' -j0 -o result-rv64-bootstrap-tools-stage2

Notes

Bootstrap tools are currently using fake URLs. They should be uploaded first and updated to tarballs.nixos.org before merging this PR.
All intermediate and final store paths are available on https://riscv64.cachix.org.
Using binfmt instead of native riscv64 platform (or QEMU VM) will make few derivations like coreutils fail the checkPhase because of changed argv[0] and emulator behavior. Some tests expects a specific failure like "invalid cwd" from binary, but binfmt qemu fails before it which produces different error message.
~~gcc10 uses PGO and therefore not deterministic (gcc10: build is not deterministic #108475). So bootstrap file hashs may also be not reproducible.~~
libatomic is required for packages linking pthread or using <atomic> (like xz) during the bootstrap. But there is another way to avoid producing libatomic in bootstrap tools, but re-configure these packages to disable multi-threading before new gcc is built (like what we do with perl)
I'm not sure which way is preferred.
I just added libatomic to bootstrap tools.
bootstrap-tools-stage2 differs from bootstrap-tools-stage1. The differences are from glibc and gcc (cc1/cc1plus).
I believe it's caused by that -frandom-seed is based on $out (stdenv: introduce -frandom-seed #102251), which is hashed from all drv inputs. Here between two bootstraps, the bootstrap tools should be functionally identical but have different hash, which changes random-seed of downstream packages.
It should not be an issue though.

diff -r ./result-rv64-bootstrap-tools-stage{1,2}

Binary files ./result-rv64-bootstrap-tools-stage1/on-server/bootstrap-tools.tar.xz and ./result-rv64-bootstrap-tools-stage2/on-server/bootstrap-tools.tar.xz differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/bin/make and ./result-rv64-bootstrap-tools-stage2/pack/bin/make differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/crt1.o and ./result-rv64-bootstrap-tools-stage2/pack/lib/crt1.o differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/ld-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/ld-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/ld-linux-riscv64-lp64d.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/ld-linux-riscv64-lp64d.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libc-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libc-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libcrypt.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libcrypt.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libc.so.6 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libc.so.6 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libdl.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libdl.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libmemusage.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libmemusage.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libm.so.6 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libm.so.6 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnsl-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnsl-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnsl.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnsl.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_compat.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_compat.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_db.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_db.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_dns.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_dns.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_files.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_files.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libnss_hesiod.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libnss_hesiod.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libpthread.so.0 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libpthread.so.0 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libresolv.so.2 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libresolv.so.2 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/librt.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/librt.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil-2.33.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil-2.33.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil.so and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil.so differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/lib/libutil.so.1 and ./result-rv64-bootstrap-tools-stage2/pack/lib/libutil.so.1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1 and ./result-rv64-bootstrap-tools-stage2/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1 differ
Binary files ./result-rv64-bootstrap-tools-stage1/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1plus and ./result-rv64-bootstrap-tools-stage2/pack/libexec/gcc/riscv64-unknown-linux-gnu/10.3.0/cc1plus differ

pkgs/build-support/bintools-wrapper/default.nix

oxalica · 2021-03-09T14:18:25Z

cc @r-burns

zhaofengli · 2021-03-29T21:26:33Z

Any progress on this? Would be great if a trusted member could produce an official bootstrap tools tarball on https://tarballs.nixos.org

pkgs/development/tools/misc/libtool/libtool2.nix

oxalica · 2021-03-31T16:15:55Z

@Ericson2314
What do you think about issue 2 about libatomic from gcc-lib? Currently I simply copy it into bootstrap tools. We can also not touching bootstrap tools, but disable multi-threading pkgs like xz before the new gcc is built (in bootstrap-stage4).

Ericson2314 · 2021-03-31T17:44:52Z

@oxalica I don't mind adding to the the bootstrap tools. Long term, I am hoping the bootstrap tools can just be the tools, statically linked, and we bootstrap all the run-time libraries up from libgcc/compiler-rt. But that is a big refactor, and will start with the better-behaved LLVM stuff first, so I don't care if we add more one libatomic.a` in the meantime.

oxalica · 2021-04-01T16:06:09Z

Rebased and rebuilt the seed.

zhaofengli · 2021-06-03T06:12:58Z

Gentle poke. Now that a basic NixOS system is running on riscv64, I'm really looking forward to seeing this PR merged in the near future. This script starts a simple qemu-system-riscv64 VM and should help with natively building the bootstrap tools for maintainers without access to RISC-V hardware.

zhaofengli · 2021-06-03T22:13:21Z

Hmm, I have a problem. The GCC in stdenv doesn't seem to be able to find spec files:

$ echo "int main() {}" > test.c
$ /nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/gcc -fopenmp test.c
gcc: fatal error: cannot read spec file ‘libgomp.spec’: No such file or directory
compilation terminated.

libgomp.spec is indeed in the lib of the wrapped compiler, and I can link libgomp directly myself:

$ ls /nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib
gcc        libasan_preinit.o  libcc1.a   libgomp.spec       libssp.a            libstdc++.a    libsupc++.a
libasan.a  libatomic.a        libgomp.a  libsanitizer.spec  libssp_nonshared.a  libstdc++fs.a  libubsan.a
$ cat /nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/libgomp.spec
# This spec file is read by gcc when linking.  It is used to specify the
# standard libraries we need in order to link with libgomp.
*link_gomp: -lgomp %{static: -ldl -lpthread }
$ /nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/gcc -lgomp test.c
$

The same commands work fine on an x86-64 host.

zhaofengli · 2021-06-04T18:14:59Z

Still kind of confused though there's a solution. Here is the list of paths that gcc on x86-64 searched:

/nix/store/54klr10i53jdfgn7322mzgza6wsai0q8-gcc-10.3.0-lib/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/54klr10i53jdfgn7322mzgza6wsai0q8-gcc-10.3.0-lib/lib/libgomp.spec
/nix/store/sbbifs2ykc05inws26203h0xwcadnf0l-glibc-2.32-46/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/sbbifs2ykc05inws26203h0xwcadnf0l-glibc-2.32-46/lib/libgomp.spec
/nix/store/35pnk5kwi26m3ph2bc7dxwjnavpzl8cn-gcc-wrapper-10.3.0/bin/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/35pnk5kwi26m3ph2bc7dxwjnavpzl8cn-gcc-wrapper-10.3.0/bin/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../x86_64-unknown-linux-gnu/lib/x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../x86_64-unknown-linux-gnu/lib/../lib64/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../x86_64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/h3f8rn6wwanph9m3rc1gl0lldbr57w3l-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../../../lib64/libgomp.spec
  -> hurray, we symlink $out/lib64 to $out/lib - success

The same applies on aarch64. However, on riscv64 it tries the following before giving up:

/nix/store/zw9qnha64sygasja7am4fhmlx5jx4yhb-gcc-10.3.0-lib/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/zw9qnha64sygasja7am4fhmlx5jx4yhb-gcc-10.3.0-lib/lib/libgomp.spec
/nix/store/m1jbr8vqnpgnpjfdzx13da897pd3aa6f-glibc-2.32-46/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/m1jbr8vqnpgnpjfdzx13da897pd3aa6f-glibc-2.32-46/lib/libgomp.spec
/nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/57q4bwd5hz1vpxr4fx7px14qxay9aw00-gcc-wrapper-10.3.0/bin/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/../../../../riscv64-unknown-linux-gnu/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/nix/store/dky7mm6fg9b30cqif055m1h8k91kihjs-gcc-10.3.0/lib/gcc/riscv64-unknown-linux-gnu/10.3.0/../../../../riscv64-unknown-linux-gnu/lib/libgomp.spec
/lib64/lp64d/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/lib64/lp64d/libgomp.spec
/usr/lib64/lp64d/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/usr/lib64/lp64d/libgomp.spec
/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/lib/libgomp.spec
/usr/lib/riscv64-unknown-linux-gnu/10.3.0/libgomp.spec
/usr/lib/libgomp.spec

At first glance we may want to link $out/[triplet]/lib to $out/lib as well, but I feel uneasy adopting a solution without knowing why. gcc/config/riscv/linux.h has these lines that may explain the lib64/lp64d paths, but not the others. This seems to be handled in set_up_specs which follows the startfile prefixes, but I'm pretty unfamiliar with the code.

zhaofengli · 2021-06-05T21:19:42Z

I've tested a naive fix which does make -fopenmp work in the final native toolchain. Would like to hear whether there is a more proper fix to the issue.

cc @Ericson2314 @oxalica

oxalica · 2021-10-23T07:08:55Z

Rebased and re-bootstrapped with QEMU VM now. PR description is also updated.

I hope bootstrap tools can be uploaded and this PR can be merged soon to make fundamental stdenv available. It makes everything, including fixing other packages, way much easier for native platform.

PS: I'm not sure who is the right person to ask for uploading bootstrap tools. Please cc them if anyone knows.

r-burns · 2021-10-23T07:18:31Z

I left a comment in #141448 (comment) about this, basically I think non-reproducible bootstrap tools might be a no-go, and we're best off trying to generate bootstrap tools on hydra so there's no leap of faith for the tarball uploader.

This could mean either:

Build exotic platforms' native bootstrap tools on hydra using an emulator
Fix or work around Cross bootstrap tools have broken stackprotector support #113977 so we can use the existing cross-compiled bootstrap tools

oxalica · 2021-10-23T13:30:59Z

@r-burns

I think non-reproducible bootstrap tools might be a no-go

I don't state that bootstrap tools are non-reproducible. They should, since the gcc issue is already resolved. I give the steps and final hash mainly for others to reproduce the same result.

Build exotic platforms' native bootstrap tools on hydra using an emulator

I assume you mean binfmt + qemu user-mode. I tried it and it works for all packages except some coreutils tests. If possible, we could disable these tests, especially, the tests running programs in non-existing CWD and expecting a specific error message, and then everything would work. I'm not sure this is a good way since we need to a workaround for an emulator rather than a platform, which is quite strange. But yeah there are already tons of workarounds of coreutils tests.

Fix or work around "Cross bootstrap tools have broken stackprotector support #113977" so we can use the existing cross-compiled bootstrap tools

Do you mean to just use cross-compiled bootstrap tools without another bootstrapping? This means I just need to revert the last commit here. It absolutely works. I doesn't run into the issue you mentioned, since I use the cross-compiled bootstrap tools (stage0) to get native bootstrap tools (stage1). And stage1 is still able to build stage2 bootstrap tools. Thus I built gcc four times in the chain without any issue.

If we need hydra to perform the build, what should I do here?

Ericson2314 · 2021-10-23T14:53:39Z

pkgs/build-support/bintools-wrapper/default.nix

@@ -81,6 +81,8 @@ let
    else if targetPlatform.system == "aarch64-linux"  then "${sharedLibraryLoader}/lib/ld-linux-aarch64.so.1"
    else if targetPlatform.system == "powerpc-linux"  then "${sharedLibraryLoader}/lib/ld.so.1"
    else if targetPlatform.isMips                     then "${sharedLibraryLoader}/lib/ld.so.1"
+    # `ld-linux-riscv{32,64}-<abi>.so.1`
+    else if targetPlatform.isRiscV                    then "${libc_lib}/lib/ld-linux-riscv*.so.1"


Should be "shared library loader" like the others.

r-burns · 2021-10-23T17:56:14Z

I don't state that bootstrap tools are non-reproducible. They should, since the gcc issue is already resolved. I give the steps and final hash mainly for others to reproduce the same result.

Right I misspoke, I just meant it might be easier to get someone to mirror a hydra-built trusted-by-default artifact than something that is trusted-if-verified.

This means I just need to revert the last commit here. It absolutely works. I doesn't run into the issue you mentioned, since I use the cross-compiled bootstrap tools (stage0) to get native bootstrap tools (stage1).

Oh, that's great! I would just say to revert the last commit then.

If we need hydra to perform the build, what should I do here?

I don't think we need to do anything else then, we just need to get someone with infra permissions to mirror the artifact from nixpkgs:cross-trunk:bootstrapTools.riscv64.build to tarballs.nixos.org

oxalica · 2021-10-25T13:11:08Z

I'm wait for the coreutils 9.0 in staging-next #141684. Hope we can have it for linux.
I'll try to patch coreutils and test bootstrapping from cross compiled seed with binfmt after that.

SuperSandro2000 · 2021-10-25T13:27:38Z

pkgs/stdenv/linux/default.nix

+    extraNativeBuildInputs =
+      lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;


Suggested change

extraNativeBuildInputs =

lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;

extraNativeBuildInputs = lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;

zhaofengli · 2021-10-26T23:55:26Z

#143060 enables argv[0] preservation in binfmt and fixes all the argv[0]-related test failures in coreutils, however the full test suite still doesn't pass (see linked logs).

oxalica · 2021-10-27T07:17:39Z

#143060 enables argv[0] preservation in binfmt and fixes all the argv[0]-related test failures in coreutils, however the full test suite still doesn't pass (see linked logs).

Almost forget it. I always have my patch #118926 applied. Besides the argv[0] issue, some other failures are:

rm/rm-readdir-fail.sh, misc/csplit-io-err.sh
LD_PRELOAD is set. libc tries to preload the riscv64 .so when executing the intermediate shell script and QEMU, resulting two lines of ERROR: ld.so: object '...' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored..
misc/date.pl, misc/sort-debug-warn.sh
LC_ALL is set. It cause the intermediate shell script to raise
warning: setlocale: LC_ALL: cannot change locale.
ls/removed-directory.sh
It tries to execute a binary in non-existing CWD, causing the intermediate shell script to raise
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory.

(1), (2) and (3) are all related to the intermediate shell script. We can fix them by,

Use a static compiled C binary to forward arguments, replacing the shell script. So environment variables and CWD doesn't affect us.
Unset LD_* and pass -E LD_*=value to set environments only for target ELF, not for QEMU itself.

@zhaofengli Are you interested in implementing this? I think we could enhance #143060 to fix all of these.

zhaofengli · 2021-10-27T07:27:55Z

@oxalica I didn't look deeper into the remaining failures - Thanks for doing that! I think for 1, one way is to build the wrappers alongside qemu, packaged as a separate output (e.g., pkgs.qemu.binfmtShims). I'll look more and give it a try tomorrow.

vcunat · 2021-11-02T06:45:00Z

I'm wait for the coreutils 9.0 in staging-next #141684. Hope we can have it for linux.

It is in master now, though there were complications because of a ZFS bug.

oxalica · 2021-11-27T06:00:27Z

Rebased and removed commits containing binary hashes.

@r-burns
I'd request a Hydra job to build cross compiled bootstrap tools on this PR, basically,
nix build -f pkgs/stdenv/linux/make-bootstrap-tools-cross.nix riscv64.build
And upload two result binaries as other bootstrap tools did.

vcunat · 2021-11-28T15:15:49Z

So we simply merge this and it will appear there, right? https://hydra.nixos.org/job/nixpkgs/cross-trunk/bootstrapTools.riscv64.bootstrapTools

(And continue in followup PRs.)

vcunat · 2021-11-29T19:32:34Z

I expect the builds to finish within a few hours: https://hydra.nixos.org/eval/1726433?filter=riscv64

oxalica · 2021-11-29T21:00:13Z

It's done building and I can match result hashes with my own build.

But how can we reference these tools by tarball.nixos.org like others?

nixpkgs/pkgs/stdenv/linux/bootstrap-files/aarch64.nix

Line 3 in 33da2dc

    
           url = "http://tarballs.nixos.org/stdenv-linux/aarch64/bb3ef8a95c9659596b8a34d27881cd30ffea2f9f/busybox";

Do we need someone manually upload build results of busybox and bootstrap-tools.tar.xz?

vcunat · 2021-11-29T21:14:36Z

For the commit in URL it's customary to use the one that generated the bootstrap, i.e. 9bd3cf0063b80428bd85a286205adab4b6ffcbd6 here. The rest is probably obvious?

I'd suggest preparing the PR and test that the bootstrap works locally (if you prefetch the files, nix won't attempt download). Then ping someone who can upload it manually; I'd suggest @grahamc. After upload, perhaps re-test and then merge.

oxalica · 2021-11-29T22:04:25Z

I've opened the following PR #147942.

I'd suggest preparing the PR and test that the bootstrap works locally (if you prefetch the files, nix won't attempt download).

I had tested building stdenv with my previous built bootstrap files. And Hydra reproduced my bootstrap files. So I could simply change URLs of FODs, which doesn't change any downstream derivations.

oxalica requested a review from Ericson2314 as a code owner March 8, 2021 12:01

ofborg bot added 6.topic: stdenv Standard environment 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Mar 8, 2021

SuperSandro2000 reviewed Mar 8, 2021

View reviewed changes

pkgs/build-support/bintools-wrapper/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Mar 30, 2021

View reviewed changes

pkgs/development/tools/misc/libtool/libtool2.nix Outdated Show resolved Hide resolved

oxalica force-pushed the test/riscv-test-libatomic branch from 044425e to 0413a5c Compare April 1, 2021 13:18

oxalica requested a review from Ericson2314 April 1, 2021 16:06

This was referenced May 31, 2021

libjpeg-turbo: Fix tests on RISC-V #125034

Merged

rust: Bootstrap riscv64 #125132

Merged

imagemagick: Allow compilation without --with-gcc-arch #125150

Merged

NixOS on RISC-V #101651

Closed

sd-image-riscv64-qemu: init #125451

Merged

SuperSandro2000 added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jun 3, 2021

r-burns mentioned this pull request Oct 13, 2021

stdenv: add pure Linux stdenv support for powerpc64le and sparc64 #141448

Draft

12 tasks

oxalica force-pushed the test/riscv-test-libatomic branch from 0413a5c to 66d9511 Compare October 23, 2021 06:09

ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Oct 23, 2021

Ericson2314 reviewed Oct 23, 2021

View reviewed changes

SuperSandro2000 reviewed Oct 25, 2021

View reviewed changes

zhaofengli mentioned this pull request Oct 28, 2021

nixos/binfmt: Add QEMU wrapper to preserve argv[0] #143060

Merged

10 tasks

oxalica added 3 commits November 26, 2021 00:32

bintools-wrapper: add dynamicLinker for riscv

3b069a2

make-bootstrap-tools: produce libatomic on riscv platform

fc3ea54

stdenv: patch autoconf script for riscv in stage 2

8cf96b9

oxalica force-pushed the test/riscv-test-libatomic branch from 66d9511 to 8cf96b9 Compare November 27, 2021 05:50

ofborg bot added 10.rebuild-linux: 1-10 10.rebuild-linux: 1 and removed 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Nov 27, 2021

r-burns approved these changes Nov 28, 2021

View reviewed changes

oxalica changed the title ~~stdenv: bootstrap riscv64-linux~~ stdenv: bootstrap riscv64-linux (Step 1) Nov 29, 2021

vcunat merged commit 9bd3cf0 into NixOS:master Nov 29, 2021

oxalica mentioned this pull request Nov 29, 2021

stdenv: bootstrap riscv64 (Step 2) #147942

Merged

12 tasks

oxalica deleted the test/riscv-test-libatomic branch November 29, 2021 22:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stdenv: bootstrap riscv64-linux (Step 1) #115406

stdenv: bootstrap riscv64-linux (Step 1) #115406

oxalica commented Mar 8, 2021 •

edited

Loading

oxalica commented Mar 9, 2021

zhaofengli commented Mar 29, 2021

oxalica commented Mar 31, 2021

Ericson2314 commented Mar 31, 2021

oxalica commented Apr 1, 2021

zhaofengli commented Jun 3, 2021

zhaofengli commented Jun 3, 2021

zhaofengli commented Jun 4, 2021 •

edited

Loading

zhaofengli commented Jun 5, 2021

oxalica commented Oct 23, 2021

r-burns commented Oct 23, 2021

oxalica commented Oct 23, 2021

Ericson2314 Oct 23, 2021

r-burns commented Oct 23, 2021

oxalica commented Oct 25, 2021

SuperSandro2000 Oct 25, 2021

zhaofengli commented Oct 26, 2021

oxalica commented Oct 27, 2021

zhaofengli commented Oct 27, 2021

vcunat commented Nov 2, 2021

oxalica commented Nov 27, 2021

vcunat commented Nov 28, 2021

vcunat commented Nov 29, 2021

oxalica commented Nov 29, 2021

vcunat commented Nov 29, 2021

oxalica commented Nov 29, 2021

		extraNativeBuildInputs =
		lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;

	extraNativeBuildInputs =
	lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;
	extraNativeBuildInputs = lib.optional (localSystem.isRiscV) prevStage.updateAutotoolsGnuConfigScriptsHook;

stdenv: bootstrap riscv64-linux (Step 1) #115406

stdenv: bootstrap riscv64-linux (Step 1) #115406

Conversation

oxalica commented Mar 8, 2021 • edited Loading

Motivation for this change

Steps I did

Notes

oxalica commented Mar 9, 2021

zhaofengli commented Mar 29, 2021

oxalica commented Mar 31, 2021

Ericson2314 commented Mar 31, 2021

oxalica commented Apr 1, 2021

zhaofengli commented Jun 3, 2021

zhaofengli commented Jun 3, 2021

zhaofengli commented Jun 4, 2021 • edited Loading

zhaofengli commented Jun 5, 2021

oxalica commented Oct 23, 2021

r-burns commented Oct 23, 2021

oxalica commented Oct 23, 2021

Ericson2314 Oct 23, 2021

Choose a reason for hiding this comment

r-burns commented Oct 23, 2021

oxalica commented Oct 25, 2021

SuperSandro2000 Oct 25, 2021

Choose a reason for hiding this comment

zhaofengli commented Oct 26, 2021

oxalica commented Oct 27, 2021

zhaofengli commented Oct 27, 2021

vcunat commented Nov 2, 2021

oxalica commented Nov 27, 2021

vcunat commented Nov 28, 2021

vcunat commented Nov 29, 2021

oxalica commented Nov 29, 2021

vcunat commented Nov 29, 2021

oxalica commented Nov 29, 2021

oxalica commented Mar 8, 2021 •

edited

Loading

zhaofengli commented Jun 4, 2021 •

edited

Loading