Building 3.0 on Arch Linux

[adaggarwal]

@alucryd commented on Thu Sep 19 2019

Hi guys, I'm trying to prepare for the 3.0 release in the Arch Linux repositories, but it looks like there are several issues that need to be addressed.

First and foremost, it seems impossible to build in a chroot with systemd-nspawn because of the libgit2 wrapper. There was an issue in 2.x with nspawn as well, #285, could it be somewhat similar?

Here is the log:

System.Exception: Something went wrong when cloning repo https://github.com/dotnet/arcade at <default branch> into /build/dotnet-core/src/dotnet-source-build/bin/src/arcade
   ---> System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception.
   ---> System.DllNotFoundException: Unable to load shared library 'git2-572e4d8' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libgit2-572e4d8: cannot open shared object file: No such file or directory
     at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()
     at LibGit2Sharp.Core.NativeMethods.InitializeNativeLibrary()
     at LibGit2Sharp.Core.NativeMethods..cctor()
     --- End of inner exception stack trace ---
     at LibGit2Sharp.Core.NativeMethods.git_clone(git_repository*& repo, String origin_url, FilePath workdir_path, GitCloneOptions& opts)
     at LibGit2Sharp.Core.Proxy.git_clone(String url, String workdir, GitCloneOptions& opts)
     at LibGit2Sharp.Repository.Clone(String sourceUrl, String workdirPath, CloneOptions options)
     at Microsoft.DotNet.DarcLib.RemoteRepoBase.Clone(String repoUri, String commit, String targetDirectory, ILogger _logger, String pat, String gitDirectory) in /build/dotnet-core/src/dotnet-source-build/tools-local/arcade-services/src/Microsoft.DotNet.Darc/src/DarcLib/RemoteRepoBase.cs:line 176
     --- End of inner exception stack trace ---

Building outside of a chroot at least gets past this. However trying to build the preview9 tag fails at corefx with no helpful error log, then trying to build the head of the 3.0 release branch passes corefx but fails at core-setup. Here's the build log: https://paste.xinu.at/BDoH58YNF6D5C6Bn/

Also I notice in the build log that the computed RID is arch.-x86 which is definitely incorrect. In 2.x I somehow fixed that by patching the relevant built files, but since you got rid of the git submodules (why? that was much more elegant) I can't patch anymore. I read about a DOTNET_RUNTIME_ID variable on another issue but it didn't help. Are you still computing the RID from /etc/os-release? Is it possible to override it without patching?

FWIW, here's our os-release:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

---

@dseefeld commented on Thu Sep 19 2019

@adaggarwal any ideas to unblock @alucryd ?

---

@omajid commented on Thu Sep 19 2019

First and foremost, it seems impossible to build in a chroot with systemd-nspawn because of the libgit2 wrapper. There was an issue in 2.x with nspawn as well, #285, could it be somewhat similar?

Here is the log:

System.Exception: Something went wrong when cloning repo https://github.com/dotnet/arcade at <default branch> into /build/dotnet-core/src/dotnet-source-build/bin/src/arcade
  ---> System.TypeInitializationException: The type initializer for 'LibGit2Sharp.Core.NativeMethods' threw an exception.
  ---> System.DllNotFoundException: Unable to load shared library 'git2-572e4d8' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libgit2-572e4d8: cannot open shared object file: No such file or directory

 at LibGit2Sharp.Core.NativeMethods.git_libgit2_init()

I think nspawn is still an issue. But I am surprised about this error. It really looks like #1083, which reproduces outside of chroots as well.

Can you try with crummel@57e01ba#diff-98f322625c74ff72f9326bd4dca549baR153 to see if that works any better?

Here's the build log: https://paste.xinu.at/BDoH58YNF6D5C6Bn/

 Could not find a part of the path '/home/alucryd/Packages/community/dotnet-core/trunk/src/dotnet-source-build/bin/src/core-setup.8f5d7b1ba4f744199ba006cac38dfcb9a2408880/artifacts/bin/linux-x64.Release/corehost

Do you have a /home/alucryd/Packages/community/dotnet-core/trunk/src/dotnet-source-build/bin/src/core-setup.8f5d7b1ba4f744199ba006cac38dfcb9a2408880/artifacts/bin/linux-x64.Release/ dir? What about an arch.-x64.Release directory instead of linux-x64.Release? This feels like another RID issue.

why? that was much more elegant

AFAIK, source-build is trying to get closer to how "official" builds are done, and reduce differences that lead to surprises. The infrastructure tools update commit hashes across all of .NET Core. Now they do it for source-build as well.

I am working on packaging .NET Core for Fedora. Using submodules was definitely nicer as a packager, but I am okay with trading some difficulty there to get source-build to match "official" builds closely and quickly.

---

@alucryd commented on Fri Sep 20 2019

Thanks for the quick answers!

I think nspawn is still an issue. But I am surprised about this error. It really looks like #1083, which reproduces outside of chroots as well.

Can you try with crummel@57e01ba#diff-98f322625c74ff72f9326bd4dca549baR153 to see if that works any better?

Bypassing Darc does allow the git repos to be cloned, thanks! Had to adapt a bit for RC1, and hardcoded versions are probably outdated because I'm getting a bunch of missing source files after the cloning is done, but at least libgit2 is no longer an issue if we can go that route.

Do you have a /home/alucryd/Packages/community/dotnet-core/trunk/src/dotnet-source-build/bin/src/core-setup.8f5d7b1ba4f744199ba006cac38dfcb9a2408880/artifacts/bin/linux-x64.Release/ dir? What about an arch.-x64.Release directory instead of linux-x64.Release? This feels like another RID issue.

The corehost directory is indeed inside a directory matching the arch.-x64 rid instead of linux-x64.

AFAIK, source-build is trying to get closer to how "official" builds are done, and reduce differences that lead to surprises. The infrastructure tools update commit hashes across all of .NET Core. Now they do it for source-build as well.

I am working on packaging .NET Core for Fedora. Using submodules was definitely nicer as a packager, but I am okay with trading some difficulty there to get source-build to match "official" builds closely and quickly.

Seems reasonable, thanks for the heads up!

---

@Saancreed commented on Sat Sep 21 2019

First and foremost, it seems impossible to build in a chroot with systemd-nspawn because of the libgit2 wrapper. There was an issue in 2.x with nspawn as well, #285, could it be somewhat similar?

I believe it can fail in a chroot because of native library's dependency on openssl-1.0. I have it in my makedepends and building the package with makechrootpkg can start without libgit2–related issues, but then it fails on corefx as well.

That said, my error seems to be slightly different and more helpful, possibly due to RID patch I apply to corefx, which actually causes it to be detected as arch-x64.

    chmod: cannot access '/build/dotnet/src/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/': No such file or directory
  /build/dotnet/src/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj(63,5): error MSB3073: The command "chmod +x /build/dotnet/src/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.
  ##vso[task.logissue type=error;sourcepath=/build/dotnet/src/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj;linenumber=63;columnnumber=5;code=MSB3073;]The command "chmod +x /build/dotnet/src/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.

Here's my PKGBUILD with embedded corefx patch: https://pastebin.com/eDeKgH2R

Also, @alucryd could you please take a look at my post here regarding some issues with how .NET Core is currently packaged on Arch? I was going to post my findings on the mailing list when 3.0 releases but if I can reach you via other means then I'd rather avoid dealing with emails.

---

@alucryd commented on Tue Sep 24 2019

@Saancreed Thanks for the openssl-1.0 tip, I thought the openssl 1.1 support was finally global so I didn't think of adding it to makedepends.

I went over your post and that's mostly how I envisioned the packaging at first, albeit with some differences, but quickly dumped everything in /opt because dotnet core was clearly not FHS-friendly. Arch devs and I care a lot about the FHS, install_location may be the thing we need to finally package dotnet in a clean way.

Ideally I would like the dotnet-host package to strictly respect the FHS, with dotnet in /usr/bin and libhostfxr in /usr/lib, bypassing the host directory completely.
Then, like @omajid is doing, I'd rather put everything in /usr/lib/dotnet.
Versioning the runtime and sdk packages is already possible, however I will not version the very latest package. Doing so would mean creating a new PKGBUILD with every release, updating the existing unversioned one is much more streamlined. Also we tend to not support old versions unless absolutely necessary, I'd rather have older versioned packages in the AUR.
Finally the custom rid is needed since we don't do a portable build.

I'll work on an updated PKGBUILD following these guidelines, also just saw that coreclr relocated the rid computing in a single file, this will make patching easier.

---

@alucryd commented on Tue Sep 24 2019

Oh, looks like it's been fixed already dotnet/coreclr@268e1e1

---

@alucryd commented on Tue Sep 24 2019

New error with the head of release/3.0, I can build in a chroot with openssl-1.0, but it no longer passes corefx: https://paste.xinu.at/25BdF5ys2mMJutPc/

There is no testhost directory, good news is the rid looks fine though.

---

@wfurt commented on Tue Sep 24 2019

The test error happens when we fail to get NetCore.App for tests. There probably would be some hint early on during restore.

---

@alucryd commented on Tue Sep 24 2019

There does seem to be another error earlier in the full console log, as opposed to the corefx build log I posted, although there's not much said about it so I'm unsure whether it's another error or the same that appears later.

Here's the full log: https://paste.xinu.at/0wG1snbVFv49gpdo

---

@adaggarwal commented on Tue Sep 24 2019

There does seem to be another error earlier in the full console log, as opposed to the corefx build log I posted, although there's not much said about it so I'm unsure whether it's another error or the same that appears later.

Here's the full log: https://paste.xinu.at/0wG1snbVFv49gpdo

I think its the same error. testhost does seems to be missing in the corefx build. I wonder what results would we get if we try a standalone corefx build with 3da667af759d88b683ca851694ced073e8dc9960 SHA1 on Arch Linux. @alucryd, would you like to give that a try?

If you'd like to try, be sure to run the tests. Something like this - ./build.sh --restore --build --bl --buildtests --test

---

@wfurt commented on Tue Sep 24 2019

Also

/build/dotnet-core/src/dotnet-source-build/.dotnet/dotnet-install.sh: line 147: VERSION_ID: unbound variable

this seems to fail on parsing version file. We should fix that as well.

---

@omajid commented on Tue Sep 24 2019

The fix for empty VERSION_ID didn't get anywhere last time: dotnet/cli#6708

---

@alucryd commented on Tue Sep 24 2019

./build.sh --restore --build --bl --buildtests --test

@adaggarwal Sure thing. It appears to build fine, although some of the tests failed: https://paste.xinu.at/fHjGgh9o9hUkxrZi/

I can get you a full log if needed, but I'll need to run it again while redirecting the output as it was too much for my terminal buffer.

---

@adaggarwal commented on Tue Sep 24 2019

I can get you a full log if needed, but I'll need to run it again while redirecting the output as it was too much for my terminal buffer.

I think its fine as far as testhost problem doesn't show up.

---

@Saancreed commented on Tue Sep 24 2019

@adaggarwal Sure thing. It appears to build fine, although some of the tests failed: https://paste.xinu.at/fHjGgh9o9hUkxrZi/

Afaik those ping failures are caused by corefx expecting to find ping6 binary in one of three well–known locations but on Arch it doesn't exist.

---

@adaggarwal commented on Tue Sep 24 2019

Afaik those ping failures are caused by corefx expecting to find ping6 binary in one of three well–known locations but on Arch it doesn't exist.

We do face ipv6 problem intermittently. I am not sure if testhost stuff is related to that. But this is something that's worth a look. #1200

cc @crummel @NikolaMilosavljevic

---

@alucryd commented on Fri Sep 27 2019

The tagged 3.0.0 still fails at corefx :/

---

@leafi commented on Fri Sep 27 2019

A Google search shows that, in iputils, ping6 was merged into ping in 2015.

'-4' or '-6' equivalent options are automatically added if ping is called from a symlink ping4 or ping6, but the symlinks are not shipped as part of iputils - it's up to Linux distributions whether they include them or not. Arch doesn't.

There are multiple ways UnixCommandLinePing.cs could be adjusted to deal with this. Adding a '-4' or a '-6' to the command line is easy, whereas choosing how to detect 'modern ping' is not.

(Suggestion: Either always fall back to s_discoveredPing4UtilityPath when GetPingUtilityPath(ipv4: false) can't find ping6 and append '-6' in the ipv6 case when ping4 utility path == ping6 utility path, or run ping in some way when ping6 cannot be found and search the output for evidence ping supports -4 and -6.)

Even if you want to leave IPv6 ping tests broken on Arch, this code should change anyway. The code uses 'ping' not 'ping4', so with a ping that supports both ipv4 and ipv6 you are not actually constructing a command line that tests IPv4 pinging in the IPv4 case but instead a command line that leaves it up to the ping command whether to choose IPv4 or IPv6.

@alucryd The pastebin says "Please check /home/alucryd/Packages/community/dotnet-core/trunk/src/dotnet-source-build/bin/src/corefx/artifacts/bin/System.Net.Sockets.Tests/netcoreapp-Unix-Debug/testResults.xml for details!". Could you have a gander and see if the failing tests are indeed 'UnixPingUtilityTests' ones?

Edit: Never mind.. The log already provided does indeed have a line about System.Net.Ping.Functional.Tests.

---

@adaggarwal commented on Fri Sep 27 2019

@leafi Did you try a local corefx patch to test if this works?

(Suggestion: Either always fall back to s_discoveredPing4UtilityPath when GetPingUtilityPath(ipv4: false) can't find ping6 and append '-6' in the ipv6 case when ping4 utility path == ping6 utility path, or run ping in some way when ping6 cannot be found and search the output for evidence ping supports -4 and -6.)

Even if you want to leave IPv6 ping tests broken on Arch, this code should change anyway. The code uses 'ping' not 'ping4', so with a ping that supports both ipv4 and ipv6 you are not actually constructing a command line that tests IPv4 pinging in the IPv4 case but instead a command line that leaves it up to the ping command whether to choose IPv4 or IPv6.

---

@leafi commented on Sat Sep 28 2019

@adaggarwal I'm sorry. I can't figure out how to get past the missing testhost error.

I thought changing the corefx hash in eng/Version.Details.xml would be enough - and it does seem like I'm checking out that hash now - but I get a build failure on ./clean.sh -a && ./build.sh --restore --build --test (after removing corefx patch 0003 which won't apply) in bin/logs/corefx.log that fails the build:

  /home/leaf/Downloads/dotnet/source-build/packages/restored/microsoft.targetingpack.netframework.v4.7.2/1.0.0/lib/net472/netstandard.dll (Microsoft.TargetingPack.NETFramework.v4.7.2.1.0.0) -> /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/netfx/netcoreapp-Release/netstandard.dll
  /home/leaf/Downloads/dotnet/source-build/packages/restored/microsoft.targetingpack.netframework.v4.7.2/1.0.0/lib/net472/sysglobl.dll (Microsoft.TargetingPack.NETFramework.v4.7.2.1.0.0) -> /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/netfx/netcoreapp-Release/sysglobl.dll
  Restore completed in 59.78 ms for /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj.
  chmod: cannot access '/home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/': No such file or directory
/home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj(63,5): error MSB3073: The command "chmod +x /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.
##vso[task.logissue type=error;sourcepath=/home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj;linenumber=63;columnnumber=5;code=MSB3073;]The command "chmod +x /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.

Build FAILED.

/home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/external/runtime/runtime.depproj(63,5): error MSB3073: The command "chmod +x /home/leaf/Downloads/dotnet/source-build/bin/src/corefx.3da667af759d88b683ca851694ced073e8dc9960/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/" exited with code 1.
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:10.63
##vso[task.logissue type=error]Build failed (exit code '1').

Edit: Just to help clarify exactly the state of my source-build checkout:

[leaf@alicorn source-build]$ git status
On branch release/3.0
Your branch is up to date with 'origin/release/3.0'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   eng/Version.Details.xml
	deleted:    patches/corefx/0003-Check-in-package-baseline-so-all-build-nodes-see-the.patch

no changes added to commit (use "git add" and/or "git commit -a")

---

@adaggarwal commented on Mon Sep 30 2019

I thought changing the corefx hash in eng/Version.Details.xml would be enough - and it does seem like I'm checking out that hash now - but I get a build failure on ./clean.sh -a && ./build.sh --restore --build --test (after removing corefx patch 0003 which won't apply) in bin/logs/corefx.log that fails the build:

Oh no worries. Thanks for clarifying.
All corefx versions may not be compatible with current source-build due to dependency and coherency issues. A good trial would be - cloning a local coreFX copy followed by checking out the version we use in source-build. And then try to build along with tests:
./build.sh -buildtests -pack -restore -bl -test

Motive - If you are able to get through this; we would be sure that testhost builds fine on your current system with standalone coreFX; but RID is somehow making the source-build mechanism to fail on testhost while building test utilities for corefx.

---

@Saancreed commented on Mon Sep 30 2019

@adaggarwal On my machine, directly building corefx@4ac4c0367003fe3973a3648eb0715ddb0e3bbcea with ./build.sh -buildtests -pack -restore -bl -test fails with error NETSDK1042: Could not load PlatformManifest from '/build/dotnet/src/corefx/artifacts/bin/testhost/netcoreapp-Linux-Debug-x64/shared/Microsoft.NETCore.App/3.0.0/PlatformManifest.txt' because it did not exist. Only the following files are in this location:

-rwxr--r-- 3 builduser builduser    25224 Sep 12 22:52 corerun
-rwxr--r-- 3 builduser builduser   109656 Sep 12 23:01 createdump
-rwxr--r-- 3 builduser builduser   438072 Sep 12 23:01 createdump.dbg
-rwxr-xr-x 4 builduser builduser    68920 Sep 10 10:58 dotnet
-rwxr--r-- 3 builduser builduser  3072816 Sep 12 22:58 libclrjit.so
-rwxr--r-- 3 builduser builduser 15514568 Sep 12 22:58 libclrjit.so.dbg
-rwxr--r-- 3 builduser builduser  9171192 Sep 12 23:00 libcoreclr.so
-rwxr--r-- 3 builduser builduser 74090472 Sep 12 23:00 libcoreclr.so.dbg
-rwxr--r-- 3 builduser builduser   739608 Sep 12 22:48 libcoreclrtraceptprovider.so
-rwxr--r-- 3 builduser builduser   740168 Sep 12 22:48 libcoreclrtraceptprovider.so.dbg
-rwxr--r-- 3 builduser builduser   921624 Sep 12 22:58 libdbgshim.so
-rwxr--r-- 3 builduser builduser  2964544 Sep 12 22:58 libdbgshim.so.dbg
-rwxr--r-- 4 builduser builduser   372424 Sep 10 10:59 libhostfxr.so
-rwxr--r-- 3 builduser builduser   327384 Sep 10 11:00 libhostpolicy.so
-rwxr--r-- 3 builduser builduser  3476552 Sep 12 23:01 libmscordaccore.so
-rwxr--r-- 3 builduser builduser 39733032 Sep 12 23:01 libmscordaccore.so.dbg
-rwxr--r-- 3 builduser builduser  2697416 Sep 12 23:01 libmscordbi.so
-rwxr--r-- 3 builduser builduser 10393392 Sep 12 23:01 libmscordbi.so.dbg
-rwxr--r-- 3 builduser builduser    68856 Sep 10 11:00 libnethost.so
-rwxr--r-- 3 builduser builduser     2709 Sep 10 10:55 nethost.h
-rwxr--r-- 3 builduser builduser        2 Sep 12 22:47 _.pdb
-rwxr--r-- 3 builduser builduser      272 Sep 12 22:47 SOS_README.md
-rwxr--r-- 3 builduser builduser    54064 Sep 12 22:48 System.Globalization.Native.so
-rwxr--r-- 3 builduser builduser   132752 Sep 12 22:48 System.Globalization.Native.so.dbg
-rwxr--r-- 3 builduser builduser  3012096 Sep 12 23:01 System.Private.CoreLib.dll
-rwxr--r-- 3 builduser builduser  1255976 Sep 12 23:01 System.Private.CoreLib.pdb

Here is the terminal output: build.log

And here is binary log: Build.binlog

---

@adaggarwal commented on Mon Sep 30 2019

@adaggarwal Aditya Aggarwal FTE On my machine, directly building corefx@4ac4c0367003fe3973a3648eb0715ddb0e3bbcea with ./build.sh -buildtests -pack -restore -bl -test fails with error NETSDK1042: Could not load PlatformManifest from '/build/dotnet/src/corefx/artifacts/bin/testhost/netcoreapp-Linux-Debug-x64/shared/Microsoft.NETCore.App/3.0.0/PlatformManifest.txt' because it did not exist.

Great! Thanks for trying out. Clearly, corefx testhost is not building standalone and that's why source-build run fails as well as we depend on this SHA1.

@Saancreed Can you now try out the latest SHA1 corefx test built run on the same system?

In case that passes, we can wait for the source-build uptake of latest coreFX and that shall resolve this problem.

---

@Saancreed commented on Mon Sep 30 2019

Unfortunately, both latest master@4eece5734d828ee8bf3393a04b71d80d5d5e4828 and release/3.0@7231a76a4825a708e2333946b66aff648b35b5d7 fail with the same error.

---

@adaggarwal commented on Mon Sep 30 2019

Unfortunately, both latest master@4eece5734d828ee8bf3393a04b71d80d5d5e4828 and release/3.0@7231a76a4825a708e2333946b66aff648b35b5d7 fail with the same error.

Thanks for confirming.

---

@adaggarwal commented on Tue Oct 01 2019

Porting to #41481