Improve the experience with GitHub's tar.gz source archives

[dagood]

GitHub prominently displays tar.gz files of source code in various places:

But they fail to build!

$ ./build.sh
Found bootstrap SDK 3.1.102, bootstrap Arcade 1.0.0-beta.19359.6 fatal: Not a git repository (or any of the parent directories): .git

Instead, you need to actually have a git clone and check out the commit to build.

We should try to improve this experience. Options:

1. Disable these links. (I haven't seen a way.)

2. Make the GitHub-provided source archive work. May involve fixes to Arcade scripts.

3. Detect the situation and give actionable instructions to fix the problem.

4. Detect the situation and fix it up with a workaround. (Maybe it's enough to git init, even without any source-build repo data?)

[ostroffjh]

Thanks for filing this. The easiest quick solution is just to have a better error message explaining the need to use a git repository, and not the tarball. git init might work, but only if there were no attempt to git pull, since there wont be any remote. If it were possible to get the github built tarball to include the .git folder, all might be well, but I don't know if that's possible.

[dagood]

Yeah, I think git init might be the simplest. I don't know why we would actually need any git data in there.

Removing the unnecessary-seeming dependency on a .git dir would be the cleanest, but might need a bit more work.

Another thing we should do is actually provide the tarball output of ./build-source-tarball.sh and indicate it's what you should use to run an offline build. Related:

• End-to-end: "create linux-x64 compatible tarball for each release", create linux-x64 compatible tarball for each release #298
• Reasons we potentially can't do this yet due to distro incompatibility: The source-tarball genrated by source-build should work everywhere #1412 (might be fixed already)
• Improving publish infra to reduce manual per-release work to get this done: Upload artifacts from all legs and publish them Arcade-style #1321

[dseefeld]

The experience for 3.1 and 2.1 will be different here, since 2.1 is still using submodules which won't be in the archive.

[asbjornu]

Another thing we should do is actually provide the tarball output of ./build-source-tarball.sh and indicate it's what you should use to run an offline build.

Yes please. dotnet/sdk#11795 (comment)

[dagood]

Another thing we should do is actually provide the tarball output of ./build-source-tarball.sh and indicate it's what you should use to run an offline build.

Yes please. dotnet/sdk#11795 (comment)

You are actually able to do this yourself: you can run ./build-source-tarball.sh <target directory to create> from a Git clone, then that target directory contains the sources/prebuilts you need to do an offline build. That's what's submitted to distros. It would mostly just save time for us to provide this as a release artifact.

Bit more info here:
https://github.com/dotnet/source-build/tree/release/3.1/Documentation#building-net-core-sdk-and-a-source-tarball
https://github.com/dotnet/source-build/blob/release/3.1/Documentation/packaging-installation.md

[asbjornu]

@dagood, I'm not running a build, I'm only fetching source and performing a compile and install. I don't have CI/CD tooling available, as I'm operating within the confines of a Homebrew formula. I'm able to point to a .tar.gz file that is automatically extracted for me, and then operate on the extracted result.

Of course, I can write whatever Ruby code I want to orchestrate the universe, but that goes sharply against Homebrew's architecture and will most likely just end up with the formula pull request being rejected.

It feels like a very small change to your CI/CD pipeline to make the result of ./build-source-tarball.sh available as a release artifact so I can point the Homebrew formula to it instead of the one GitHub bundles up automatically.

[dagood]

Understood, thanks for expanding on that. And FWIW I can understand the underlying problem that producing and hosting a tar.gz yourself and feeding it into Homebrew (rather than using some artifact directly provided by this repo) seems like it'd be rejected as well for provenance reasons. (If it's even possible to feed it in that way.)

It feels like a very small change to your CI/CD pipeline to make the result of ./build-source-tarball.sh available as a release artifact

We actually don't have CI/CD for GitHub Releases. We have automation for some things, but nothing's published in a way that we could link it in each GitHub Release page. But, this is the purpose of those issues I linked to--to make this possible. I don't think these are hard things, but we're significantly pressed for time, and getting to a better and more sustainable place with source-build infra (in particular for 5.0) as well as continuing to service 2.1 and 3.1 are currently the highest priorities (vs. implementing new infra on existing servicing release trains to help new distro maintainers).

[dagood]

4. Detect the situation and fix it up with a workaround. (Maybe it's enough to git init, even without any source-build repo data?)

I'm going to give this a quick try. It would be trivial if possible, and it has essentially zero chance of adverse side effects because when .git doesn't exist, the build has no chance of doing anything useful anyway.

[asbjornu]

Understood, thanks for expanding on that. And FWIW I can understand the underlying problem that producing and hosting a tar.gz yourself and feeding it into Homebrew (rather than using some artifact directly provided by this repo) seems like it'd be rejected as well for provenance reasons. (If it's even possible to feed it in that way.)

Indeed.

It feels like a very small change to your CI/CD pipeline to make the result of ./build-source-tarball.sh available as a release artifact

We actually don't have CI/CD for GitHub Releases. We have automation for some things, but nothing's published in a way that we could link it in each GitHub Release page.

I have no idea how to set this up with AzDO, but with GitHub Actions, this is just 10 lines of YAML. I would have submitted a pull request if you would consider sprinkling some GitHub Actions on top of your existing AzDO pipeline.

4. Detect the situation and fix it up with a workaround. (Maybe it's enough to git init, even without any source-build repo data?)

I'm going to give this a quick try. It would be trivial if possible, and it has essentially zero chance of adverse side effects because when .git doesn't exist, the build has no chance of doing anything useful anyway.

That would be awesome. I'll be the first one to test and let you know how it works out. 👍

[dagood]

I have no idea how to set this up with AzDO, but with GitHub Actions, this is just 10 lines of YAML. I would have submitted a pull request if you would consider sprinkling some GitHub Actions on top of your existing AzDO pipeline.

We won't accept GitHub Actions, but the uploading itself is pretty easy on AzDO too. There's a deeper problem though: we build internally for many servicing releases (e.g. if they have security fixes) so we need to have some new CI thing that pulls artifacts from private -> public storage, or rebuild the entire release but with a flag set to publish those artifacts to public storage. Rebuilding is the easy path here for dev time but wastes CI machines. (There could also potentially be problems with nondeterminism, but we're inflicting that on everyone whether they build from a ./build-source-tarball.sh output or from our source code, so that isn't a consideration here.)

In any case... nothing insurmountable but someone needs to spend some time on it.

That would be awesome. I'll be the first one to test and let you know how it works out. 👍

I posted a PR at #1695, seems to work locally although I've only let it run as far as corefx so far. I included links to zip/tar.gz with the workaround included. 😄

[dagood]

#1695 with the git init approach is merged into release/3.1, and local build with the GitHub tar.gz worked for me. Closing.