Provide buildable source tarballs as release artifacts

[MichaelSimons]

This request comes from the .NET 8.0 Preview 1 release announcement.

The source tarballs Assets associated with the GitHub Release are not buildable. In order to get a buildable source-tarball, you must use ./eng/pack-source.sh. To improve the distro maintainer workflow, .NET should provide a buildable source tarball.

Goals

• Modified build.sh to take in a url and sha for the source to use by sourcelink. If not provided require the build is executing in a git context and read the values from it.
• Update the release pipeline to no longer generate the source tarball and attach it to the release. We will use the sources attached by default.

Factor out the logic to generate a bootstrap artifacts from the prep script into a standalone script. Invoke this at the end of CI so that we produce the bootstrap artifacts with every build.
Modify the release pipeline to upload the new bootstrap artifacts to blob storage
Modify the release pipeline to attach the SDK, bootstrap artifacts and prebuilts to the release.

[omajid]

There are some issues we should re-open (or re-examine) if we want to provide tarballs:

• create linux-x64 compatible tarball for each release #298
• Servicing VMR doesn't work without root gitdir #2927
• Change layout of source tarball to have a single directory  #2895

[tmds]

#298 is a much older request for a tarball.

Besides the sources, this issue also talked about archiving the needed pre-built.

So then, when given a (portable) bootstrap SDK, the source tarball, and a prebuilts tarball, you'd have everything to do an offline build.

Such prebuilts archives would be per portable rid.

@ayakael mentioned ./eng/pack-source.sh is mostly about some git files:

those files are missing .git/HEAD and .git/config

For my understanding: why does source-build need these git files?

[omajid]

For sourcelink: #3112 (comment)

[tmds]

For sourcelink: #3112 (comment)

Yes, I was assuming that would be it.

The source-build based builds will refer to https://github.com/dotnet/dotnet for their sources then.
That won't include any local patches, but it would be accurate for most source files.

[MichaelSimons]

The source-build based builds will refer to https://github.com/dotnet/dotnet for their sources then.
That won't include any local patches, but it would be accurate for most source files.

The idea would be that distro maintainers would either push their patches upstream or create a downstream of the VMR with any customization applied and build from there.

[premun]

@MichaelSimons I believe I can easily take care of this as I am working on the automation that will push the release+tag to dotnet/dotnet. That is, calling pack-sources.sh and appending the tarball to the release.

I will have to check whether I can get rid of the default assets and I need good naming for the new asset to make it clear which one is the one people should download.

[MichaelSimons]

We discussed this in triage a bit but ran out of time to finish. The discussion was that it would be desirable to be able to build from a GitHub tarball/zip. We know that isn't possible today because of the sourcelink requirements. A suggestion was made that if building outside the context of a git repo, we could provide a way to specify a sha and url for sourcelink to utilize. The build would then init a .git/HEAD and .git/config from these build options.

This approach has the benefit of being able to use the native constructs to produce the tarball versus the custom pack-sources.sh.

[premun]

I think that sounds like a better solution maybe. It would be good if the build script checked for the existence of these and print a nicer error message too btw.

Btw, I just tried to publish a release from the automation and it looks like this:

[MichaelSimons]

It would be good if the build script checked for the existence of these and print a nicer error message too btw.

Agree, I envision the build script considering these options required executed outside the context of a git repo.

[premun]

@MichaelSimons I have already implemented appending the exported tarball amidst my current SB pipeline changes. We only need to settle on a suitable name for the attachment. I'd keep it there until we have a solution on SB infra's side?
It's now part of #3276

[tmds]

Besides the sources, this issue also talked about archiving the needed pre-built.

I wonder if there is an interest in archives with pre-builts for the portable rids?

Without such a tarball, a bootstrap build starts by downloading nuget packages from the Internet.

It seems better for reproducibility to archive pre-builts too.

@omajid @ayakael what do you think? Is this something you consider useful?

[ayakael]

Besides the sources, this issue also talked about archiving the needed pre-built.

I wonder if there is an interest in archives with pre-builts for the portable rids?

Without such a tarball, a bootstrap build starts by downloading nuget packages from the Internet.

It seems better for reproducibility to archive pre-builts too.

@omajid @ayakael what do you think? Is this something you consider useful?

I think it'd be useful in terms of making it explicit where to get the prebuilts. I remember when first packaging dotnet that it was harder than it should've been to find prebuilts as the link to them isn't explicitely put anywhere and then you had to dig around to find the version number (i.e. in eng/Version.Details.props)

[MichaelSimons]

[Triage] Unfortunately as noted in the preview 1 release announcement, the current state requires a rid specific prebuilts (not including the SDK or prevously source-built packages) that distro maintainer need to provide themselves. This is to say we cannot provide the full set. Ideally the best solution is to get to zero prebuilts as quickly as possible to address that problem.

We can start attaching the SDK, artifacts and prebuilts to the release. This does require some changes to our CI process so that it produces and captures a bootstrap version of the artifacts.

The following is the work I see here.

• Modified build.sh to take in a url and sha for the source to use by sourcelink. If not provided require the build is executing in a git context and read the values from it.
• Update the release pipeline to no longer generate the source tarball and attach it to the release. We will use the sources attached by default.
• Factor out the logic to generate a bootstrap artifacts from the prep script into a standalone script. Invoke this at the end of CI so that we product the bootstrap artifacts with every build.
• Modify the release pipeline to upload the new bootstrap artifacts to blob storage
• Modify the release pipeline to attach the SDK, bootstrap artifacts and prebuilts to the release.

[MichaelSimons]

I felt like this issue is being overloaded. I logged Provide release artifacts necessary to perform bootstrap build offline to cover the remaining discussed here. I am therefore closing this issue.