Announcing the .NET Virtual Monolithic Repository

[premun]

Building the .NET product from individual product repos in the dotnet organization is a complex process requiring orchestration infrastructure that is available only to Microsoft. This is a hindrance to the .NET community as it's currently not possible to easily reproduce the official product build in the open. Furthermore, the complexity of the official build and release infrastructure tends to grow over the years. This presents many reliability and sustainability challenges.

To address the abovementioned issues, we'd like to announce a new repository - https://github.com/dotnet/dotnet - which is the first step on the way to simplify building and shipping of .NET and making the product build process transparent to and reproducible by the community.

This new repository is a Virtual Monolithic Repository (VMR). It includes all the code needed to build the .NET SDK, mirrored from the individual product repos (such as dotnet/roslyn or dotnet/runtime). It also includes source-build, our whole-product build system used to build .NET by our Linux partners.

The VMR has the following main characteristics:

• Monolithic - The repository is monolithic so that a single commit fully identifies the whole source code necessary to build the .NET SDK
• Virtual - It's a mirror (not a replacement) of the individual product repos that make up the whole product, placed side-by-side in one repository. Changes to the underlying individual product repositories are synchronized to the VMR.
• Experimental - Currently, the repository is part of our initiative to streamline the release process, not meant to be depended on. We reserve the right to delete the current instance and create a new, different one in its stead. We expect the repository to be persistent by the release of .NET 8 Preview 1, planned for February, 2023.

The repository is a work-in-progress and has a lot of limitations but we believe that even with these limitations it can be already useful in a variety of scenarios.

🎯 Goals of the Virtual Monolithic Repository

• The main purpose of the dotnet/dotnet repository is to have all source code necessary to build the .NET product available in one repository and identified by a single commit.
• The VMR also aims to become the place from which we release and service future versions of .NET to reduce the complexity of the product construction process. This should allow our partners and and 3rd parties to easily build, test and modify .NET using their custom infrastructure as well as make the process available to the community.
• Lastly, we hope to solve other problems that the current multi-repo setup brings:
  • Enable developers to make and test changes spanning multiple repositories.
  • Fulfill requirements of .NET distro builders such as RedHat or Canonical to natively include .NET in their distribution repositories.
  • Simplify scenarios such as client-run testing of bug fixes and improvements. The build should work in an offline environment too for certain platforms.
  • Enable the standard down-/up-stream open-source model.
  • More efficient pipeline for security fixes during the CVE pre-disclosure process.

📖 Further reading

In case you are interested in knowing more about the why's and how's of VMR's purpose, design and architecture, you may find the following set of documents interesting:

• VMR's Design and Operation
• Upstream/Downstream Relationships
• Code and Build Workflow
• Strategy for Managing External Source Dependencies

The build of the VMR is made possible by the .NET Source-Build.

[mungojam]

did you consider git subtree or submodules for building the virtual mono-repo? I expect you have, so would be interested to know why it wasn't considered the best option

[richlander]

@dotnet/distro-maintainers

[mungojam]

did you consider git subtree or submodules for building the virtual mono-repo? I expect you have, so would be interested to know why it wasn't considered the best option

Actually I see now there is a very clear design doc that covers the possible approaches and why you chose not to use submodules

[svick]

The links in the list of components in the README are wrong: every single one of them is 404, because the URL has github.com twice.

Edit: the problem is that the URL in the source is missing a slash: it's e.g. https:/github.com/dotnet/linker/commit/dc5e60f5f2becf0b462d37ad918443126e80b49b.

[MichaelSimons]

There is something wrong with the links in the list of components in the README: every single one of them is 404, because the URL has github.com twice.

@premun - Looks like github.com is duplicated - https://github.com/github.com/dotnet/linker/commit/dc5e60f5f2becf0b462d37ad918443126e80b49b

[premun]

Actually I see now there is a very clear design doc that covers the possible approaches and why you chose not to use submodules

@mungojam yes, though this document gives you only a part of the answer. It doesn't quite explain why not a subtree (or other things such as subrepo). Outside of merging the repositories into one, we've had more requirements such as being able to filter what files/folders we want to bring in (for instance, we cannot allow binaries to get into the VMR). Part of the decision process, what options we leveraged and why we went with a custom solution is described here.
More details on the synchronization process are here.

[ltrzesniewski]

Looks like the razor-compiler repo should be replaced with razor.

[peteraritchie]

The links in the list of components in the README are wrong: every single one of them is 404, because the URL has github.com twice.

Edit: the problem is that the URL in the source is missing a slash: it's e.g. https:/github.com/dotnet/linker/commit/dc5e60f5f2becf0b462d37ad918443126e80b49b.

Github markdown automatically replaces SHA hashes with a link to the commit. I'm not sure what the heck it does during replacement, but it does the replacement over the invalid URLs (i.e. start with https:/github.com) to come up with a differently invalid URI :/. I've added a PR to fix it: dotnet/dotnet#1

[mmitche]

@peteraritchie The repo is populated/updated via changes in dotnet/installer, including the link generate. So I think the fix needs to be in there. But @premun knows best.

[svick]

@peteraritchie I've looked into it and that list is auto-generated, so to fix it permanently, I've just submitted dotnet/arcade-services#2105.

[mmitche]

Looks like the razor-compiler repo should be replaced with razor.

razor-compiler stayed the same. razor-tooling got renamed

[jaredpar]

razor-compiler stayed the same. razor-tooling got renamed

razor-compiler is effectively archived though. All the functional sources that participate in the SDK are now in razor.

[RussKie]

I'm not seeing dotnet/winforms, dotnet/wpf or dotnet/windowsdesktop in the list. What's the thinking around those?

/cc: @merriemcgaw @dreddy-work @singhashish-wpf

[richlander]

We're starting with the components of the Linux SDK to start. Windows components will come later.

[mmitche]

@RussKie They'll probably get added in towards the end of net8, or at whatever point we get close to being able to build windows.

[RMcD]

Some of the issues addressed around patch management seem to rhyme with solutions I've seen built on https://www.gerritcodereview.com/ maybe room for future collaboration?