proposal: all: decouple standard library from runtime/compiler

[aykevl]

At the moment, the Go standard library is strongly coupled with the particular compiler in use. This is problematic for other compilers that would like to use a different runtime. For example:

• The reflect package is bundled with the standard library, while the specific RTTI format may differ between compilers. Other compilers will have to override this package to implement their own.
• The time package calls runtime.nanotime directly, runtime.nanotime is not exported by the runtime, of course, but the time package calls it with some //go:linkname magic. This is not a public nor stable API. Compilers that use a different runtime will need to keep track of such changes and play catch-up every Go release. It would be much more useful if there was a more-or-less stable interface that is at least well documented.
• A compiler must use one of the supported GOOS/GOARCH combinations in build tags, otherwise the standard library will fail to compile. For example, if you use GOOS=none or GOARCH=avr for bare-metal targets, many standard library packages will fail to compile.
• Go uses a special assembly syntax only supported by gc. The rest of the world uses standard assembly formats leading to friction (see c2goasm, for example). This problem comes up in a few places, for example many functions in the math package are implemented in assembly, even though they might just call fallback functions written in Go. A new Go compiler would either have to parse this custom assembly format with uncommon calling convention, or provide the necessary symbols that call through to the fallback functions.

This close coupling is painful for Go compilers that try to do something unconventional. For example:

• TinyGo compiles Go code for bare metal microcontrollers. Because the standard library doesn't support it, it pretends to be linux/arm for all baremetal targets which is not optimal but ~works. TinyGo also overrides a select list of standard library packages (reflect, runtime, os but not os/user). This feels like a hack. (Note: I'm the original author of TinyGo).
• GopherJS had to resort to hacks to get the standard library to compile by supplementing and overriding some symbols on the AST level.

I would like to propose to somehow separate the standard library from the compiler and runtime implementation. For example, the platform- and compiler-independent bits of the standard library could be shipped as a new Go module while packages that are closely related to the compiler/runtime could be bundled with the compiler. This may not only be useful for other compilers, but also for more invasive changes to the standard library in the future.

This is not a new proposal. @ianlancetaylor has proposed something like this before and I support this, although I might disagree with the exact list of core packages in this proposal.

[mvdan]

I believe @bradfitz had similar ideas in the past for the http-related packages, since the need to move faster pushed the development onto the x/net repository.

I think this is a great idea and it should happen in the long run; my only worry is making the process smooth and well executed.

I also imagine @bcmills @ianthehat @rsc will have some strong opinions here.

[networkimprov]

Another feature in this area is to let the runtime and stdlib support third-party compilers implementing other languages, as with Kotlin & Clojure in the Java ecosystem.

[4ad]

As someone who works on custom Go compilers and special environments like embedded software and kernel-level software, I fully support this.

[thepudds]

For example, the platform- and compiler-independent bits of the standard library could be shipped as a new Go module while packages that are closely related to the compiler/runtime could be bundled with the compiler.

Partially related discussion in #30241 and golang-dev thread, which focus on the creation of two special modules std and cmd in 1.13.

[ianlancetaylor]

As an idea, I support this. But as always the details are complex. To make this happen will require more than expressing a desire: it will require a plan for an end goal and specific steps to reach that goal.

[rsc]

In the abstract, separating these pieces sounds like a good idea, but the concrete details matter and their costs may outweigh the benefit. What this proposal needs specifically is concrete detail about exactly what to do to achieve this idea, what the implications would be for other parts, and so on. This separation would be a major change so there would be a lot to consider.

For example, the first concern I would have is how this interacts with the release process and making sure that Go releases remain well tested. If the runtime/compiler and standard library have to be released separately then there is twice as much testing, more combinations to worry about, and so on. That's an example of a cost to doing this. There are more.

If the motivation is TinyGo and GopherJS, then perhaps a narrower focus would be equally helpful. If there are specific packages that could be made easier to drop into other systems (for example by moving code between files and/or adding appropriate build tags), that is much simpler with lower cost.

It's hard to believe the runtime package in particular would ever be separated from the compiler, but maybe the rest could.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[networkimprov]

Probably should be re-opened and tagged NeedsInvestigation instead of WaitingForInfo.
@bcmills @rsc

[ianlancetaylor]

I think leaving this closed is correct for now. We don't need issues open for abstract ideas. Those kinds of issues just sit around untouched for years without adding any value to the project.

[justinclift]

Hmmm, it's not really an abstract idea. It's something that's really desirable to get thought through then implemented. Can't really do that if it's been closed. 😦

[ianlancetaylor]

Experience shows pretty clearly that the issue tracker is not a good place to think through a complex subject. The biggest problem is the lack of threading, and a secondary problem is the fact that when there are many comments GitHub hides the ones in the middle. Without a specific focus we see that conversations on the issue tracker spin off and become impossible to track.

[aykevl]

I agree there is no concrete proposal yet. I'll work on one when I get to that again...
But to reply to a few specific concerns:

If the runtime/compiler and standard library have to be released separately then there is twice as much testing, more combinations to worry about, and so on.

This is not my intention. Basically everything would mostly stay as it is. In particular:

It's hard to believe the runtime package in particular would ever be separated from the compiler, but maybe the rest could.

No, that's not the intention. There are basically 3 components here, (over)simplifying a bit:

• The compiler.
• The runtime. With this I mean the packages runtime, reflect, sync, and various internal packages used by the net and os packages related to scheduling. Also: the implementation of time.Sleep that actually lives in the runtime package.
• The standard library, which is all importable packages under the src directory in the source tree excluding the runtime packages.

My proposal would be to define a clearly defined (and perhaps versioned) API between the runtime and the stdlib. The compiler<->runtime interaction would mostly remain an implementation detail, just as it is now. This allows compilers to provide their own runtime implementation while reusing the (rest of) the standard library.

Probably the most difficult part here is namespacing. For example, while the net package is closely coupled with the runtime, the net/http package is not. Same with os and os/exec (I think).

---

I hope to write down a more formal proposal in the future, but hope that this makes it all a bit more clear what this proposal is and is not about.