clarify Go support policy for secondary ports

[ianlancetaylor]

Background

This proposal started out as a GitHub discussionn at #53060.

Go supports a number of different GOOS/GOARCH targets. We've defined a policy for adding new ports, described at https://go.dev/wiki/PortingPolicy.

Ports are divided into first class ports and secondary ports. The current first class ports are:

• darwin/amd64
• darwin/arm64
• linux/386
• linux/amd64
• linux/arm
• linux/arm64
• windows/386
• windows/amd64

The current secondary ports are:

• aix/ppc64
• dragonfly/amd64
• freebsd/386
• freebsd/amd64
• freebsd/arm
• freebsd/arm64
• illumos/amd64
• linux/ppc64
• linux/ppc64le
• linux/mips
• linux/mipsle
• linux/mips64
• linux/mips64le
• linux/riscv64
• linux/s390x
• android/386
• android/amd64
• android/arm
• android/arm64
• ios/arm64
• ios/amd64
• js/wasm
• netbsd/386
• netbsd/amd64
• netbsd/arm
• netbsd/arm64
• openbsd/386
• openbsd/amd64
• openbsd/arm
• openbsd/arm64
• openbsd/mips64
• plan9/386
• plan9/amd64
• plan9/arm
• solaris/amd64
• windows/arm
• windows/arm64

The core Go team maintains the first class ports. It is less clear how the secondary ports are handled.

The existing porting policy says:

• Every secondary port must have a maintainer who will make required updates in a timely manner.
• Every secondary port must have a builder, and a person who maintains that builder.
• If the builder for a port is failing for more than two weeks, a new maintainer is needed.
• If a builder fails for more than four weeks or is failing at the time of a release freeze, and a new maintainer cannot be found, the port will be removed from the tree.

However, in practice we do not follow those rules.

• It has not been clear who the secondary port maintainers are, and there was no mechanism for adding or removing maintainers; this was recently improved by creating the @golang/port-maintainers GitHub team and subteams.
• When a change requires an update to the port, it is the person making the change who takes responsibility for updating the port, not the port maintainer.
• Maintainer responsibilities are in general unclear.
• Several secondary port builders are maintained by the Go team.
• We do not remove ports from the tree if they are failing for four weeks (we do remove ports if the OS is no longer supported, such as darwin/386).

The effect is that the work required to maintain secondary ports falls on people who are not familiar with those ports. This was not the goal of the porting policy, and it tends to slow down overall development of the core Go systems and discourages the adoption of new secondary ports.

We propose both loosening and tightening the current porting policy to address these concerns.

Proposal

• The Go team will stop publishing binaries for secondary ports.
  • The Go team currently publishes binaries for the following secondary ports, which we would stop doing if we adopt this part of the proposal:
    • freebsd/386
    • freebsd/amd64
    • linux/ppc64le
    • linux/s390x
    • windows/arm64
  • We encourage port maintainers to publish them instead; we can link to download pages from https://go.dev/dl as appropriate.
  • We will investigate building binary releases by cross-compiling to secondary ports from a first-class port. This may be feasible by doing pure Go builds, while leaving any cgo builds to be on the user's machine using the user's C compiler.
  • Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.
• Require at least two maintainers for each secondary port.
  • If maintainers resign or become unreachable and can't be replaced, we mark the port as broken; see below.
  • This means that for some existing secondary ports we will have to find additional maintainers. As I write this I think that additional maintainers are needed for android, dragonfly, illumos, ios, js/wasm, openbsd, and solaris.
• In general developers will not be required to make their code work on every secondary port.
  • They are encouraged to do so if it is straightforward.
  • If not, they are encouraged to notify port maintainers about pending requirements as soon as feasible, and to work with the port maintainers to get the ports working again.
  • That said, ultimately the port maintainers are responsible for keeping their ports working.
• Notify maintainers when a port breaks.
  • Open an issue, CC'ing maintainers
    • Perhaps also send an e-mail to golang-dev.
  • We can do this preemptively if we know that some change is going to break a secondary port.
  • Ideally this is an automated system, but for now it may be manual.
    • For example, perhaps the system that creates https://build.golang.org/ can send e-mails about all failures to a newly created list. Maintainers can filter that list for the ports that they care about.
    • Currently maintainers can track build dashboard triage log #52653 to see problems.
• Clarify maintainer responsibilities on the PortingPolicy wiki page.
  • A GOOS maintainer is responsible for any _GOOS files, any files with just that build tag, and blocks of code guarded by if runtime.GOOS == "mygoos".
  • Similarly for a GOARCH maintainer.
  • If technically possible, GOOS/GOARCH maintainers who are not already approvers should be given the right to +2 changes to files for which they are responsible.
    • These CLs will still require a +1 review for style and sanity but not for detailed correctness; we will defer to the maintainers for that.
    • Note that with two maintainers one of them can +2 changes by the other.
• We will introduce a new concept: the broken port.
  • The existing policy says that problems must be fixed within 4 weeks, which is not realistic; people are busy.
  • If a port stops working, including the case where a builder stops working, we can decide to mark the port as broken.
    • Or in some cases we can roll back the change that broke it; this is a judgement call.
  • In general, a port can be considered broken if its builder has failed multiple times in a development cycle with a failure mode that does not occur for first class ports, and that failure mode is not believed to have been fixed or suppressed by a change in either a Go repository or the builder's configuration, and maintainers are not actively working on a solution.
  • Any approver can declare that a port that meets these criteria is broken.
  • The list of broken ports will be maintained in cmd/dist and will appear in the release notes if broken at release time.
  • Attempting to build a broken port will fail unless make.bash or go tool dist is invoked with a new option -force.
  • If a port is broken in release 1.N, then the core Go team can choose to remove the port from release 1.N+1.
    • This is not obligatory and will depend on whether anybody is willing and able to maintain the port going forward.
  • We currently have a list of incomplete ports in cmd/dist/build.go; these should probably be treated as broken ports.
    • Currently the only entry on that list is linux/sparc64.
  • The goal here is not to get ports out of the tree; if people are actively working on the port they should have as much as latitude as possible to fix it.
• The default set of trybots will change to only cover first class ports.
  • The https://build.golang.org/ page will continue to show all ports.

Discussion

This is not intended to be a big change to the current process. However, it is intended to be a change. It is intended to take some of the porting load off of the core Go team, while making it easier for port maintainers to make changes. It is intended to make it easier to add new ports to the tree.

In the long run it would be good to support out of tree ports. However, that requires a bunch of technical work, and there is no design for it.

[erifan]

In general developers will not be required to make their code work on every secondary port.

Does this mean that we can only consider the first port when adding new features in the future, such as regabi.

And will we move a lot of secondary port related code out of the master branch, so that our code can work without regard to these ports.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[ianlancetaylor]

@erifan

Does this mean that we can only consider the first port when adding new features in the future, such as regabi.

I think it depends on what you mean by "consider." I think an implementation for anything that covers all GOARCH values but differs for each one must as least consider whether and how it can be implemented for each GOARCH. We shouldn't add features that can never work for some GOARCH. And the initial implementation should not break any other GOARCH. But it's OK to implement something like regabi one GOARCH at a time, as indeed is being done for regabi.

And will we move a lot of secondary port related code out of the master branch, so that our code can work without regard to these ports.

I am not proposing that at all. That would be what I call an "out of tree port." I think it would be great to support that, but that is very hard, and it is not this proposal.

[erifan]

Thanks @ianlancetaylor , I see. This sentence "This is not intended to be a big change to the current process. However, it is intended to be a change" is very accurate.

[beoran]

@ianlancetaylor

The core problem with porting Go to other platforms, and making out of tree ports now is that the go runtime, standard library and compiler are not very modular when it comes to operating system and architecture support. Now, packes have files for all platforms confounded are in the same package using build tags. I think this should be refactored and split up into packages per operating system and/or architecture.

One should be able to write a platform support for the Go compiler, runtime and std lib just by writing a module which the compiler , etc. could then import.

If course this would involve a serious refactoring of Go, but I think it will be for the better, and encourage third party ports.

[ianlancetaylor]

@beoran Thanks. I agree with all of that. But this is not that issue. The changes proposed here should not have to wait for the changes that you suggest. The changes you suggest will take a long time to design and implement, and nobody is working on them.

[beoran]

@ianlancetaylor Well I wanted to bring it up but I agree it is a separate issue in the end.

[aclements]

@beoran , we are making baby steps toward better modularization in the runtime by moving the per-OS/arch syscall layer into runtime/internal/syscall and reorganizing it to have significantly less per-arch code. We're part way through migrating the Linux arches (we've migrated the raw syscall functions for all Linux arches, but not yet the Go interface). It's a small step, but should help some.

[rsc]

I think the discussion helped a lot with addressing most of people's concerns, but I'm still a bit surprised about how little discussion is happening here. Does anyone have any remaining objections that we should discuss?

[qmuntal]

Note: when and if reliable testing hardware is available, the Go team is likely to promote windows/arm64 to be a first class port.

Microsoft dev here. We would like to help promoting windows/arm64 to be a first-class port. If it is just a problem of having reliable hardware, we can discuss providing reliable windows/arm64 hosts running in our own infrastructure. Would that be enough? If not, what else would be missing?

[ianlancetaylor]

CC @golang/port-maintainers

[ianlancetaylor]

CC @golang/release See comment by @qmuntal above regarding reliable windows/arm64 hardware. That should probably be discussed on a separate issue. Thanks.

[paulzhol]

..but I'm still a bit surprised about how little discussion is happening here.

I'm not affiliated with any company (just a FreeBSD user). Trying not to offend anyone here but there is not much in a way of open items up for discussion.
The core Google Go team doesn't wish to maintain the secondary ports and would like to have it done by other parties.
Unfortinetly to me the conditions above look like part of a contract or SLA you'd sign with another company providing development services instead of volunteer members of the open source community.

Additionally maintaining the official (because they are also used to build the Go release binaries) enviornments for freebsd/386, freebsd/amd64 outside of Google is extreamly frustrating.
It entails waiting for a member of the release team to run a bunch of bash scripts under a Linux VM to create and deploy new GCE images every time a new OS version is updated. This can sometimes take more than a year due to their limited avilability.
As for freebsd/arm, I don't plan to invest in new hardware after already upgrading twice. Maybe I'll be able to get a spare rpi3b to pass the build - but I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

[beoran]

@aclements That's a great first step, and i am all in favor of that.
@paulzhol It sounds like maintaining a secondary port is pretty labor intensive and /or inconvenient. Maybe that too is something that should be improved?

[ianlancetaylor]

@paulzhol

The core Google Go team doesn't wish to maintain the secondary ports and would like to have it done by other parties.
Unfortinetly to me the conditions above look like part of a contract or SLA you'd sign with another company providing development services instead of volunteer members of the open source community.

You're not wrong. But I think there is another perspective. For a project like Go that has millions of users and that aims to provide a very high level of stability and reliability, it's not fair to our users to say "here is a port that may work, we don't know." We want to say either "this port works to the best of our knowledge and ability" or "you are on your own, good luck." To millions of Go users, we are in fact "another company providing development services." And to treat those users well, we have to carry that attitude through all core Go development. So, yes, we depend on volunteer members of the open source community, but we have to be clear about what they, and the core Go team, can and can't promise to Go's users. Again, you're not wrong, but that doesn't mean that nothing about the current porting policy should change.

I do think that there are things up for discussion, like: will this wind up hurting Go's users and the Go ecosystem? Obviously, I don't think it will, or I wouldn't have proposed it, but I could certainly be making a mistake.

Additionally maintaining the official (because they are also used to build the Go release binaries) enviornments for freebsd/386, freebsd/amd64 outside of Google is extreamly frustrating.

I'm sorry to hear that. I don't know much about what is required, or what would have to change, but I really hope that we can make things better somehow. Separately, I don't mean to be facile but I'm not sure that this proposal affects that one way or the other.

I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

Can you say more about that? Thanks.

[beoran]

@ianlancetaylor If this proposal makes it harder for ports to be available then this will damage the community.

Maybe I am staying the obvious, but would it be possible for Google to invest a bit more in this project and hire a few more porting engineer and provide some more hardware for them? That seems like a more optimal solution.

While this is an organizational issue, the underlying problem also seems to be technical. It seems to me that investing some work in improving the tools for ports would already help to alleviate the work needed to maintain a port. So it seems worth while to also discuss this here.

[ianlancetaylor]

Maybe I am staying the obvious, but would it be possible for Google to invest a bit more in this project and hire a few more porting engineer and provide some more hardware for them? That seems like a more optimal solution.

Unfortunately I can't see that happening.

While this is an organizational issue, the underlying problem also seems to be technical. It seems to me that investing some work in improving the tools for ports would already help to alleviate the work needed to maintain a port. So it seems worth while to also discuss this here.

I think it's worth understanding how we can make it easier to maintain a port.

I'm not sure it makes a difference one way or another to this proposal. What do you think could or should change in the proposal?

[beoran]

@ianlancetaylor With regards to this proposal:

1. Seeing the current difficulty in maintaining a secondary port. I think the main Go project should provide as many resources as possible.

2. The main Go project should be a bit more lenient on these secondary ports. While it is true that ideally, all ports should be of equal quality, in practice this doesn't seem to be possible right now. The people who need such secondary ports are likely to accept that they have to do more testing themselves. The value of the secondary ports is in them existing, and allowing users on such systems to keep using Go.

[ianlancetaylor]

Seeing the current difficulty in maintaining a secondary port. I think the main Go project should provide as many resources as possible.

I agree that that sounds good, but I don't know what it really means in practice. For example, I think we would all like it if Google would hire a full time person to do nothing but work on ports. But nothing like that is going to happen. On the other hand, Google is already prepared to donate significant Google Cloud Platform resources to ports, but of course that doesn't really help for ports to GOARCH values that GCP does not support.

One of the things I'm trying to do with this proposal is get more specific. What can we really promise beyond "we'll do our best?" What can people count on us to do beyond "do as much as possible?" What is possible?

The main Go project should be a bit more lenient on these secondary ports.

I'm already trying to do that in this proposal. The proposal is explicitly more lenient than the current porting policy, which says, for example, that a port will be removed if it is broken for four weeks.

[tuxillo]

2. The main Go project should be a bit more lenient on these secondary ports. While it is true that ideally, all ports should be of equal quality, in practice this doesn't seem to be possible right now. The people who need such secondary ports are likely to accept that they have to do more testing themselves. The value of the secondary ports is in them existing, and allowing users on such systems to keep using Go.

Fully agree here. I, user of one of the systems which would fall into the secondary port category, would greatly benefit of having a "best effort" approach for it rather than a "save yourself" approach because some projects just don't have the required resources to keep up with such effort, which will be (most likely) a big one.

On the other hand, if there are no new hires in the horizon to support the current burden that means that the burnout within the Go team will increase and that surely helps no one either, hence, some changes in the compromise towards those so-called secondary ports will be required. So, it seems things aren't going to get any easier in this front either ...

[dmgk]

I'm generally OK with this proposal, one thing that is somewhat bothering me is

The default set of trybots will change to only cover first class ports.

Does this mean that there will be no usual pre-commit checks performed on secondary port builders? That could easily lead to a situation when innocuously looking commit breaks a secondary port and maintainers will have to fix it after the fact. I'm not sure this will be manageable by volunteers.

[ianlancetaylor]

Does this mean that there will be no usual pre-commit checks performed on secondary port builders? That could easily lead to a situation when innocuously looking commit breaks a secondary port and maintainers will have to fix it after the fact. I'm not sure this will be manageable by volunteers.

That is a fair point. There is some discussion of introducing a submit queue, which would address this problem: submitting a CL would not submit it directly to the repo, but would instead run it through more comprehensive tests and then submit if those tests pass.

Perhaps we should drop or delay that part of the proposal until the submit queue is created.

[mengzhuo]

I'm OK with this proposal.
Just a nit: There are ports are maintained by company/full-time developers like arm, ppc, loong, windows.
Could we show this information on go.dev/dl and "community support" like freebsd, plan9, riscv ?

[paulzhol]

@beoran

It sounds like maintaining a secondary port is pretty labor intensive and /or inconvenient. Maybe that too is something that should be improved?

Yes, I mainly find myself spinning up and maintaining multiple setups to test my code and verify cgo -godefs output on all the supported FreeBSD GOARCHes before submitting changes. It sometimes can take longer than the actual development.
Recently I've learned of TRY=freebsd-386,freebsd-amd64,freebsd-arm64,freebsd-arm. It with mentioned submit queue would help, but it still not a substitute for actual CI machines. Maybe allowing full gomote access could address that.

@ianlancetaylor

I'm sorry to hear that. I don't know much about what is required, or what would have to change, but I really hope that we can make things better somehow. Separately, I don't mean to be facile but I'm not sure that this proposal affects that one way or the other.

The proposal starts off with dropping official releases for freebsd/386, freebsd/amd64. I reasoned that means the port maintainers will have access to some GCE project/credit so we could provide more frequent builder image releases based on the upstream project's GCE images instead of the qemu-inside-Linux being done now and maintained by the release team. I may have misunderstood.

I think there's a huge difference between embedded plaforms with limited RAM and "full" builders which is not addressed by the proposal.

Can you say more about that? Thanks.

These are much more constrained environments. To limit sdcard wear, the root filesystem is usually mounted read-only (at least for my builder). There's much less available RAM (in total and per-core) which is shared between the running Go toolchain and the page cache for both build artifacts and go list walks.
On my builder I use spinning rust over iSCSI as scratch workspace and swap - this adds additional load on the networking stack causing network tests to be flaky.
Additionally the slow IO coupled with the generally much slower CPU cores introduces behaviors which are dependent on the kernel scheduler and its preemption of user threads in the middle of profiling tests (making those flaky as well).
There is a way to address it I think, by splitting the bootstrap+compilation and just running of actual tests on the builder. I think it is/was done on the IOS builder?
But maybe it should be the norm for secondary builders on embedded hardware (tier 3 if you will). Preferably without the requirement of each port maintainer to have to roll their own version of it.

[beoran]

@paulzhol Thanks for providing concrete pain points and suggestions. I think it is important they are brought up here.

@ianlancetaylor It kind of worries me that Google does not seem to want to give more resources to the Go team. I hope it is not a bad sign of things to come. Dropping official support for several platforms is not going to help improve the future of Go language, and that is what I fear this proposal may lead to.

As we can see from some replies above, the burden for porting Go is also technical, I would say, even due to technical debt. I stated this before on occasion, but I feel that there should be release of Go, say, next year, with no new features at all, which focuses apart from bug fixes on fixing long standing issues, ease of porting, and lessening this technical debt. After such a release, then with the Go compiler and runtime in an optimal shape, would be a better time to start devolving the secondary ports and advance this proposal.

[rsc]

@laboger That sounds like an interesting idea. Thanks. I filed #54375.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[ianlancetaylor]

I've updated https://go.dev/wiki/PortingPolicy with the changes described in this proposal.

[bcmills]

@ianlancetaylor, is there anything remaining to do for this proposal? (Can it be closed as complete?)

[ianlancetaylor]

What remains here is completion of #53862, and adding the notion of a broken port to cmd/dist (along with a way to build a broken port).

[ianlancetaylor]

Filed #56679 to track broken ports.

That completes the work for this proposal. Thanks to all the commenters.

[ernado]

@ianlancetaylor can you please clarify:

1. What is a process for becoming a port maintainer?
2. How can maintainers publish a binary for port? (x/website: add linux/riscv64 downloads #59113)

I'm currently working on bringing riscv64 to Kubernetes (and friends), some dependencies rely on Docker Official Image packaging for golang, which gets binaries from https://dl.google.com/go and I'm not sure that maintainers will accept unofficial binaries.

Thank you!

[tianon]

Speaking as one of the maintainers of that image, we do support non official builds (there's a case statement in our build that will either download or build from the official sources); the problem with riscv64 is support in mainstream distros (still edge only in Alpine, still ports only in Debian, for example).

[ianlancetaylor]

@ernado

What is a process for becoming a port maintainer?

See https://go.dev/wiki/PortingPolicy. Basically, at least for an existing port, file an issue and get the current port maintainers to agree.

Our hope is to be able to publish binaries for all supported ports. See #53862.