-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: decide policy for sub-repositories #17244
Comments
Nice summary. Thanks for writing this up. I agree with your opinion too. |
I have a very different opinion. I think there is significant value in having a set of "semi-blessed" packages, and random packages that can be found (more or less easily) on godoc.org do not meet that bar. I think one of the big reasons that languages like Python have taken off is that they are "batteries included"; their ~standard libraries include a wide swath of functionality, such that the average program can be written without adding any third-party libraries. If anything, I think more should be included than we have now. I think the x repos today have already improved on the "batteries included" model, by making it clear that "these are good packages, but they're not great packages, and we don't make as many promises about them". I would actually like to see a future where we include a snapshot of the x repos in release tarballs, attaching the same disclaimer. I see your proposal as directly advocating for fewer packages in x. I disagree that that is a worthy goal. That said, I don't think that #2 and #3 necessarily have to live in the same place as #1 and #4. I would be happy to separate the "random part of the Go project" repos from the "semi-blessed packages" repos. |
@quentinmit I think we absolutely do need a way to tell people "these are good packages" and I think that is what @adg was alluding to with his point 4 above. The question is what that mechanism should be. The disadvantage I see with the current golang.org/x system is that these packages are effectively owned by the core Go developers. That isn't scalable or even desirable. If we can solve that problem within the golang.org/x name space, then, fine. But how? And who controls the golang.org/x name space? I think it would be better to have a scalable mechanism for identifying and maintaining good packages, and I don't see why that should be, or even can be, associated with golang.org/x. |
Great summary of the issue. What @bradfitz said. |
As a consumer of many third-party packages, I think I agree with @quentinmit (or at least with @ianlancetaylor). If I need a package for $task and there is an /x package for $task, I've generally found that while it might be incomplete, it's usually of high quality (the scope and style matches the standard library; the test coverage is quite good; the documentation is thorough and concise). If I have to go browsing through $task packages on godoc.org and pick the best one, it's often of such poor quality that I either do an extensive cleanup/refactoring/testing/docs pass before bringing it into my repo or else I choose to write my own package instead. @adg said:
but I'm skeptical that a process outside the Go team will produce a quality bar that is usefully high for me. I'd like to see [high-quality package identification mechanism] exist and be proven to work well before any trimming down of /x. |
I'm advocating that we fix this problem.
The "Go team," such as it is, is only a handful of people, and they do not have the cycles to review any new packages. If you want more packages in x/ then they must be written, reviewed, and maintained by other people. That being the case, there's no reason why they should be in x/. They're not going to be any better or worse for it. |
@quentinmit said:
Both Python and Go have suffered from having large standard libraries, too. In some cases, the standard libraries have become out of date and superseded by other packages in the ecosystem (or in the case of Python's urllib, by urllib2 in the same standard library!). Then we're left with prominent packages that are not suitable for common use. That's a net loss for our users. If I understand your sentiment, you're advocating for including (a lot) more packages in the x/ repositories, as an extension to the standard library. Producing our current standard library (not the sub-repositories) was a massive concerted effort over many years, by people who are mostly domain experts in the functionality of those packages. I have two main questions:
|
From my perspective, being able to work under an x allows me to reach out to the go approvers and this is what I care when I am designing ecosystem-wide core libraries for a niche topic. I don't think it is possible once the experimentation is not considered a part of the Go project. |
To expand a little on my goals here: by defining what does or does not belong in x/, I hope to remove bureaucracy. Today, if someone wants to put something in /x/exp they need to ask permission. People shouldn't have to ask for permission to experiment; that has a chilling effect on experimentation. Similarly, if something is to be included in an /x sub-repo, they need to ask permission. Why should they? Shouldn't they be able to put their high quality code in a repo somewhere else, and have it be regarded just as highly as anything in /x? Code should be judged on its own merits, not by its location. |
I'm not sure I think I would agree with "a lot", but yeah, I think x/ should perhaps be twice the number of packages it is now.
Go project members will do the work, just as they do today. The people who are proposing these new packages are working in their area of expertise. But I also don't think it's "outside" our areas of expertise; our area of expertise is designing idiomatic Go APIs, and that can be equally applied to something like video compression as it can to XML or JSON marshalling.
That's a hard question. For me, I have a soft definition of "low-level" support packages that I think deserve to be centralized. To use an example I am personally familiar with, I think video and audio formats should be in x/video or x/media, ala the current image support we have in x/image. I think a streaming server built on top of those libraries, or a GIMP clone, does not belong in x/ because that is not something foundational. I do think making high quality third-party packages easier to find solves a large part of this problem, but we are not there yet. And even if we do make them easy to find, packages that are separate from Go still have more adoption problems we can't possibly solve: for example, they may be under various licenses, and it may be cumbersome to import into a commercial codebase. That may be appropriate for a library to interface with a Siemens PLC from the 1990s, but I don't think it's appropriate for OpenGL. |
The reality is that there are hundreds of open CLs (some pending for more than a year) and thousands of open issues. The work is not getting done. The present structure does not scale. To encourage more development that we cannot support is irresponsible and unfair to the contributors.
My argument is that we will never get there if we use the sub-repos as a crutch. |
I think external packages will see more high quality reviews if they are more prominent. Everyone in the ecosystem has an interest in improving the most useful (and used) packages. |
You are mistaking your experience on the high-level community-driven packages with fundamental packages. For example, no one in the community will find debating a low-level audio API on a personal project useful or productive given there is no arbiter or a person who has an authoritative final say. I support structure, your proposal suggests distributed chaos and blessing of the popular because it is more successful. The role of the Go project should be to review and bless fundamental things, especially the APIs. And that's where the proposal system can be sufficient enough. If the Go project doesn't have bandwidth to maintain the proposed package, the proposal should be frozen for bandwidth reasons. The ideas can keep been communicated on the proposal even though multiple implementations can live in people's personal projects. The current state of the x packages is not what I support. But, throwing people out of the project because the core people doesn't have bandwidth is not a solution. No one ever desires a language ecosystem dominated by broken third-party solutions like Node's or Python's, but languages evolve there. I worry that it is happening for Go so soon. |
Also people who are +1ing this proposal who has all the privilege to merge their stuff to Go should better be quieter on this one. This proposal is not going to affect your daily interaction with the project. It is about the contributors who are spending their weekends to contribute to the various parts of the project. We should ensure quality is preserved without discouragement. I would like to hear more from the actual part-time contributors and I am devastated by "+1, ship it" type one-line responses. |
Never contributed, but use packages in /x. I'd rather it be well maintained than well stocked. Frozen packages in the stdlib are annoying enough. |
I don't think that's true. For example, @peterbourgon has had great success with his go-kit project, attracting people to do high level design work. I could be wrong, but I don't think you'll see much difference between participation at "github.com/goaudio" or "golang.org/x/audio". I personally would be just as interested either way.
To repeat what I think you said: the equivalent of package I agree with this completely.
I am sensitive to your concerns about "throwing people out of the project". That's certainly a non-goal, here. But, for the record, nobody has said "ship it" because there is nothing to ship. Brad and Robert have agreed with what I said, but nobody has proposed any kind of policy yet. I too would like to hear from more contributors (or potential contributors), particularly to expand upon the list of reasons that code should live in /x, and where people think the line should be drawn. |
I don't think go-kit a foundational package. It is a very high-level library to do certain things in a certain way and is very successful in its domain. A foundational package is a core network protocol implementation, a core abstract (io package abstractions), an open standard implementation, an industry-wide standard, etc.
I totally agree with that. I think one of the Go project's goals should be prioritization of requirements from the language and collaborating with experts to review designs. This proposal should address that aspect before being finalized.
Well, ok. I am just a bit worried when influential people are on the threads without arguments. Because it influences which side people lean towards in the end. |
I added a 5th bullet to the list of reasons in the first post. |
@rakyll I'm sorry, I don't understand the reference to throwing people out of the project. I hope that nobody is suggesting that. I agree that we have a problem. I've been talking to people about this problem for years. I don't have a solution. I'm not aware of any language that has a good solution. Other languages have varying solutions of varying quality. None seem clearly better than what Go is doing, which is basically godoc.org. I think that golang.org/x is not a solution. I think it's basically exclusionary. It privileges the packages that the core Go team happens to care about, and relegates the rest to limbo. That doesn't scale. But letting all packages into golang.org/x also doesn't work. The end result of that is basically what we have today--an undifferentiated list of packages of varying and unknown quality, but with different package names. What we need is some way to identify quality packages. That is the problem. It seems to me that golang.org/x is not the solution to that problem. But I could of course be wrong. Perhaps golang.org/x is the solution. But not as it exists today; it has to be different. So how do we change it to make it work? |
Thinking of #16218, could CLA clarity as a reason for inclusion in golang.org/x use a callout here? Perhaps that falls under reasons 4-5 already. |
@rakyll Sorry for simply repeating @bradfitz 's and thus @adg 's words. I don't have a solution either, but (repeating what's been said before) as the sub-repos are used now, they imply some form of "approval" by the Go team, yet they have essentially historically grown. In fact, several packages from the std/lib landed in sub-repos (for lack of a better place) when we cleaned up the std repo a few years back. And I have "stuff" in there (x/tools) which perhaps should be elsewhere. I'd love if our sub-repos were blessed somehow and of high quality and maybe even maintained by us (if we could). But it doesn't scale and it's not necessarily even good for the community. I agree w/ @ianlancetaylor that some mechanism to qualify and identify high-quality packages would be a much better solution. The question is how to do that. The Go team has neither the manpower nor the expertise (we don't have domain experts in all areas) to do that at scale. Perhaps part of the solution is to bless a location where such high-quality packages land in the first place, and then come up with a process to rate them. |
I disagree. I spent a significant amount of my time, over a number of months, on design and code reviews for golang.org/x/mobile/audio (later moved to golang.org/x/mobile/exp/audio). This was time I wasn't spending, day to day, on my other work. I spent my time this way because, as @cespare alluded to above, there is a community expectation that anything under golang.org/x is of high quality. I was also actively working on golang.org/x/mobile at the time, and a number of people have said that they'd like an audio or audio/visual or media Go package to have pluggable codecs the way the standard library's image and image/* packages works and, well, I designed and wrote those image packages. You say that "being able to work under an x allows me to reach out to the go approvers" and sure, that's a benefit for you, but it is a cost for the Go approvers. If it's easier for everyone to demand @bradfitz's attention for anything potentially under golang.org/x/net, or @ianlancetaylor's attention for anything potentially under golang.org/x/sys, then that's less attention that they could spend on e.g. landing HTTP/2 support, or keeping gccgo up to date. It is not a free lunch, and as others have said above, it doesn't scale. It's true that golang.org/x/exp/... (and similar paths like golang.org/x/mobile/exp/...) have lower quality and support expectations than the rest of golang.org/x, but my opinion is that the right response is to move those packages out of golang.org/x, rather than add more exp packages. |
I largely agree with Quentin here. In my experience, having a solid base of fundamental libraries is key to being productive in a language. Look at Java, Objective-C/Swift, .Net. All have comprehensive standard libraries. adg's reasons for inclusion have a lot of positive sides too:
I understand that the Go team does not have enough cycles. But in my opinion, a blessed package model doesn't scale for users and will hamper Go adoption. The solution, I think, should come from scaling the subrepo approach, maybe by using a hybrid form. Here are some thoughts:
To have a good quality list of "blessed" packages will either be also a lot of work (verifying APIs of probably more packages than you would normally include in an x subrepo), or it will inherently mean it will be of lesser quality than an official list. If it is of the same quality, it still can never instill as much trust as confidence as an "official" list, unless we include "blessed" packages in an official release, which seems even more work to me. Even worse, never having had "official" subrepos is one thing, but giving them up is a clear signal of throwing in the towel. So to me, trying to reduce/spread the burden of maintaining golang.org, rather then shedding it, seems crucial. |
The mobile package is planned and executed by this team as an experiment. The entire thing should have never been under /x. As this proposal calls out, there are a few packages under this category and should have been bootstrapped elsewhere, e.g. how bradfitz bootstrapped http2 outside of the tree.
I thought this discussion already acknowledged the bandwidth issue. What @adg asks for is how others are affected by such a policy and it is very critical to understand what a proper solution to community-driven packages might look like. I mentioned "being able to work under an x allows me to reach out to the go approvers", because it is a necessity. If people were able to escalate a couple of API sanity questions to a group of people they trust, they wouldn't care where they bootstrap or maintain their packages. This is an actual requirement, please don't dismiss it by repeating what we already know. And that's not a benefit for me. It is a benefit for the language to allow different people collaborate on a medium and escalate things about API review. adg mentions "/x" being prestigious but I disagree. In my unique experience, /x is a very invisible point. There is no true benefit someone is contributing a package to /x if they don't care about the quality or design feedback. Before I finish, I want to say I really appreciate what @ianlancetaylor and @griesemer suggests. We may need a successor plan or strategy and think about the indicator of quality. The ideal location where these packages bootstrapped or maintained might not be /x, but organizing the distributed effort towards enhancing the Go library-space should be a critical project on its own. |
Regarding the reachability of Go developers and comfort of using the typical Go tools for development, would it help if I finished up my @LetsUseGerrit bot so any repo on Github could use the Gerrit code review process? |
I think Gerrit might not be critical given Github PRs are becoming better. How people reach out to quality reviewers might be a more interesting problem. Would it be a reviewers mailing list? If so, how to communicate it is entirely voluntary work? Should it come from the Go project or driven by the community? I have some ideas but don't have any concrete proposals at this point. |
My (subjective) impression is that the burden of being responsive and reviewing code is burning out core team members. Is that true? If so, I think it's a significant factor for consideration. |
Summary: a search engine that finds and ranks Go projects by code quality is being called for. |
Rust's bug tracking discoverability and ranking: rust-lang/rfcs#1824 |
As noted above, this issue is basically on hold for package management to settle. I think a lot of the pressure on the x repos is caused by bad broader support for package management. |
Taking off hold to see if we can make any progress on this as 1.13 rolls out, as prompted by a comment on #19700. |
I'd like to find resolution on this topic, at least for a time. I see the current categories of content in the x repos being:
I think the effective policy these last few years has been:
In light of the effective policy, issues on hold for this issue should be rejected unless they meet that criteria. Secondly, existing repos such as x/exp and x/mobile should be considered to be moved outside or explicitly marked as grandfathered in as part of the policy. The x/time/civil proposal does not meet these policies, I would expect it and the CL to be closed. |
@kardianos, thanks! That looks pretty accurate. Keep in mind that a major reason (perhaps the only one) for having x/exp not be on GitHub already is for the same Gerrit code review tool, same builders/trybots/etc. So if we moved it, we'd need to decide whether we're just changing its import path (e.g. As for x/mobile, it was originally meant to be official, and for some time had official staffing. It wasn't until some reprioritizations happened that the Go team stopped working on x/mobile. It might do more harm than good to move it at this point. |
There is a great need for standard libraries, but they do not need to be done entirely by the Go team. If you increase the size of the Go Team, you also increase internal communication and decrease productivity. I am very much in favor of the divide and conquer approach. You do not need more people, but you do need more teams, SIG they are following standards and are producing reliable code that everyone can use. BindData was a very useful package, so I vendored it and wrappered it, one day it was gone. Not a problem for me, but an outcry from the community. I love using other peoples packages, but for my clients, I only do so after vendoring and code review unless the package is extremely popular and the maintainers well respected. I would respectfully suggest that when someone has a mature package of what they consider high quality, that they be able to submit it to a review process. I have some very specific ideas of how to conduct the review process, track the reliability of the coders and the code reviewers. For me, selecting packages to use is a painful process, that should not be the case. |
[this is not a proposal yet, just a problem statement]
There are several proposals to add new packages to the
x/foo
"sub-repositories". (See: #16353, #13432, #12332, #16257, #15985, #15406, and more.) The proposals are all different, but they circle around the same issue: what should and should not be in a sub-repository.Currently, the sub-repositories contain x different things:
To me, it's clear that 1-2 belong in the sub-repositories, and should be an official part of the Go project.
For 4-5, this code only lives in the sub-repositories either for historical reasons or mere convenience. The Go contributors are accustomed to the process (tools and reviews) of writing code in that style, so the sub-repos were just the obvious place to go.
(3 is a fringe case, as the tools could be worked on elsewhere, but the people working on them work on Go full time and work on that code solely to support the project, so I am inclined to ignore this code for the purposes of this issue.)
As the project grows, it attracts people who want to contribute new packages to these sub-repositories (see the issues cited at the beginning of this issue). The reasons for inclusion include:
[What are the other reasons? I'd like to enumerate them all here.]
My opinion begins here:
None of the above reasons are good arguments to add new packages to the sub-repositories. Taking each in turn:
[see discussion below for other points]
In general, I'd like to make the Go project smaller rather than bigger. (In the same sense that we would remove stuff from the standard library, if we could.)
As I said, this is just my opinion and perspective. I have created this issue to gather feedback from other [potential] contributors.
Personal opinion ends.
We need to set a policy for what belongs in the sub-repositories, and in doing so more clearly define the boundaries of the Go project.
The text was updated successfully, but these errors were encountered: