proposal: dep: move out of github.com/golang #29639
Comments
|
For some additional context, I tweeted about this today and Sam Boyer replied:
|
|
CC @spf13 |
bradfitz
changed the title
|
Assuming @sdboyer et al want this (please confirm), we can discuss at our next proposal review meeting. We're a bit behind due to the holiday but we'll catch up in the next couple weeks. |
|
|
I don't understand what you mean by this. The source code copyright is held by "The Go Authors", which is the group of relevant contributors. Google does not hold the copyright and therefore cannot transfer it to anyone else. The license is BSD. It already grants permission for anyone to make a copy, make changes, and distribute those changes, with the usual notice requirements. Google is not going to change the license on this code any more than we are going to change the license on the Go project itself. If you want to start a fork of dep, the "source and licenses" are not stopping you.
This is different. Here you are asking that the GitHub project itself be transferred. That's something we could do (while leaving the copyright and licenses alone), but I am not even sure that this is appropriate. Dep both helped us learn what package management in the go command might look like and also still serves as a useful tool until modules are completely ready. I don't see why that wouldn't stay in github.com/golang. And once modules are ready, dep should go away. If others want to fork it and maintain a fork then that's fine. But the official dep that the Go project recommends should probably be the one in our GitHub org. If more people need commit access to github.com/golang/dep we can clearly do that. I hadn't seen the gofrs org until today. I am skeptical of that org because it purports to be (according to the headline at github.com/gofrs) "The Go Community". The only crisp definition of "The Go Community" that I've ever seen is "all Go programmers". Clearly the gofrs github org is a small, self-selected set of developers, not the entire Go community. A small group of 17 people literally claiming to be (or even to represent) the entire Go community is for me a red flag. Why should we trust gofrs to take over dep? |
|
Sorry, I clicked the wrong "send" button; meant to leave this open for further discussion. |
|
@ianlancetaylor @carolynvs @bradfitz Thank you so much for your 👀 on this, and I appreciate you being willing to discuss it when getting back to things after the holidays. |
|
@rsc Thank you for taking time to look in to this and reply. There are quite a few points you've raised, so hopefully I can address them individually. I'm sorry that this ended up becoming a bit more long-winded than I had originally intended, but I think sharing context transparently is the best I can do to help alleviate any concerns that you or others have.
It's probably worth starting with some context sharing on the Gofrs, since I think that might be the more important piece here. It's unfortunate you weren't able to make it to GopherCon this year; I'd have tried to get you to come to my lightning talk[1] where I broke ground on the idea. That said, not knowing me from Adam it might have been a hard sell to get you to attend. 😄 The Gofrs initiative originally came out of the general channel on the Slack Workspace (where we now have over 35.3k registered users). While it started there, it's an effort we're hoping to keep open to the larger Go Community no matter how you interact with it. I do volunteer on Slack as an admin, so I'm generally keeping an eye on things throughout the day for things to improve. Whether in how the community is running, or for feature requests to Go itself. In doing so I noticed two patterns emerging that made me feel like there was a gap to be filled:
One of the biggest things that continually comes up and sticks with me is the number of people who comment that our community is one of the most supportive and helpful they've ever interacted with, especially when coming in with a Go problem and looking for some insight. It brings me lots of pride and joy to know that we were able to build that community together, and that everyone (even non-admins) is biased towards making sure it remains welcoming. Knowing that made me wonder if there was more we could do as a community to support those who are investing their time in making our ecosystem attractive to so many developers, and could we also build a more clear set of community-maintained "trusted" packages that we can suggest to newcomers? That is basically the foundational origin of The Gofrs. If there's more I can share I'd be happy to do it, but I'm not sure this issue would be the best medium for that.
You're totally right that we're self-selected, and we see it as being similar to how these community spaces (Slack, the forum, etc.) and GopherCon itself were started. We feel they've ended up being effective because of the passion of those who are involved, and we hope to use that same kind of passion in The Gofrs.
Specifically around the number of users in the Gofrs organization, most have chosen not to publicize their membership so we're actually above 40 members. I see this only growing as we put in more time in supporting the community, and our needs become more diverse. That said I don't think our size should be the judge of how effective we can be, but rather the passion we have for the Go programming language and the community that's formed around it. We really care about making Go, and those who use it and build libraries in it, successful.
This is a great question. Trust is something that needs to be built, and we feel that our work on the Not only that, longer-term we're hoping to use the projects we manage as an opportunity for more junior developers to contribute and pair with more senior Gophers and to learn from that experience. While we're not set up for this yet, it's honestly something I wish I could spend more of my community time doing (pair programming).
I think it could be an agreeable solution to simply have the project moved, the license and copyrights for current code retained, and future contributions kept under the same licenses albeit copyrighted to a different organization. Retaining the history is absolutely the most important part for us. I believe I crossed the streams in my head with the Google CLA and another, which is why I mentioned the copyright change. Apologies on that one.
In its current form the latter is becoming less true, and is what prompted this issue being raised. If you are a dep user, and one of your dependencies moves to modules and changes their import statements to use SIV, Things like this are starting to become more common as adoption grows, and so we're hoping to address these sharp edges sooner rather than later.
Our goal here is to provide the best support/experience to the Go community and we're worried that having it alongside the official Go repositories could cause unnecessary confusion, especially if we intend to continue development of This isn't an ask to take our cookies and run away, but rather giving us the freedom and responsibility to support the community and to provide a solution to those who have needs outside of the scope of
Heh. I do this all the time, I really wish they'd make it harder to do. [1] https://www.youtube.com/watch?v=BB09FyaQN5I Edit: I am traveling in Japan to see family; so with timezones and what-not I may be a bit delayed on responding. I'll definitely do my best to get back to any questions/concerns as soon as I can. |
|
I have no current interest in modules, |
rsc
changed the title
|
We transferred this to the main golang/go repo to be with the other proposals (and to see how well "Transfer Issue" worked). |
|
We talked about this at the proposal meeting today. We have concerns about the history of all the issues and repo completely disappearing from github.com/golang. Those events happened in that org, and we would regret the history just being gone. On the other hand we do support moving the responsibility for the code to people who want to maintain it well. What would be ideal is if GitHub would let us both (1) mark this project archived, so that we retain a golang copy of all the issues and other project data; and (2) send a copy of all the issues etc over to gofrs/dep. I'm also concerned about old "go get" (which is still how people get dep) getting confused by the automatic redirect implied by a transfer. It has never handled those well. I don't believe GitHub can do exactly what we want. So probably the best thing would be for you to fork it and for us to post a prominent note here that development has moved to gofrs/dep. We can even mark the project archived. Then people can see that and move over to the new place. The issue history won't transfer, but otherwise you'll be off and running. Does that work for you, @theckman? If we can get an explicit "Yes, let's do that." from @theckman, @sdboyer, and @carolynvs, then we'll do it. Thanks. |
(Note that contributions to the Go project are not copyrighted by a single organization, the CLA is not a copyright assignment, and each contributor retains copyright over their contributions. You are free to use a different arrangement in a fork of course, but I wanted to make this clear.) |
That won't help all the blog posts, projects and scripts people use to download "dep". All of those would need to be updated if golang/dep doesn't redirect to gofrs/dep. See: |
I'm fine with the forking solution if golang/dep is updated with a prominent notice that it has been deprecated in favor of the forked gofrs/dep. I don't mind updating my scripts to point to gofrs/dep. |
|
If we're open to suggestions, adding some logging letting the user know that the project is now over at gofrs could help in the transitionary period. |
I'd argue the opposite. If someone does I share Russ's reservation about the Gofrs organisation. There have been other orgs in the past like pkg, but none tried to represent the Go community, which already has its own organisation on GitHub. Confusingly enough, googling "golang community github" and similar keywords consistently shows the gofrs org before golang on my system, presumably because the gofrs one uses the appropriate keywords to be more visible. |
|
My preference is to do whatever it takes to keep dep alive and maintained for those who want to use it. So if @theckman wants to maintain it over at Gofrs, then that's great! I disagree that keeping the repo and issues archived under golang/dep helps people who still want to use dep going forward. Having it under golang has always confused people (even its maintainers) about if it was an official Go project, worked on by the Go team anyway. 😀 I am not familiar with the known problems of continuing to use If y'all ultimately decide that transferring won't work, @kris-nova helped the dep project out during the gps move by using a bot to copy the issues between repos. So we could use the same process to preserve the old issues. It backlinked to the original issues, did a few other helpful things. |
|
FWIW I am still concerned about the fact that github.com/gofrs claims to be "The Go Community", as I noted above. Now that I think more about it, I would like to see that overbroad claim removed before we transfer dep. |
|
Are there concrete, technical reasons that keeping dep where it is would make ongoing maintenance more difficult? If not, I am against moving it for principally optical or philosophical reasons. |
|
Does the fact nobody has fixed deps incompatibility with modules even though someone has been trying to work on a PR count as a technical reason? |
|
I think that is a chicken and egg problem. If you give the gofrs time to try, they might end up with something that is compatible with modules. But you need to give them time to try. |
|
As far as I can tell, there is nothing stopping the people in gofrs from contributing to dep as it stands today. |
|
@anjmao Let me try to address you points, and reach back to earlier comments where necessary.
Modules are experimental. There are people who are not yet comfortable moving, and so they are continuing to rely on tools like
To bring up a quote from the opening comment:
Over the past few days of different discussions it's cemented the idea that a few prominent features of modules are fundamentally different in their foundation, making it hard to see a path in which modules handles both. So I believe there is a possibility that
I hope this helps. |
|
I see the arguments for dep as is remaining under the Go org. It has been the 'official experiment' for quite some time, and people will still be using it for some years, I should think. And I see the arguments for keeping the official version more or less as it is now. But I also sympathise with the people who worked on dep, and want to keep working on it. While forking software is sometimes a bad idea, this seems like an ideal candidate for an 'amicable fork'. An alternative dependency manager for Go, which supports both modules and some 'dep++'-type functionality, would be no bad thing in my view. I probably wouldn't use it myself, but plenty of people would, I dare say. Maybe it should have a new name, to distinguish it from 'dep classic'. dep started out as an independent project, and was moved under the official Go umbrella. You can't unscramble that egg, to mix a metaphor, so maybe the future of dep is as an independent project—just not this project. |
I think the concern is that the members of the gofrs don't want to have to "ask permission" to add someone to the list of people who can collaborate on dep, or ask permission to get PRs landed etc. It sounds like that may be doable if the go team is willing to grant one or more of them admin access to the dep repo. |
dep has 170 contributors total, which in the 2.5 years it existed averages to approximately one new contributor every 5 days. If you want to give the commit flag to, say, 20% of the contributors that's one new committer every month. It doesn't seem like it would be a big deal. |
|
I feel like the question we're potentially not asking, that should be addressed, is how to make it more possible for outside contributors to join Go projects. I'm sure there's a reason it's not being asked, so forgive my naivety if that's the case. A point of reference for this is how Kubernetes is managed versus Go. It seems a lot of the bumpy road around community involvement and especially the need for an organization like Gofrs to jump in and help would be smoothed over by such a framework. Kubernetes seems to function off of special interest groups (SIG's) and working groups (WG's) in order to satisfy collaboration from multiple corporations and contributors. |
|
@Mattouille Let me reply to your points individually:
What do you mean when you say "Go projects"? Are you referring to the For this specific issue we wanted to focus solely on Now if the question is how do people not on the core team get permission to merge changes, that's a discussion well outside the scope of this issue.
If someone in the wider k8s ecosystem builds a tool that interacts with the k8s API, and a lot of people use it but the author stops working on it, does the Kubernetes project have a procedure for their maintainers to adopt that project? I am not aware of such an effort, and I really don't think it should be on those maintaining k8s to adopt the maintenance of what would effectively be a k8s user. In the same way, I don't think what we (The Gofrs) are hoping to help with should be on those directly maintaining the language. That said if someone from the Core/Language Team is passionate about supporting the ecosystem's packages/tools/etc, and has the spare cycles to join us and help out, we'd love it. I don't think we expect that.
I think there is some value in this point, albeit discussed through different channels (not this issue). That might be an issue on the My focus on this issue is |
The offer ☝️ that @rsc posted is a generous response I think because when we consider the wider Go community, and not the gofrs Go community, encouraging dep to live on in an official fashion linked to or redirected to by golang/dep won't be in the best interest of the wider Go community. The wider community includes a lot of developers who need to build things and not dwell on which dependency manager to use, in the same way we don't dwell on how to format Go code because of gofmt.
Having multiple dependency managers in the Go ecosystem will do more harm than good. We have other ecosystems to look at to see how that turns out. e.g. Java with Ant/Ivy, Maven, Gradle, Bazel. When I started writing Java the build ecosystem was overwhelming, and integrations with each option were inconsistent and fraught with issues. I expect having to navigate multiple dependency managers will become a huge hurdle to newcomers to Go, and a distraction to everyone else.
Can we have one dependency manager? 🙏 |
|
@leighmcculloch I think comparing the Java ecosystem's tools to ours is a bit unfair, because the tools you cite often do way more than just manage dependencies. There's also the issue, as you say, with them being inconsistent / not interoperating well. That's not the path we'd ever pursue here. This is a bit of a heavy question, but I'd love to hear your answer: If there are cases where the MVS methodology does not make the user comfortable with Go, or it just doesn't work for them, would you just tell them to pick another language? Would you tell them to write their own dependency management toolkit and then hope it works well with Modules? The goal I have in mind with If it's something people can opt-in to if Modules doesn't fit their needs, and it will play well with Modules (including rendering the appropriate metadata files) I don't see an issue. At the worst case scenario I'd much rather we take the approach that provides the option of alternative resolution while working well with modules, over ending up with a myriad of tools out there that don't work well with Modules. The latter would feel more like where we are today, and it's not good. Edit: I'd also feel a bit bad about telling someone to find another language that has a dependency manager that fits their needs. |
|
@leighmcculloch I forgot another thing I wanted to ask. I'd love to hear your ideas on solutions to the differences in philosophy, and how to resolve them without two separate tools (where one resolves itself to pay the tax of supporting the other). |
|
@theckman FWIW, I don't think MVS vs. SAT is any different than GC vs. ARC, or nominal vs. structural subtyping, or the decentralized import-path convention vs. monolithic package servers. Yes, I'd totally tell people that Go isn't for them if they are not comfortable with any of these - that just seems the decent thing to do. If they are unhappy with a decision Go made and feel like they can't use it, but we think the decision was right and don't intent to change it, then telling them that is absolutely the right thing to do. That is not the same, however, as preventing them from working on an alternative dependency manager (or whatever) while sticking with the language itself, if they so choose. We don't even have to talk in hypotheticals here: The Go project decided to move ahead with modules and you and others don't like that decision. The Go project is still sticking with it and you are free (and using that freedom) to invest your time into an alternative package manager. IMO the discussion of whether or not dep should continue exist really isn't getting us anywhere. I don't believe anyone seriously doubts that some people will continue to invest in it. Or that the Go project will continue to move ahead with modules and will eventually recommend against the usage of dep. This should really focus on what the Go project, the dep authors and the gofrs group agree on is the best way to make that future happen. In that spirit, to summarize what I see as the state of that conversation (apologies if I'm misunderstanding someone, don't want to put words into people's mouth):
This discussion might be redundant, given that there's a clear suggestion on the table - and while it's probably not ideal, I would've thought it's a compromise everyone can live with :) |
|
I just want to observe that, while a lot of words have been written in this thread, there has still been no concrete technical reason given for making this change. As far as I can tell, nobody who wants to contribute to golang/dep is blocked from doing so, contributors with commit/merge rights are still active, and it's possible to expand that group of people if useful/necessary. Moving a repo for a widely-used tool is disruptive to the ecosystem, we shouldn't do it without a very good rationale. |
|
@peterbourgon as far as I can tell, I think you are right -- there are no technical reasons preventing anyone from contributing to golang/dep in its current location (aside from perhaps knowledge/expertise, which perhaps rightly could be considered "not a technical issue"). But the discussion here is not just about technical matters? Perhaps you could say a few words about either (a) your viewpoint on the non-technical matters, or perhaps instead (b) a few words about why you view this as purely a technical discussion (which you might or might not actually be thinking)? Or a brief comment hitting something along those lines? In the spirit of the attempt by @Merovius to help summarize (and at the risk of me making a hash of it), I think there are some open/unanswered non-technical questions if the repo was to stay at golang/dep, including but not limited to some of the questions asked by @theckman in #29639 (comment), such as:
|
|
I think the issue is visibility and potential confusion. If someone is having issues with dep, it is more confusing to say "oh, you have to use this other dep in this other repo". It is possible to fork a project, but not possible to replace it when the original is more popular. I imagine having two deps out of phase with each other is worse than having one broken one. That said, I am more trying to clarify from my point of view why there would be a non technical facet to this than advocating or endorsing dep (as my support for modules over dep is clear from my previous post above in this discussion). |
|
Thanks, everyone, for your thoughtful points. We appreciate the passion and insight shown in this discussion. We will not be transferring the dep repo. It is an important part of Go’s development history and still a useful tool for many. It will remain in the golang organization indefinitely. If you wish to contribute, the original owners still have all their rights and are active as mentioned above. Once modules is moved out of experimental (we are targeting Go 1.13), we will likely archive the dep repo shortly after to avoid confusion on which dependency management solution the Go project officially supports. We will do our best to communicate when exactly the archive will happen to avoid surprise, but it should be planned for. The gofrs organization is, of course, free to fork golang/dep. Any messaging in the README or elsewhere is up to the current stewards of the golang/dep project (as I understand it, this is @carolynvs and @sdboyer). |
|
@andybons Well, since there are issues outstanding with dep and an issue with availability of the current maintainers, how can we work to get fixed merged in? How can I become a maintainer? |
|
cc @carolynvs & @sdboyer on the above. Trying to understand how we can improve things here. |
|
Both @peterbourgon and @carolynvs, who have been active on this thread, are admins of the repo. I will leave it up to them how they would like to proceed. |
|
@andybons thank you. |
|
Here's one technical reason - the primary maintainers of the project are not Googlers and the Google CLA bot actively impedes getting PR's merged even when all maintainers have signed the CLA. I suppose with administrative privileges we could merge things without the CLA. |
|
That's very annoying. @andybons could you be a |
|
@peterbourgon sure. Anyone on the @golang/osp-team should be able to help in these situations. If it becomes too onerous then we’ll bring it up with the googlebot folks. |
|
@andybons I don't actually have write access to that repo. You might have to fix the permissions to give access to osp-team. |
|
@FiloSottile done. Thanks for the heads up. |
and others
Hi there,
Over the past few months the signals we've gotten is that The Core Go team has no interest in continuing the work of
dep, or to even contribute code to it to help make the modules transition easier. After the tweet from @rsc apologizing for how the dependency management story wasn't handled well[1], I'm not aware of active work from the core team to help improve the current situation which isn't a confidence booster.As a result, I'd like to ask that the
depsource and licenses be transferred out from Google/Golang to the Gofrs GitHub organization so that, at a minimum, we can try to makedepwork with projects that have modules enabled / use the semantic import versioning. We'd like to retain the current issue and PR history, while having full administrative control over the project and repository.We may also want to pick up additional work to continuing to support
deplong-term, as there is a deep philosophical divide between those who want a dep-like experience and those who are happy with modules. I don't believe these disagreements can be resolved in a single solution or implementation, and we should instead work together to ensure we can interoperate.While I'd love to be able to have modules work for me the way I want, I don't think that's a reasonable goal. I now think it's more realistic to support
depin the interest in strengthening the Go community and the tools that are used to support it. I also think competition in dependency management tooling is as important as having competing businesses to help drive progress, and so I believe adoptingdepis one step we can take towards that.Who would be the best person to talk to get the ball rolling?
Cheers!
-Tim
[1] https://twitter.com/_rsc/status/1022588240501661696
Edit: I had someone reach out to me privately to clarify what I meant by the following line:
The current situation I was citing here was the gap in interoperability between modules and
dep, which can result in people being in a less-than-ideal situation if their dependencies have adoptedmodulesbefore they are comfortable moving.The text was updated successfully, but these errors were encountered: