Buy-in for Technical Proposal #47 #145
Comments
|
Hi, what does "buy-in" mean? I'm a bit confused how this can be accepted by any body other than CLC. Scope and API of base is a CLC matter. Is the working group expressing their opinion on a potential upcoming vote? Cheers, |
We would like to humbly request your judgement and determine how we can get your approval for such a plan
We have also petitioned the GHC core team and they are discussing their position on this proposal, as this greatly impacts their work as well. |
My opinion is that this can't be decided unless CLC figures out what CLC is not aligned on this as far as I can tell and doesn't have a joint opinion whether we like base to be minimal etc. My last attempt at getting clarity in that area didn't go anywhere: #141 Why I think this is important? Well, because we're laying the foundation for a new approach to |
|
I appreciate the amount of work spent on refining the proposal but I have troubles seeing the reasoning behind it. Maybe all is good, and the proposal text just needs to be slightly changed to make currently implicit things explicit, but for now I have a few questions. Problem and unoptimal solutionCurrently, the proposal defines only the following problem:
The proposed solution to this problem is (in my understanding of the proposal text):
This sounds like a massive waste of time and money for solving the problem as stated. A much-much simpler solution in my vision is to put a single documentation line e.g. "This module is internal for GHC and it's API may change in a backwards-incompatible way" to all GHC-specific modules. I don't see this option in the section of alternative solutions with the corresponding comparison. base-compatAgain, this is not written in the proposal explicitly, but from the discussion I have an impression that one of the final goals is to decouple I find this goal admirable but it's not written explicitly. So if it's actually one of the goals for the proposal, I would recommend to:
I would like to see the above two points addressed before I can vote on this proposal as a CLC member. |
|
@hasufell my proposal does not change the interface of base. It is just about changing the implementation. In prior occasions @Bodigrim has in fact said the opposite, that the CLC weigh in on their sort of thing at all. So consider this thread not an attempt to bypass the CLC, but figure out what parts or this the CLC is to sign off on and formally get the CLC to sign off on those parts. There still needs to be an HF proposal because the proposal explicitly has HF paying for things, but that doesn't mean the CLC needs to be bypassed in the slightest. |
Strong agree. This is the first step we should take before embarking on anything more elaborate. Hopefully it's quick and easy! But we should do it. |
|
@Ericson2314 the MR in it's current form does neither change scope or API of base. However, that is the ultimate goal: splitting base. Or is it not? (See point 4. @chshersh raised) So my opinion is this needs CLC approval regardless of whether any API is currently affected or not. This is about deciding on the future of base. CLC doesn't have a joint opinion on the future. So my concern is that we may (e.g. accidentally) regress if we decide on all matters of base in an ad-hoc basis. |
By the way I already did the equivalent for |
I think the "new approach" part is actually pretty based agnostic --- it is more about decoupling (stable, intended for public consumption) library major bumps and compiler major bumps in general. I have already started writing something on that with much the same thought of it being good to nail down policy and vision separate from implementation. |
So let's leave aside existing modules for the moment. Why should new GHC-specific implementation details go into base it all? Why is it is good to have a "divided Berlin" library serving two different purposes? (Stable standard library for regular user, misc exposed implementation details for adventurous users.) I think the GHC devs need some place to put random junk that is out of the way, rather than constantly be butting heads with the CLC on this. Existing modules still need to be triaged as to who is using them, where they belong, etc. etc. but this has proven to be contentious and even without contention I would submit it is actually a lot of work. So I do think the policy parts @hasufell mentions are important to handle up front --- do we actually want separate homes for unstable implementation-specific stuff vs stable implementation-agostic stuff? But once that is agreed upon the triaging and dividing can happen over time with clear goal post in sight.
Yes great points. Sorry this motivation got lost when the proposal was split. |
Strong agree with this too. |
It's a mystery to me why modules with GHC-specific implementation details happened to be in
Since the amount of work to split GHC internals from
Strong agree with this as well. But is it a technical problem or a communication one? If GHC devs want to exempt some But if there's a technical limitation (e.g. having GHC-internals in |
Keep in mind that this effectively means that even Haskell's standard library can't adhere to PVP. Unless you mean "exempt from CLC supervision, but still bump version according to PVP". At which point I'm not sure anymore what's worse. |
Moreover, even if it's not written in PVP explicitly, it's a common practice to have Of course, this depends on specific modules. I wouldn't want sudden breaking changes to |
You mean it does so for every major GHC release: https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/libraries/version-history
I find that this is not good enough and the wrong approach. PVP is governed by CLC and hackage trustees. Please raise a proposal for PVP before we silently bypass that channel as de-facto standard in base. It breaks tools, it breaks user expectations. Especially if it's a line in some haddock documentation probably no one ever reads. Making GHC aware of internal modules and being able to emit a compile time warning sounds much more exciting. But as of today, I'm neither excited about the |
|
Almost forgot the proposal already exists: haskell/pvp#46 |
Agreed. The history here is complex in that I think at one point base did have CPP for e.g. GHC vs Hugs. But still odd!
Yes. the
What is being proposed at this time is not huge. It is a first step to unblock a long incremental process of untangling. It is OK to get partways through that process and then stop --- e.g. because more new language features are needed to "liberate" the remaining code in
I am confused, this sounds like a tautology to me? I think the cost issues are the most important thing above for us to work out @chshersh. I agree we can't just pay for untangling everything up front, but that doesn't mean there is no value in kicking off a smooth incremental process. This is the sort of thing where we just need to learn more to understand what it is easy and hard, but we can't do that very well today. I am quite confident that at least some things can easily be separated from |
|
And yes I agree with @hasufell --- documentation is not good enough, because people generally only read docs when they get stuck. And if the modules are in Arguably one could also imagine tooling that automatically extends |
|
@Ericson2314 yes. But... my major concern is that we're drawing API boundaries based on projects (or: people) and not based on what makes sense for the end user. How do we envision this working long-term? Will API flow back into base? Under what circumstances? Forgive me for being really frank here, but I don't see GHC HQ pushing for stable API anytime soon. Once we have a split and an "internal" playground, I expect very frequent breakage in the split out base part. And yet people will likely use it too. And here's my question: will this serve GHC development, will it serve base, or will it even serve the end user? I feel this is really hard to answer. |
I also agree it's not enough. I don't think @chshersh was saying that either. But if we know certain APIs are unstable and we don't take the most simple and straightforward means of communicating that to the user, then I think we're derelict in our duty. Let's not let perfect be the enemy of good here: we should document what we know to be unstable whilst also looking for a good, general, technical solution. |
Some backstory which helps underline the motivation here: #105 In that lengthy discussion the idea that some modules be exempted was floated, and the conclusion from the CLC was that this is not a good idea. A lot of the motivations in all directions are discussed there, along with a fair amount of useful empirical investigation. While the approach of denoting some modules internal or some functions such was discussed there, and while I was indeed very sympathetic to it, I found the discussion pretty convincing and I understand why it was not approved of. On to @hasufell 's question, I think the proposal will serve all three arenas. Here are some (but not exhaustive) reasons. (Though I find it a bit odd to speak of serving ghc devs and end users -- both groups of people -- and also of serving "base" -- which is a body of code, and for now, last i checked, not sentient).
I imagine it would also serve the CLC by making it possible for it to focus on the exposed functionality of "genuine base" without also having to directly worry about additional things in |
I agree that most package divisions are done for conway's law reasons, and that's not good. Long ago, the same people worked on the GHC agnostic and specific parts of base; it was one repo. now there are separate groups; it can be two repos. This is indeed not reason enough! That actual goal is:
This is reasoning separate from who is working on what. I think it's good! :)
Yes I hope it will. But to be clear we don't have to wait for that before the benefits kick in.
Per the above, I think that is perfectly fine!
That is fine!
That's on them. There is the crucial bit: Anything reexported from Clearly, having "delicate porcelain" in the "playground" is no fun, and having to remember which parts are reexported is also no fun. So for the sake of having a proper sandbox where one doesn't have to be so careful, I think they will be likewise incentivized to try to move that functionality back to They also are incentivized to remove "grandfathered in" unstable stuff from |
|
@tomjaguarpaw Definitely base should be triaged, and if something cannot be moved somewhere else, adding a note in the docs is better than nothing. No disagreement! I don't mind for this process to slow that down. This can proceed in parallel until the initial split is merged, and then it simply opens up more possibilities for that triage process which hopefully help it go smoothly. As @gbaz says, it seems like that is the lowest hanging fruit, but after the conversation in #105 it isn't clear that it actually is! I get that doing a technical project to "hack around" not being able to do a simple documenting task might seem like a "cop out", but in this case I think it is fair to say at least establishing the long-term goal and direction (namely, |
Indeed this is a useful first step which we have proposed in #146. This is something that I have been quietly working on in the background for some weeks now; I do wish I could have opened #146 before this issue arose to avoid much of the confusion in this ticket, but sadly things did not work out that way. However, to be clear, I personally think that splitting internal modules into a new To make matters worse, experience has shown that users tend to treat exposed modules as "stable" regardless of what documentation says. Consider, for instance, the fact that a significant fraction of the modules which we are proposing to be internal are already documented as being "internal" via their Haddock documentation. Nevertheless, many have significant numbers of users despite, in many cases, their being more stable places to import the provided functionality from. Drawing a clear package boundary between |
|
I should probably clarify because I probably wasn't clear: I believe that modules that we know to be internal/unstable should be clearly documented as such whilst we pursue other options. I don't believe that documentation makes the other options unnecessary, and I'm in support of the direction of travel of @Ericson2314 @bgamari and others. |
|
@Ericson2314 would this also remove the no haddock pragmas from those internal modules? |
This is really a matter of definition. I would have to go digging for examples but I am certain that there are cases where we have broken the PVP in minor releases due to changes in implementation. I think in practice we do a reasonably good job at versioning public interfaces but technically, when one includes the
To be clear, the amount of work needed to break out The only open question that comes to mind is how we ensure that error messages continue to refer to
Indeed, see #146. However, I do still believe there is value in having a package distinction. As noted elsewhere, documentation tends not be be seen, is not easily auditable (mechanically or otherwise), and can fall out of date (e.g. there are a significant number of quite stable modules in
I'm not sure I understand what you mean, @hasufell. Until very recently we have had precisely the split that you describe (where our playground was largely inside of
If an interface is intended to be public (note that many are not) and it is possible to provide the degree of stability which GHC's users expect, then I would indeed expect that the interface will be propagated back to |
Mention one or two. Otherwise that point is easily dismissed. |
Happily, I was unable to find even a single example of this in the last few releases. However, the principle stands: if we need to change, e.g., the type of |
What happens when GHC devs need to add an internal definition that (a) depends on parts of
I'm not convinced this is true. You seem to think the proposed split of
It has a good record of following PVP because the major version of
I would assume that |
|
As a point of order, this thread is not the best place to discuss ideas around or possible future developments of the HF proposal. It is a request to buy in a specific written text, and I am unwilling to get behind uncertain speculations.
If a new definition is depended upon by other parts of I do not quite enjoy discussing theoretical possibilities, there are over 9000 them. Please come up with a realistic scenario and explain it in the proposal.
Maintaining one library is hard, maintaining two libraries in sync is hard squared. The burden of proof lies with the proposer. The proposal does not discuss maintenance costs and makes no attempts to convince readers that they are somehow negligible.
As a Hackage trustee and a maintainer of multiple core packages, I find this argument very weak, but it does not matter because the current proposal does not seem to reference it.
In my opinion this is not a workable arrangement. It makes it too easy for GHC contributors to break things inadverently, to the degree that I would find CLC position untenable. It also makes it too tempting to put something generally useful in |
I am curious to know what concretely this is referring to. The GHC maintainers have never, as far as I know, placed something in |
|
I am deeply sorry, my rhetoric got in the way of clarity. The quoted passage relates to the previous sentence only, not the entire paragraph. I was referring to the fate of |
Was The entire point of the proposal is to make it coordinated and formalize the approach, so that things don't randomly evolve. I find your stance a bit hard to follow... somehow you seem to point out that random evolution is problematic, but also suggest that coordinated effort is too much effort. I think we'll be doomed to pick one. Does the current proposal sufficiently describe how that coordination looks like? No. But it could. |
|
I would also say in general, take the non-normative parts with a grain of salt. They perhaps still reflect my own opinions more than a general consensus. I hope we can reach an agreement on the normative parts, and then I am find with any outcome for the rest. |
No worries. Indeed communications can be tricky.
Yes, this is true and admittedly GHC has not been as careful in this regard as we could have been. However, I do think it is reasonable and feasible to draw a clear line between the interfaces provided by Haskell-the-language and GHC-the-implementation. For instance, the Haskell report specifies that an implementation should provide a type named |
|
For what it's worth, I have recently stumbled across a use-case where One way to address this would be to move the implementation of |
|
I commented on the same @bgamari comment elsewhere. #146 (comment) and next few comments. I think that will be bad. I probably would just use TL;DR trying to solve short term problem making more long term ones. Not great. |
I think you may have misread the comment, @phadej. I did not suggest that |
You right. Your comment looked too similar to the other one, and it confused me. |
|
Ahead of the next HF technical working group meeting (Thursday) I am trying to figure out what's going on in this mad related threads, and it is confusing. I hope we can draw this to a close soon. |
I think the main proposed alternative in terms of "base split" is to do a gradual approach: instead of moving everything out and making base a reexport shim, we just let GHC devs create a boot package and do innocent stuff there without touching base (doesn't need CLC approval at that stage). The next stage could be:
The last stage could be:
All that would be negotiated in isolation, when it is actually proposed. And it could start without CLC involvement. The upside of this is that it won't burn out CLC and is a conservative approach, that only requires CLC involvement at certain stages (that may or may not happen). The downside is that it's unclear whether we'll ever get a proper "split" and achieve the "base is reinstallable" goal. |
I like the gradual approach too. But I have a technical question. Provided the modules, exports, and API of
This decision is just an implementation matter, isn't it? Of course, under (2) any changes in Maybe there are issues I'm not aware of. It'd be great to lay them out. Incidentally in another thread you drew attention to https://nikita-volkov.github.io/internal-convention-is-a-mistake/, describing it as the proper pattern for exposing internal modules. It's great that this |
|
Thanks, both of you.
Do note that if we move things up and out of The only reason I proposed moving things "down" to
Absolutely this is a goal. I didn't go in to this expecting to drag in the CLC much at all, and I think something got lost in translation. Based on the @hasufell's response, I think I might have spotted a source of what (to me) is causing unexpected acrimony. I think it is the fact that
I've tried to make clear that anything reexported as-is by
To make it absolutely clear that is not the intention, perhaps we shouldn't use
This makes clear that code that has yet to be untangled is not biased towards either one of GHC devs or the CLC: it is still in the middle and has yet to be worked through, and everyone must be cautious when changing it. Additionally, this allows the "peal off the bottom" and "peal off the top" approaches of proceed concurrently. While I suspect "peal off the top" to work much better in practice (and hence I didn't propose and empty |
|
Adding another layer doesn't solve the problem, it only makes it more confusing. |
|
Then think about just a Whatever It is perfectly possible that won't happen, and we'll just get stuff pealed off the top of |
|
I understand the conceptual desire for three layers, but as a practical matter I agree two is plenty. I think you're right to note that ghc-base will be doing double duty -- but moving stuff for conceptual purposes will I think confuse both public and intra-committee messaging further. Let me point out a third component as well on top of the other two, which I think would make a further split even more of a pain. You said
On top of that, we have
That said, between the "move and reexport" (ghc-base is initially "overpopulated") and the "create without moving" (ghc-base is initially sparsely populated) approach, I think both could work with ultimately similar end results, and its just a matter of which people think creates the nicest collaborative process going forward. |
Ah yes. That is what I meant; to me that is just a more precise statement of the same idea: It's the GHC's team responsibility precisely because of that coupling.
Right, and that's where the rubber meets the road. I am very pessimistic about peeling stuff from the bottom; I am very optimistic about peeling stuff from the top; to do the latter (without moving things out of |
|
@Ericson2314 if I understand you correctly, you are leaning towards keeping most implementation in I am remain rather skeptical that this is feasible. Many parts of |
|
Other way around: I share those exact same concerns. |
|
More broadly, the focus of the propose is not what division do we want in the long term but what untanglings make sense in the short term. That is entirely where the "move and reexport' comes from. This is supposed to be the beginning of a process, an experiment, not the end of it. In the other thread @Bodigrim writes
And that doesn't seem to me to be true at all: it is incredibly easy to recombine libraries that live in the same repo. And that is good, because in the spirit of trying things and experimenting, everything should be reversible. |
|
There is some debate above about what code might live in which library. But I'm puzzled. Let me ask my question again:
This decision is just an implementation matter, isn't it? Of course, under (2) any changes in ghc-base that affected the API of base would require CLC agreement. For example, if But It's an implementation matter where the function is actually defined. So I'm puzzled by the above debate. Maybe there are issues I'm not aware of. It'd be great to lay them out. |
|
@simonpj I agree, it is puzzling. My guess that the overloaded function of That's why I hoped renaming @mixphix and @gbaz said 3 layers was too much, and because I confused @bgamari yet me reiterate: yes, I don't actually think we can separate GHC internals beneath everything else well at all. That means we wouldn't get 3 layers. I suspect what would happen instead is this:
No 3 layers in that plan, because of the great difficulty in doing any refactor to If avoiding moving untriaged code to a library with |
|
We had a very productive meeting with @Ericson2314 at Zurihac, and worked on a new proposal based on haskellfoundation/tech-proposals#47 (comment) suggestions, soon to be made public. Thanks all who participated here, helping us to understand the underlying problem and perspectives better. Let me close this discussion, it's been quiet for the past month and it will be rendered obsolete by the new proposal soon. |
Dear CLC members,
In recent weeks, John Ericson has fine-tuned a Haskell Foundation Technical Proposal to split
baseinto two libraries:ghc-baseandbase, the latter simply re-exporting everything forghc-base(for now). You can read about the rationale and specifics more in details in the proposal itself: haskellfoundation/tech-proposals#47Note that this proposal has recently been streamlined into a form which is more focused than its initial state, and might be worth a re-read.
The Haskell Foundation Technical Working Group has reached a consensus that this work will benefit the Haskell community. Moreover, the Haskell Foundation has agreed to spend some of its resources to implement this proposal, which would start by ensuring the completion of ghc/ghc!7898.
This work will affect the decisions taken by the CLC. Therefore, the Technical Working Group would like to get buy-in from the committee before formally accepting this proposal. Since you are on the frontlines of this matter this is extremely important for us to know about potential reservations or enthusiastic approval from your part. :)
The text was updated successfully, but these errors were encountered: