Expose the new primops isByteArrayWeaklyPinned# and isMutableByteArrayWeaklyPinned# from GHC.Exts

[AndreasPK]

We will add these new primops to GHC in 9.12 (https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13144) and likely backport them to 9.10 as well.

I expect them to be stable and used widely going forward and would like to avoid people depending on ghc-internal for this feature. Therefore I propose we expose it from GHC.Exts in line with most other primops.

For background on the issue see https://gitlab.haskell.org/ghc/ghc/-/issues/22255

For details about what these primops do see the changes to docs/users_guide/exts/ffi.rst in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13144

[tomjaguarpaw]

I'm afraid I've forgotten the latest status, according to the CLC, of the GHC. hierarchy in base, and particularly the status of GHC.Exts. Did we make a decision on this? According to @bgamari's Base stability spreadsheet the "Desired visibility" of GHC.Exts is "see document", but I don't know what document that is.

And for reference, here is a big previous discussion: #146

[AndreasPK]

I assume it's the document in the foot note which merely raises the question:

## 4. The question of `GHC.Exts`

Historically `GHC.Exts` has been the primary entry-point for users wanting access to all of the primitives that GHC exposes (e.g. primitive types, operations, and other magic). This widely-used module poses a conundrum since, while many of these details are quite stable (e.g. `Int#`), a few others truly are exposing implementation details which cannot be safely used in a GHC-version-agnostic way (e.g. `mkApUpd0#`, `unpackClosure#`, `threadStatus#`). There are at least two ways by which this might be addressed:

 * Export only the subset of primops that we can stabilize (e.g. things like `Int#`, `Weak#`, `newArray#`, etc.) in `GHC.Exts`, leaving the rest to only be exposed via `GHC.Prim` (which should not be used by end-users), or
 * Declare the entirety of `GHC.Exts` to be unstable and export the stable subset from another namespace (e.g. `Word#` and its operations could be exposed by `GHC.Unboxed.Word`)

What eventually was voted on was this comment: #146 (comment), according to which the following applies to GHC.Exts:

The API of this module is unstable and not meant to be consumed by general public. If you absolutely must depend on it, make sure to use a tight upper bound, e. g., base < 4.X, not just base < 5, because the interface can change rapidly without much warning.

All that being said none of that really makes a clear proposal about how these or other future primops should be handled in regards to GHC.Exts

[phadej]

@AndreasPK IMO, there is ghc-experimental, so you can leave out GHC.Exts, and expose new primops in ghc-experimental. (I personally am fine depending on ghc-prim if I need primops; so IMHO they don't need to be re-exported anywhere).

[AndreasPK]

@AndreasPK IMO, there is ghc-experimental, so you can leave out GHC.Exts, and expose new primops in ghc-experimental. (I personally am fine depending on ghc-prim if I need primops; so IMHO they don't need to be re-exported anywhere).

Personally I would prefer a wider design/agreement on how to expose primops via ghc-internal/ghc-experiment/base using a sensible module structure over use of GHC.Exts. But without time to design and implement such a proposal exposing them via GHC.Exts seems like the next best option to me.

---

That being said I'm not adverse to adding it to ghc-experimental instead. But as I understand it the purpose of ghc-experimental is:

ghc-experimental, initially empty, depends on base. Functions and data types here are intended to have their ultimate home in base, but while they are settling down they are subject to much weaker stability guarantees. Example: new type families and type constructors for tuples, ghc-proposals/ghc-proposals#475.

So if the CLC wants to stop exposing any primops from base (maybe even deprecating existing ones in the process) that wouldn't be unreasonable but then it seems unclear if these primops should still be exposed via ghc-experimental or simply remain relegated to ghc-internal.

But then we also don't want to encourage dependencies on ghc-internal. Making for no obvious solution from my perspective.

In light of all these complexities and unanswered questions to me the simplest and most consistent solution would be to expose those via GHC.Exts in line with existing primops and hope for someone to draw up a design for primops in the future.

Ultimately the decision lies with the CLC. If the proposal is rejected however I would still welcome concrete suggestions on how to best expose these primops in particular or primops in general.

[mixphix]

I agree that the most sensible solution is to export it from GHC.Exts as the module hierarchy currently stands, if they are to be so stable and widely-used.

Would that those interested in adding primops be the ones drawing up a design for primops...

[Bodigrim]

I'm afraid I've forgotten the latest status, according to the CLC, of the GHC. hierarchy in base, and particularly the status of GHC.Exts. Did we make a decision on this?

No, there was no particular CLC decision on GHC.Exts. I recall that @adamgundry was recently arguing against re-exporting all new primops from GHC.Exts by default?..

In this particular case the primops are of primary interest for packages like bytestring, so I'd strongly support re-exporting them from base instead of ghc-internal / ghc-experimental. Whether it's GHC.Exts or something more specific like a hypothetical GHC.Pinnedness could be up for discussion, if anyone feels strongly enough about it. Otherwise, if there is no huge interest to bikeshed, I'd suggest defaulting to GHC.Exts.

[adamgundry]

I recall that @adamgundry was recently arguing against re-exporting all new primops from GHC.Exts by default?..

In the long term we should move away from the monolithic GHC.Exts, because it is an unstable and unnecessary point of coupling between base and GHC internals, as well as being too large to be easily understood by users. And in the meantime, yes, we certainly shouldn't expose all new primops by default.

That said, there's clearly work to be done to come up with a new design that answers the questions raised in this thread, and that won't happen overnight. Meanwhile, if specific new primops are sufficiently widely useful and stable enough to be desirable in base, following the status quo by adding them to GHC.Exts seems reasonable.

[Bodigrim]

If anyone is in favor of re-exporting these primops from base but has strong objections against doing it via GHC.Exts, please voice your concerns and suggestions until early next week. If there are none, I'll ask @AndreasPK to prepare an MR which we can vote on.

[phadej]

I think new primops pop up regularly enough (this, #203, #188) that it would be great to figure out for the new primop export design sooner before the next primop is being added.

I opened a GHC issue, https://gitlab.haskell.org/ghc/ghc/-/issues/25242 as I think it's more of GHC than CLC issue.

[Bodigrim]

Oops, sorry, this fall through the cracks.

Dear CLC members, let've vote on the proposal to expose the new primops isByteArrayWeaklyPinned# and isMutableByteArrayWeaklyPinned# from GHC.Exts as implemented in https://gitlab.haskell.org/ghc/ghc/-/merge_requests/13351/diffs. While in future things might drift to other conventions, historically all new primops were exposed through GHC.Exts, so this proposal is business-as-usual.

@tomjaguarpaw @mixphix @velveteer @hasufell @parsonsmatt @angerman

---

+1 from me. Using these primops allows to leverage implicit pinnedness, which allows for zero-copy ByteArray / ByteString operations and is a major win in many scenarios. The primops restore capabilities lost after https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9254 (which was done for a good reason!). It would be a shame to depend on ghc-internal / ghc-experimental to access them, they are no more experimental or internal than isByteArrayPinned# / isMutableByteArrayPinned#, which are (and always were) in GHC.Exts.

[parsonsmatt]

+1

[velveteer]

+1

[Ericson2314]

I am not a voter. but I think this is bad.

We don't need to bikeshed a new design, we just need to reexport things elsewhere. Pinned and unpinned arrays is a concept I do not think ever belonged in base because it is too tied to the RTS --- for example, it is unclear if it would apply to e.g. a Web Assembly backend using the built-in GC.

Really we should default to not adding things to base. Needing a CLC proposal is an added fraction that should help with it.

That said, there's clearly work to be done to come up with a new design that answers the questions raised in this thread, and that won't happen overnight.

This strikes me as FUD. Create a new library, put in ghc-experimental, I don't care. Just stick it somewhere and move on with life. This is supposed to be an easier and more regret-free process than putting things in base: erring on putting things in base should never be a way to avoid "new design that answers [..] questions", that's utterly backwards.

If we are like "well, everything else like this is already in base, so fuck it, let's put this in there too", we're going to keep on making that decision, keep on sticking things in base, and never end up with the separation of concerns. We have to fight path-dependency and break old habits.

[Ericson2314]

It would be a shame to depend on ghc-internal / ghc-experimental to access them, they are no more experimental or internal than isByteArrayPinned# / isMutableByteArrayPinned#, which are (and always were) in GHC.Exts.

So some functions are in the wrong spot, so these functions should also be in the wrong spot?

Also, we don't have to limit ourselves to just ghc-internal / ghc-experimental, library names are cheap! If it neither experimental or internal, just make a unboxed-arrays library or something.

We should never say "this should go in base because x and y other libraries are worse". Being in base is the highest level of stability, and should therefore only be justified in positive terms, not negative terms.

[hasufell]

-1

---

I think we need to start being stricter. Just because the module already exists and is named GHC.WhatEver doesn't mean we need to keep adding to it.

Move it elsewhere.

[angerman]

Weak +1 for exposing more and more primops.
-1 for exposing them from base while we try to slim down base conceptually; that seems to run counter to that idea?

[angerman]

To clarify this for @hasufell, who asked me to kindly stick to one integer.

-1 as proposed.

[adamgundry]

Also, we don't have to limit ourselves to just ghc-internal / ghc-experimental, library names are cheap! If it neither experimental or internal, just make a unboxed-arrays library or something.

@Ericson2314 Library names may be cheap, but designing, publishing and (indefinitely) maintaining new libraries is not. Somebody has to maintain them, and GHC HQ is sufficiently stretched already that it isn't reasonable to add maintenance burden alongside GHC releases. That's why for things like primops that are tightly coupled to GHC version, I think it makes sense for the GHC-maintained user-facing export to be in ghc-experimental. Anyone is then welcome to make up a new library that selectively re-exports parts of ghc-experimental to provide a more stable interface across GHC versions.

See also discussion on the role of ghc-experimental at https://gitlab.haskell.org/ghc/ghc/-/issues/25326.

(This is to some extent orthogonal to the question of whether these primops are useful enough to be worth exposing from base, on which I have no clear opinion, so apologies for intruding on the vote.)

[tomjaguarpaw]

-1

---

I agree with @hasufell and @angerman

[mixphix]

Somebody has to maintain them, and GHC HQ is sufficiently stretched already that it isn't reasonable to add maintenance burden alongside GHC releases.

Somebody has to maintain base, too. And I think it is safe to say that GHC.Exts is as maintained as it is documented. How long would import GHC.Exts be in an average file if it was made explicit? I think it is worth investing time into sorting this out for the sake of GHC itself, let alone for the stability of the rest of the Haskell ecosystem. For that reason I'm -1.

[Ericson2314]

@adamgundry

Library names may be cheap, but designing, publishing and (indefinitely) maintaining new libraries is not. Somebody has to maintain them, and GHC HQ is sufficiently stretched already that it isn't reasonable to add maintenance burden alongside GHC releases.

You are saying that 1 big library is much less work than N smaller libraries 1/N as large. That...really ought not to be true! That's really bad if it is true!

That's why for things like primops that are tightly coupled to GHC version, I think it makes sense for the GHC-maintained user-facing export to be in ghc-experimental

Sure, I have no problem with that at all. The raw blanket reexport of all primops is indeed highly experimental, and doesn't belong in base. Until it is removed, we'll never be able to achieve the goal of having the same base version across two compilers, liberating us from bound-bumping hell.

But the rest of this discussion was saying that ghc-experimental was not good enough, because these new primops are more stable, bytestring doesn't want to depend on ghc-experimental, etc. It's just in response that assertion that I say it belongs in a new library.

[mpickering]

I think it makes sense to put these primops in ghc-experimental. If they are in ghc-experimental then anyone in this thread is free to make their own libraries to provide stable interfaces for these primops if they want to use them in their own packages. This insanity of expecting GHC maintainers to perform all these tasks has to stop.

At the moment there are multiple proposals where GHC contributors have tried to persist the status quo. In these threads, the CLC has decided that they don't like the status quo anymore. How are GHC developers supposed to know which one this is before starting to work on a patch?

Perhaps the CLC could do some work to write some proposals which remove and deprecate the unstable parts of base which they don't want to maintain. It's my personal view that at the moment the whole process is hugely demotivating GHC development. It would be much better if the CLC could communicate beforehand which parts of the base API they don't want to keep maintaining rather than people repeatedly discovering this after performing a reasonable amount of work.

[Ericson2314]

If they are in ghc-experimental then anyone in this thread is free to make their own libraries to provide stable interfaces for these primops if they want to use them in their own packages.

Fine with me, we can see just how much bytestring maintainers don't want to depend on ghc-experimental when deciding who foots the bill.

It's my personal view that at the moment the whole process is hugely demotivating GHC development.

I wanted splitting base to be motivating, not demotivating, for both GHC dev and the CLC, and I agree that that doesn't appear to be happening yet. Something should change.

Perhaps the CLC could do some work to write some proposals which remove and deprecate the unstable parts of base which they don't want to maintain. [...] It would be much better if the CLC could communicate beforehand which parts of the base API they don't want to keep maintaining rather than people repeatedly discovering this after performing a reasonable amount of work.

I am quite sympathetic to this. So long as base exports a bunch unstable crap, we'll never reach the goal of not having annoying base bumps every GHC release. But if we could proactively deprecate all those things, then after N releases we have the opportunity of ripping of the band-aid, and never dealing with base version churn bullshit ever again.

I strongly encourage everyone to do the big deprecation, then the big breaking change, and get us out of the shitty status quo of dozens of people spending hundreds of hours doing hackage revisions every GHC release. It's absolutely worth the one-time pain of moving out a bunch of modules.

And yes, we'll also end up with clear guidelines of what goes where going forward too, avoiding a lot of confusion in these CLC threads going forwards.

[bgamari]

I am quite sympathetic to this. So long as base exports a bunch unstable crap, we'll never reach the goal of not having annoying base bumps every GHC release. But if we could proactively deprecate all those things, then after N releases we have the opportunity of ripping of the band-aid, and never dealing with base version churn bullshit ever again.

FWIW, agree with this. IMHO base is too large and is the cause of too much churn. My stability assessment (specifically the "desired visibility" column) was an attempt to identifying the kernel of base; something that we could in principle support across GHC versions, with other functionality living in other (hopefully topical and therefore readily versioned) packages. However, a great deal of work is needed to get there.

[tomjaguarpaw]

@mpickering, I definitely don't want GHC developers to feel demotivated! Can you help me understand how the CLC can help them feel more motivated?

As far as I understand it, when GHC developers want to add a feature to GHC they can do so without involving the CLC. If their feature requires something to be exposed from a library then, as you suggest, they can do that too, from a variety of libraries controlled by GHC HQ, including ghc-prim, and now ghc-internal and ghc-experimental too. There's no need to involve the CLC in that process either. They can also develop stable packages through which to expose these library features, or any community member can do so, or any community member can propose the features for inclusion in base at a later date.

You elaborate:

anyone in this thread is free to make their own libraries to provide stable interfaces for these primops if they want to use them in their own packages. This insanity of expecting GHC maintainers to perform all these tasks has to stop.

Could you please clarify what you mean by "all these tasks"? Do you mean the task of creating stable packages for functionality exposed from a GHC package? I must have missed people suggesting that GHC maintainers "perform all these tasks". Could you link to the posts in question?

My immediate suggestion to GHC developers would be: if, in the first instance, you expose new functionality through packages under the purview of GHC HQ then there's no way that CLC can stand in your way. Is that a good enough first step? After that is an established expectation then we can think about how to improve things further.

Or have a missed something that means that that particular suggestion is unworkable?

[hasufell]

How are GHC developers supposed to know which one this is before starting to work on a patch?

That's a general problem with committee driven workflow.

You simply can't know beforehand and the proposal process requires an up-front implementation.

However, you're free to inquire informally beforehand if something takes a lot of time to implement. But note however that those are still non-binding agreements.

A different angle was #141 with the end goal of developing holistic policies for exactly that purpose.

Perhaps the CLC could do some work to write some proposals which remove and deprecate the unstable parts of base which they don't want to maintain.

As of now, the CLC isn't receiving any funding to do larger work and we are all volunteers.

I expect GHC developers to be on board with the base split and assign resources to drive this forward, including classification of stability of modules.

If you lack resources, please contact the HF.

When we all discussed the base split, I thought it was clear that this will not come for free. And during that discussion we also agreed that we'd rather vote on a per-module basis, afair. So no one needs to come up with an exhaustive up-front list that will drown in bikeshedding.

In the end, we are aiming for a shared goal: less intertwined GHC and base is less work for CLC and for GHC HQ.

[Bodigrim]

If you lack resources, please contact the HF.

In theory CLC could also ask the HF to provide more resources.

So long as base exports a bunch unstable crap, we'll never reach the goal of not having annoying base bumps every GHC release.

Stable base and reinstallable base are largely orthogonal goals.

I strongly encourage everyone to do the big deprecation, then the big breaking change, and get us out of the shitty status quo of dozens of people spending hundreds of hours doing hackage revisions every GHC release. It's absolutely worth the one-time pain of moving out a bunch of modules.

Sorry, I've heard the pitch "let's wage the war to end all wars" quite a few times, and I'm not buying it. There is a never ending list of "one-time pains", which will cause nothing but resent and discontent.

Ultimately the decision lies with the CLC. If the proposal is rejected however I would still welcome concrete suggestions on how to best expose these primops in particular or primops in general.

Judging from https://gitlab.haskell.org/ghc/ghc/-/commit/39497eeda74fc7f1e7ea89292de395b16f69cee2, I take it that we settled on ghc-experimental to expose them.

[Ericson2314]

Stable base and reinstallable base are largely orthogonal goals.

There are only orthogonal if we do a huge amount of shimming.

war to end all wars

I don't want to do anything dramatically high budget. I just want to not be forced on a breaking change to base every GHC release. This is not an unreasonable request, and actually I don't care how we get there. Shim modules / remove modules, it doesn't matter, so long as it gets done.

But I am skeptical it will get done by shimming every last shitty GHC.* module because the Haskell community is not that wealthy right now.

I want the CLC, and you in particular, to be open to some breaking changes of module removal, because the costs of breaking some packages are not necessary higher than the costs of everyone else dealing with base version bump for packages that aren't actually using a part of base that changed. I'm pretty sure the costs of the former are in fact lower.

[Bodigrim]

But I am skeptical it will get done by shimming every last shitty GHC.* module because the Haskell community is not that wealthy right now.

It's either we are wealthy enough both to break things and provide shims, or we are poor - in which case it would be wise to do neither.

I want the CLC, and you in particular, to be open to some breaking changes of module removal, because the costs of breaking some packages are not necessary higher than the costs of everyone else dealing with base version bump for packages that aren't actually using a part of base that changed. I'm pretty sure the costs of the former are in fact lower.

I'm not sure why you are singling me out, because in fact I'm on less conservative part of CLC spectrum. I'm personally quite happy to remove things from base, provided that there is a (short) deprecation period and a proposer has prepared patches for affected packages.

That said, judging from my experience as a Hackage trustee, I disagree with your costs analysis here. Assuming nothing was broken for real, even if a maintainer does not have two minutes per year to make a revision, a client can say --allow-newer and carry on. Pure version bump is never a blocker, but breakage is.

[AndreasPK]

Thanks everyone for the lively discussion!

For something like base I think there is a natural tension between a desire for more features and the desire to avoid breaking changes and major versions. Personally I think there is a wide range between bare bones but stable, and feature rich but unstable that can be seen as reasonable.

The decision made here gives a good idea of where on this spectrum the CLC thinks base should be. And while it's far closer to the "slim but stable" end than I would have expected, I think it's still in the reasonable range.

Given that isByteArrayPinned# was stable for over 5 years, and the primops in question in this proposal would most likely have been just as stable. I think it's fair to treat this vote not just as a vote on these particular primops, but also as guidance for how high the bar for stability is for anything new to make it's way into base.

However there is a vast amount of features currently in base falling short of this bar, and there will be new features doing so in the future. At least for those features somewhat closely coupled to GHC I opened https://gitlab.haskell.org/ghc/ghc/-/issues/25326 and would welcome feedback on how GHC should do things from it's side there.

---

Sure, I have no problem with that at all. The raw blanket reexport of all primops is indeed highly experimental, and doesn't belong in base. Until it is removed, we'll never be able to achieve the goal of having the same base version across two compilers, liberating us from bound-bumping hell.

At least technically when adding primops additional exports only require minor version bumps. There is no reason why we couldn't have base-x.y.z without newPrimop# and base-x.y.(z+1) with newPrimop#. This would allow newer versions of ghc to still work with older versions of base without issue.

Or am I missunderstanding your point? In terms of stability most primops have been very stable over time.And I agree that primops that are expected to be unstable in their behaviour or interface shouldn't be exported from base.

---

As far as I understand it, when GHC developers want to add a feature to GHC they can do so without involving the CLC. If their feature requires something to be exposed from a library then, as you suggest, they can do that too, from a variety of libraries controlled by GHC HQ, including ghc-prim, and now ghc-internal and ghc-experimental too. There's no need to involve the CLC in that process either. They can also develop stable packages through which to expose these library features, or any community member can do so, or any community member can propose the features for inclusion in base at a later date.

My immediate suggestion to GHC developers would be: if, in the first instance, you expose new functionality through packages under the purview of GHC HQ then there's no way that CLC can stand in your way. Is that a good enough first step? After that is an established expectation then we can think about how to improve things further.

I think for new features that’s reasonable. After all there are no existing users that could be inconvenienced! But it becomes more difficult for changes like this one.

I proposed to expose those primops via base because this makes it easy for people depending on isByteArrayPinned# via base already to work around some of the issues with this primop that have been recently discovered.

This has been rejected and that is fine. But a lot of the arguments left me with the impression that the rejection had less to do with how base is, but with how base should be in the future.

So now ghc maintainers and users of base alike have to find new ways to work around this. Does someone create a package that gives a more stable interface? Should packages just depend on ghc-experimental? Something else? Who has the time to do any of that? All that causes additional friction justified by the premise that it will be better when those primops will be removed or reorganized in base at some point by someone with no concrete plan in place.

It would probably be better to have a ghc-all-array-things or similar package, and to deprecate all these primops in base. But implementing a well designed library takes time and competes with a lot of other tasks for ghc maintainers.

This isn’t helped by expectations about how many resources GHC HQ can contribute to such efforts which doesn’t seem to be in line with what’s currently possible.Causing further frustration as expectations on various sides are at odds with each other. Inevitably causing frustration on all sides.

[tomjaguarpaw]

The decision made here gives a good idea of where on this spectrum the CLC thinks base should be. And while it's far closer to the "slim but stable" end than I would have expected, I think it's still in the reasonable range.

Given that isByteArrayPinned# was stable for over 5 years, and the primops in question in this proposal would most likely have been just as stable. I think it's fair to treat this vote not just as a vote on these particular primops, but also as guidance for how high the bar for stability is for anything new to make it's way into base.

I don't actually get the impression that the negative votes were to do with "stability" directly. Mine certainly wasn't. Of the four negative votes only @mixphix's mentioned stability. Even if I was given a sworn affidavit that these primops would never change for the next 100 years I still wouldn't want them exposed from base because they are specifically GHC concerns, and should be provided from a GHC-specific package for everyone's benefit: GHC HQ benefits because they don't need CLC's approval to add new ones (or change them), CLC benefits because it's out of the loop, users benefit because it is much clearer who has responsibility for the package.

I think for new features that’s reasonable. After all there are no existing users that could be inconvenienced! But it becomes more difficult for changes like this one.

What distinction are you drawing that makes this change not a feature?

Does someone create a package that gives a more stable interface?

That would be my suggestion, yes. If I were a GHC developer then one thing I imagine I might do is advocate for a package ghc-external to put everything within GHC's remit that is believed to be stable and supported for external use. Alternatively I might advocate for ghc-prim-external to put all primitives that should be visible externally. There are many ways to cook this, of course.

I may have missed a good reason that this is unworkable, because I'm not a GHC developer and so I am missing context. Is it unworkable? If so, why?

Who has the time to do any of that?

I'm curious what aspect of creating a new package, purely for re-exports, is time consuming. Could you please elaborate?

[Bodigrim]

That would be my suggestion, yes. If I were a GHC developer then one thing I imagine I might do is advocate for a package ghc-external to put everything within GHC's remit that is believed to be stable and supported for external use.

This is a very tempting position to take for CLC, but I'm afraid that kicking the can down the road does not help anyone. The only effect of this pursuit of stability for base will be an increasingly fragile ecosystem, depending on ghc-internal, ghc-external, ghc-experimental, etc. Sure, CLC would be able to claim virtuously "not our problem", but this gives me little solace.

[tomjaguarpaw]

I don't suggest anyone depend on ghc-internal or ghc-experimental. I suggest that anyone who wants to depend on something within the remit of GHC HQ advocates for them to make it stable and export it from a stable package (or even takes the initiative and makes such a package themselves).

The status quo is that the CLC forces GHC HQ to make their library exports stable by asserting control over base. Are you suggesting that once CLC stops forcing their hand, GHC HQ will no longer be able to keep their library exports stable themselves? That sounds very pessimistic to me.

I have every confidence that stable packages will be produced, within GHC HQ's remit, for everything that they want to make available to end users. (Admittedly, it's less clear to me how the transition period will work.)

[Bodigrim]

Free work is in short supply, so I'm pessimistic. But we can say "not our problem" indeed.

[tomjaguarpaw]

I mistakenly assumed that everyone was already on the same page about how the issue was to be resolved. Perhaps @angerman, @hasufell and @mixphix can share their thoughts about where they assumed these primops would live if not base, because if they have yet another idea, that doesn't match mine, that would be rather concerning and suggest that we do some work reaching a shared understanding before tackling issues like this again.

[Bodigrim]

AFAICT no one volunteered for anything else than exposing these primops from ghc-prim / ghc-experimental, which pushes every package dealing with conversions between pinned / unpinned memory (bytestring, text, primitive, but it also applies to a wide range of packages dealing with FFI) to depend deeper on unstable part of the ecosystem. As a maintainer of some of these packages, I'm unhappy about it, because it means more free (and unsatisfying) work for me.

It is a fine line to walk. IMO there is no point in dogmatic cleansing of base: you might achieve a rock solid, reinstallable and compiler-agnostic package, which fits absolutely no one's purpose to base their work upon.

[tomjaguarpaw]

If I make a package for you that exposes the primitives you need for the purposes mentioned above (byte-array-primitives maybe?) will that resolve your concern about that specific issue? (I wouldn't maintain the package indefinitely, but at least long enough to get the ball rolling and prove it's a viable idea.)

[Bodigrim]

Boot packages such as bytestring / text cannot depend on non-boot packages, so we would have to make byte-array-primitives a boot package as well. This is a high cost. And if you are not going to maintain it forever then who will and how much stability guarantees I can expect? None, because GHC developers are in their right to nuke such primops at any convenient moment. We are just making our dependency graph bigger and incur additional points of failure. I don't see how this is more appealing than even depending on ghc-prim / ghc-experimental.

There is no sizeable chunk of ecosystem which depends on "pure" part of base only. If you aim to work towards pruning base a lot more design and planning is needed, and a sole introduction ghc-internal was not meant to serve such purpose. (I recall that early versions of the split base proposal argued to such effect, but the ratified one does not)

[tomjaguarpaw]

@mpickering and @AndreasPK have both suggested that someone from outside GHC HQ could make a package to provide a stable interface to those primops.

then anyone in this thread is free to make their own libraries to provide stable interfaces for these primops

It would probably be better to have a ghc-all-array-things or similar package, and to deprecate all these primops in base. But implementing a well designed library takes time and competes with a lot of other tasks for ghc maintainers.

Therefore, I'm willing to put my free work where my mouth is. I'm willing to make and maintain the boot package. I don't believe it can actually be very hard. Maybe I'm terribly naive, but if so I'll come back with a convincing report in six months about how we really ought to reexport the primops from base. At the very least we'll all have learned something.

No one maintains a package indefinitely. We all donate our free work and good faith efforts to the best of our abilities. That's what I commit to do in this initiative.

[hasufell]

Yes, it's time that GHC is getting a proper designed API with a set of packages that don't randomly change.

Afair, there was a HF project to help with that: https://discourse.haskell.org/t/charting-a-course-toward-a-stable-api-for-ghc/7646

I don't know what happened to it.

However, abusing base for such purposes is wrong. We need better boundaries.

---

Edit: to be clear, I'm aware that the linked HF project was about GHC API (the package). I'm suggesting this needs to be expanded to whatever GHC HQ is producing and those efforts are the responsibility of GHC HQ.

[Ericson2314]

We are just making our dependency graph bigger and incur additional points of failure. I don't see how this is more appealing than even depending on ghc-prim / ghc-experimental.

@Bodigrim I guess this is why I signaled you out. I really believe this sort of reason is 100% backwards -- more smaller libraries and a fine-grained dependency graph is more stable. It is (for sake of argument) the same code, and thus the same points of failure, but failures are less contagious because there are fewer spurious dependencies.

I really would like to convince you of this, because ultimately I think we want same/compatible things, but I have no idea how to do so.

---

Thank you so much @tomjaguarpaw for agreeing to create the package. I expect the experience to be quite easy, and I hope everyone can learn from that.

[AndreasPK]

Even if I was given a sworn affidavit that these primops would never change for the next 100 years I still wouldn't want them exposed from base because they are specifically GHC concerns, and should be provided from a GHC-specific package for everyone's benefit

Very few people gave explicit reasons and the majority of reasons given seemed to had to do with stability/maintainability.
So while I I might not agree with your position it's helpful of you clarify your position in this aspect.

Personally I think base should be as wide as it can be without affecting stability but it seems I'm a minority in this regard.

What distinction are you drawing that makes this change not a feature?

I consider the feature "allow me to check if I need to copy this bytearray for ffi". The new primops don't allow you to do something "new" in that regard because isByteArrayPinned# already exists. It just allows one to do the same thing better.

In my mind that's similar to adding a method to a class, a field to a datatype etc.

Who has the time to do any of that?

I'm curious what aspect of creating a new package, purely for re-exports, is time consuming. Could you please elaborate?

It's easy to create a new package by just throwing all kinds of re-exports into it without giving it too much though.

But I think if this happens one should sit down and carefully think about which concerns the package should cover and what functionality it should therefore expose and how the module structure should look like.
How many packages should there be? Should their functionality overlap? Do we create only a primops package? Does that expose all primops or do you skip some for stability concerns? Or should Is it just an array package? Should there be a primops package and an array package? I think it's a big design space and finding a good point in that takes effort imo.

If the goal is just to make "GHC.Exts" but in a different package of course that's easy. But the benefits of approaching it that way seem questionable. It would most likely not be any more stable than ghc-internal at that point. That wouldn't really help anyone.

Yes a package filling this role doesn't need to be perfect but even doing a good job already takes effort.

Therefore, I'm willing to put my free work where my mouth is.

If you want to contribute to efforts there I would love for you to give feedback on https://gitlab.haskell.org/ghc/ghc/-/issues/25326 which is about more than just ghc-experimental. Especially on the parts which you intend to cover in your package.

@mpickering and @AndreasPK have both suggested that someone from outside GHC HQ could make a package to provide a stable interface to those primops.

Yes there is no reason why a boot package can't be maintained by someone outside of GHC HQ. It just requires some coordination with GHC HQ especially around ghc releases.

[tomjaguarpaw]

Very few people gave explicit reasons and the majority of reasons given seemed to had to do with stability/maintainability.

Those reasons were primarily not given by members of the CLC, as far as I can see.

I'm curious what aspect of creating a new package, purely for re-exports, is time consuming. Could you please elaborate?

It's easy to create a new package by just throwing all kinds of re-exports into it without giving it too much though.

But I think if this happens one should sit down and carefully think about which concerns the package should cover and what functionality it should therefore expose and how the module structure should look like.

I agree, though that's asking much more than the status quo! Currently we just throw all (well, a subset of) primitives into GHC.Exts and call it a day.

Therefore, I'm willing to put my free work where my mouth is.

If you want to contribute to efforts there I would love for you to give feedback on https://gitlab.haskell.org/ghc/ghc/-/issues/25326 which is about more than just ghc-experimental. Especially on the parts which you intend to cover in your package.

Sounds like you are well ahead of the game. Excellent! I'll continue the discussion there.