Time to fork bytestring into a stable legacy and new feature releases unblocking progress.

[vdukhovni]

The bytestring package has for some time been mired in a backwards-compatibility straitjacket. Support for GHC going back all the way to ~7.0 makes it difficult to release updates that take advantage of new GHC features.

I propose that the after 0.10.10.1 is cut, the next bytestring version should be 0.11.0.0 (or 1.0?), and should require the version of base included with GHC 8.12.1. This can merge #191, #174, and any other worthwhile PRS that are challenging to support with GHC 7.

This would not mean that bytestring becomes unsupported with GHC 7.x, and early 8.x releases. Rather a separate stable branch can still be used to fix bugs or backport worthy features that would be released as new 0.10.x versions. Presumably, when cabal is choosing a version of bytestring to build with GHC 7.x and early 8.x, it would not choose one that requires base from 8.12.

It might even make sense to make the base version constraint for the new bytestring be strict equality, rather than a floor, and cut a new version with each release of base. This stops short of actually merging bytestring into base, but recognises that it has not really been possible to make independent updates anyway, because bytestring is a boot library bundled with GHC.

One way or another, the current stasis should I think be resolved.

[chessai]

There is a lot of truth to what you are saying, but I disagree with the way forward, and I certainly disagree with a fork. At this point that ship has sailed.

The first issue is that of forking bytestring: You will never get enough adoption. That is just a reality. I say this as someone who has spent a large portion of their time using an alternative ecosystem to bytestring written by @andrewad and me.

Here is my take on supporting older GHCs: There is a reason it is useful. Mostly to track compiler regressions across a wide range of compilers. However, GHC 7 is very old. Not only in terms of the number of releases since, but also in years. It has been almost a decade since its release. When GHC 7 came out, I was a freshman in high school, and I still had a full head of hair. Why should we continue to support such old compilers? In my opinion, we shouldn't. Recently, primitive, random, and vector have all dropped support for anything pre-7.10. For bytestring, I can see doing that, but I do not think dropping anything post-7.10 is wise right now. (7.10 came out 5 years ago). Even if the lower bound is up for debate, there is no way a new version of bytestring should have 8.12 as a minimum version of base. The maintenance burden incurred by versions needing 8.12 is small; just some CPP that can eventually be removed, maybe some years down the road.

What I think is preferable is to actually pool all of the changes that require a majour version bump into a branch, keep it rebased on top of master, and then release them all at once.

Also, I think this issue comes with the very understandable frustration that there has been a lack of maintainership for too long. I have added 4 new maintainers just today, 5 if you count myself (though I probably won't be doing much in the way of PRs). So hopefully your stress will be alleviated by their efforts.

Those maintainers are: @sjakobi, @hsyl20, @bgamari, @Bodigrim.

EDIT: primitive has not done this yet, but it will soon.

[vdukhovni]

I don't actually mean forking the project, I mean maintaining multiple stable releases, with appropriate base constraints. So there's no issue with adoption, the package is still bytestring, but there are multiple actively supported versions, all compatible with the vast majority of projects, but some able to take advantage of newer facilities in base, and perhaps offer some new features.

For example, I am a co-maintainer of Postfix (admittedly an MTA not an API), and we support four concurrent stable versions, each year one is retired, and a new one is released. Each version is supported for four years.

[chessai]

What are the benefits of that over using CPP to provide the same benefits? Now there are multiple source trees to keep in sync.

…

On Wed, Jun 24, 2020, 11:16 PM Viktor Dukhovni ***@***.***> wrote: I don't actually mean forking the project, I mean maintaining multiple stable releases, with appropriate base constraints. So there's no issue with adoption, the package is still bytestring, but there are multiple actively supported versions, all compatible with the vast majority of projects, but some able to take advantage of newer facilities in base, and perhaps offer some new features. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#234 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEOIX2YTQJWLMQKVLLVO6Q3RYLTVBANCNFSM4OH4NVNQ> .

[vdukhovni]

CPP should be used exceedingly sparingly, mostly to import FFI constants and the like into modules that offer interfaces to kernel APIs.

Otherwise, conditional compilation of features, and especially conditional selection of implementation of (fragments of) functions can make code really difficult to maintain. Thus, for example when introducing PatternSynonyms, just cut a new major release that requires at least base for 8.0, and skip CPP. To the extent that the older release still warrants support, bugfix or backport as appropriate.

The bytestring library cannot make new major releases that conflict with the bundled boot library, because it causes massive "recompile the world" dependency churn. It makes a lot more sense to not introduce new bytestring versions except when users migrate to a new version of GHC. Then they recompile everything anyway.

[vdukhovni]

As to keeping source trees "in sync", the whole point is that the older branches are stable releases, barring significant bugs, they would not change much if at all, and would NOT be "kept in sync" with the evolving head branch.

[vdukhovni]

The main thing I'm really suggesting is to unblock making releases, by setting the minimum (or strict equality) constraint on base to the next version of base, and so avoiding introducing changes for users who are doing fine with older releases...

This means releases of bytestring largely coincide with releases of GHC, and may be pinned to the specific GHC base version, but don't have to be directly maintained by the compiler team, who are busy enough...

[andrewad]

The first issue is that of forking bytestring: You will never get enough adoption. That is just a reality. I say this as someone who has spent a large portion of their time using an alternative ecosystem to bytestring written by @andrewad and me.
Just a heads up that you meant a different Andrew

[bgamari]

This stops short of actually merging bytestring into base, but recognises that it has not really been possible to make independent updates anyway, because bytestring is a boot library bundled with GHC.

Putting aside the other suggestions for a moment, I would strongly advise against this. The fact that boot libraries are often essentially frozen in an install plan is currently a headache, but is at least can be avoided by not depending upon the ghc library. By freezing bytestring to a particular base release you needlessly make this headache far worse. Specifically:

• users lives get harder, since they can only use the bytestring release shipped with their compiler
• maintainers lives get harder, since they must now backport changes which otherwise users at least had a chance of getting with a simple cabal update under the status quo
• bytestring developer's lives get harder as they now are tied to a particular compiler when working on the library.

All-in-all, I'm unclear on what benefit we get in return for all of these issues. Yes, there are certainly ways in which bytestring can be improved, many of which are embodied by the open PRs. However, I don't believe that breaking backwards compatibility is necessary or desirable in a package as critical to the ecosystem as bytestring.

While it's true that CPP does impose some overhead, this is simply the cost of maintaining such a central package.

[vdukhovni]

But a major side-effect of not tying bytestring to a particular version of base has been that it has been very difficult to make any releases at all. At least @hvr has been very reluctant to introduce any changes that might affect existing users of older GHC versions. And also very reluctant to cause build-cache churn when a new version of bytestring appears, and everything that didn't pin the boot library version gets recompiled. It is also not quite clear to me (at least in the case of bytestring) that there are users clamouring for bytestring feature improvements that are separate from taking advantage of new GHC versions.

So, while I understand your point, and it is in principle valid, in the specific case or bytestring things haven't been playing out that way as far as I can see... Perhaps there are other solutions, but I think examining the release model and how integrates with GHC is warranted, even if what I proposed isn't quite it...

[Bodigrim]

I think dropping support of GHC < 7.10 is a right thing to do in 2020 and it will remove a lot of friction. It is insane to ignore all progress done and not to benefit from new features.

random has dropped support of GHC < 7.10 and the next release of vector will follow suit. I suggest we do the same here.

[vdukhovni]

I think dropping support of GHC < 7.10 is a right thing to do in 2020 and it will remove a lot of friction. It is insane to ignore all progress done and not to benefit from new features.

random has dropped support of GHC < 7.10 and the next release of vector will follow suit. I suggest we do the same here.

In that case, how about a follow-on bytestring release soon after 0.10.10.1 that sets the floor at GHC 8.0 (base ≥ 4.9) which would be released prior to GHC 8.12.1, intended for inclusion as the boot library for GHC 8.12. That release can merge #174, and #191 as well the CI fixes in #227, and anything else we're holding back that warrants release at that point?

[vdukhovni]

Such a (0.11.x.y or 1.0.x.y) release would still cause build-cache churn for projects that don't set bounds on bytestring, provided they're on GHC 8.0 or later. If that's acceptable, I'm game to see how it turns out...

[Bodigrim]

Let's continue the discussion on release planning in #241.

[vdukhovni]

This looks unlikely to go anywhere at present.