Release for GHC 8.8

[bgamari]

While GHC is quite behind in the alpha cycle this time around due to shuffling around of infrastructure, I would still like to make sure we have plenty of time to stabilize submodules before the final release in February. Our original plan stated that core library version numbers would be set by mid-December and core library releases for 8.8 would be ready by late January. How are we looking compared to these benchmarks?

[hvr]

@bgamari btw, I was in the midst of updating lib:Cabal in GHC HEAD to a more recent snapshot and got side-tracked cause there are some API adaptation necessary and those exposed some bugs that needed to be fixed before the update can proceed... and this prompted #5812 ... and.... long story short... we need to resolve this before it even makes any sense to bump the version (which we should be able to do soon afterwards; the version bump was supposed to happen together with #5800 ...)

[bgamari]

Any updates on this?

[hvr]

@bgamari #5812 just got merged, and I'm right now trying to fwd port https://gitlab.haskell.org/hvr/ghc/commit/d6428a5e3533e0c94a68f854f8b543f6a52405b1 to it

[bgamari]

How is this looking?

[bgamari]

How close are we to being able to pin down a final submodule commit?

[23Skidoo]

@fgaz @hvr What's the state of the public sublibraries feature? Do we need anything else on the lib:Cabal side?

[fgaz]

I need to write a fallback for ghc<8.8 in #5848, then multilibs will be complete

[hvr]

@23Skidoo if we're talking about an actual hard freeze of 3.0, I think there's a few loose ends (one of which might be the 3.0 variant of #5837 which might require changes to lib:Cabal, @fgaz might know this) which should ideally be resolvable within 1-2 weeks; weekend's coming up, so I'll be able to take a closer look a the 3.0 situation and give a more concrete answer

[fgaz]

#5837 requires no changes to lib:Cabal

[fgaz]

There's also this parsing bug: #5846

But It's on new functionality and there are no regressions, so it it's not super important

[bgamari]

Thanks everyone! It sounds like we are converging. It would be great if we could have this sorted out in the next week or so.

[bgamari]

How are things going here?

[fgaz]

Public sublibraries: I'm awaiting review on #5848

[bgamari]

This is really starting to become urgent. Given the how far behind GHC's release schedule is I really do not want to start the GHC pre-release process before Cabal has finalized its version number. This is not to say that Cabal-3 has to be final; I merely want to avoid having any boot library version bumps after the first pre-release is out.

[harpocrates]

Would it be possible to know what the expected version is for this release of Cabal? That way, Haddock can tentatively declare support for it (and provide a tentatively final submodule commit for GHC).

[23Skidoo]

@bgamari

@hvr told me he plans to bump the versions on master soon.

@harpocrates

It should be 3.0.

[hvr]

See #5915

[bgamari]

Note that currently GHC's bkpcabal01 test fails with Cabal master (e.g. see https://gitlab.haskell.org/ghc/ghc/tree/ghc-8.8). I have opened a GHC ticket to track this. Judging by the debug output on the ticket I'm reasonably certain this is a regression; there are no appreciable differences in the package database that Setup.hs configure is passed yet with Cabal 3 it fails.

[23Skidoo]

/cc @ezyang

[ekmett]

Is there anything still blocking this? I ask mostly because I'm quite eager to be able to use multiple sub-libraries in the wild.

[fgaz]

@ekmett
Regarding #5848 I have to fix the failing tests (quick: I can just blacklist properly fix ghc <8.2 (done!)) and add the flag mentioned in the comments (quickish). I think I'll have some time later this week.

edit: all done, I'm awaiting review

[bgamari]

At this point Cabal, unix, parsec, and text are the only libraries which GHC lacks releases for. Given the number of changes present, Cabal is the most concerning to me. Please, let's try to wrap this up this week.

[phadej]

There's also #6033 and other milestone 3.0 things: https://github.com/haskell/cabal/issues?q=is%3Aopen+is%3Aissue+milestone%3A3.0

[23Skidoo]

Sorry, I was moving and wasn't able to spend much time on managing the next release lately.

Most of the "enhancement" tickets targeted for 3.0 can be postponed, but we should merge the library visibility thing and the dist/v2 rename. With #6033 it depends on whether @hvr can finish it in time, otherwise it'll have to wait until a point release.

[bgamari]

Thank you, @23Skidoo. I'll move ahead with the next 8.8.1 alpha using 5d25853. However, it would be great if we could finish this up this week. I would like to cut the first (and hopefully last) release candidate on 21 June.

Good luck with your move!

[bgamari]

Has there been any motion on this? Alpha2 went out and nearly all of the other submodules have been finalized. I still plan to cut the RC on 21 June, but this will require a final, releasable commit from Cabal.

[bgamari]

Ping.

[23Skidoo]

@bgamari I cherry-picked some additional patches to the 3.0 branch and created a Cabal-v3.0.0.0-rc1 tag. Unless something currently unforeseen happens, this commit will become the 3.0.0.0 final release of lib:Cabal.

[bgamari]

Mikhail Glushenkov <[email protected]> writes:

@bgamari I cherry-picked some additional patches to the `3.0` branch and created a [`Cabal-v3.0.0.0-rc1`](https://github.com/haskell/cabal/releases/tag/Cabal-v3.0.0.0-rc1) tag. Unless something currently unforeseen happens, this commit will become the 3.0.0.0 final release of lib:Cabal.

Thanks!

[bgamari]

@hvr, I am reasonably certain that c5395b8 is broken. The GHC build fails with linker errors when built against v3.0.0.0-rc1.

@23Skidoo, can we drop this refactoring from 3.0?

[23Skidoo]

We can, though this patch should have only affected packages that use {asm,cmm}-sources.

[bgamari]

Yes, GHC uses both; if I'm not mistaken cmm-sources was actually introduced specifically for GHC.

[23Skidoo]

I'll wait for @hvr to comment before reverting it.

[quasicomputational]

@23Skidoo, can you also cherry-pick #6127 (7fec503) for 3.0, please?

[bgamari]

@23Skidoo, how can we get this unstuck? I've been trying to get a hold of @hvr via email, IRC, and GitHub for a week now and have heard nothing. GHC 8.8 is currently blocked on this issue.

[23Skidoo]

@bgamari @quasicomputational I've cherry-picked 7fec503 into 3.0, reverted c5395b8, and tagged Cabal-v3.0.0.0-rc2.

[bgamari]

Thank you @23Skidoo!

[hvr]

@bgamari if we drop this, we're back to square one, and we won't be able to upload certain boot libraries to hackage...

[bgamari]

@hvr, by all means please do fix the patch if you have the time. However, at the moment my understanding is that there is no one with the time to do so and the GHC release needs to move on.

Can you clarify which boot libraries we will be unable to upload? cmm-sources has been used by basesince 2017 yet we haven't had any trouble uploading it so far. What has changed?

Other than base the only other boot library which uses {cmm,asm}-sources (other than the RTS, which we don't upload) is ghc-heap. Even if it is true that we won't be able to push ghc-heap to Hackage I don't consider this to be a reason to stall the release: To date ghc-heap has no presence on Hackage but this doesn't seem to have bothered users. Another GHC release without a having ghc-heap on Hackage doesn't seem like the end of the world to me.

[phadej]

Ryan have bothered Herbert multiple times, eg in haskell-infra/hackage-trustees#203

…

Sent from my iPhone

On 20 Jul 2019, at 18.38, Ben Gamari ***@***.***> wrote: @hvr, by all means please do fix the patch if you have the time. However, at the moment my understanding is that there is no one with the time to do so and the GHC release needs to move on. Can you clarify which boot libraries we will be unable to upload? cmm-sources has been used by basesince 2017 yet we haven't had any trouble uploading it so far. What has changed? Other than base the only other boot library which uses {cmm,asm}-sources (other than the RTS, which we don't upload) is ghc-heap. To date ghc-heap has no presence on Hackage but this doesn't seem to have bothered users; another GHC release without a push to Hackage doesn't seem like the end of the world to me. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[phadej]

FWIW, I'm very tempted to rip off the whole cmm-sources thing from Cabal as only GHC uses it, and GHC could revert to use whatever (external definition) hack was there before; until a proper patch to Cabal is provided. The functionality exists in Cabal, but to my understanding you cannot build a package with asm-sources using Cabal / cabal-install, only parse the package definition.

Why the incomplete cmm-sources patch was accepted into Cabal in the first place. There are way too many examples of how "let's merge now and fix later" doesn't work in this project.

While those buildinfo fields were added to the parser some time
ago via 57d7f28 and
4a28765 that work was never completed
by implementing the necessary build/sdist logic in Cabal.

This is a rant. I'm not happy.

[bgamari]

Ryan have bothered Herbert multiple times, eg in haskell-infra/hackage-trustees#203

The problem here does not appear to be cmm-sources. Afterall, base uses cmm-sources and that is present on Hackage. I believe the ticket you are citing is not due to any technical issue but rather the busy schedule of a core community member.

There are way too many examples of how "let's merge now and fix later" doesn't work in this project.

Perhaps I am missing some context here but I am a bit perplexed: none of the issues being fixed now were mentioned to @angerman during code review (#4857), as far as I can tell.

Regardless, I agree with the sentiment; merging half-finished patches is almost always something that I come to regret. It is quite unfortunate that we have arrived at this point. Had I known that the cmm-sources patch was not in an acceptable state I would have pushed back against relying on it in GHC. However, this is water under the bridge at this point; we are now in a situation where released (and soon-to-be-released) compilers depend on the feature.

The decision of how to proceed at this point is squarely in Cabal's court. If you decide to rip out cmm-sources then we will need to make the appropriate adjustments in GHC. If we really think that Cabal-3.0.0.0 is not releasable without @hvr's patch and @hvr can't find the time to finish it himself then it is possible that I could find some time to debug the linking issue myself.

[phadej]

The problem here does not appear to be cmm-sources. Afterall, base uses cmm-sources and that is present on Hackage. I believe the ticket you are citing is not due to any technical issue but rather the busy schedule of a core community member.

if you look closely, the ghc-heap is still not uploaded: https://hackage.haskell.org/package/ghc-heap

I'm not 100% sure, but I suspect that the case is that there's nothing special in ghc-heap (except cmm-sources or whatever) so if someone decides to reinstall it, there should not be any blockers for that. Except they cannot, as build doesn't work with those. So uploading ghc-heap to Hackage, could confuse someone (unlikely bugs expose themselves at the moment you less want them to occur).

base on the other hand is hardcoded into cabal-install:

cabal/cabal-install/Distribution/Client/Dependency.hs

Lines 400 to 405 in 0d2d9ca

	, pkgname <- [ mkPackageName "base"
	, mkPackageName "ghc-prim"
	, mkPackageName "integer-gmp"
	, mkPackageName "integer-simple"
	, mkPackageName "template-haskell"
	]

Currently cabal-install will never try to upgrade base. Except that people do work on making base reinstallable.

So it boils down to:

• fix cmm-sources
• hardcode ghc-heap to be non-upgradeable package

---

Perhaps I am missing some context here but I am a bit perplexed: none of the issues being fixed now were mentioned to @angerman during code review (#4857), as far as I can tell.

I would blame @hvr as well, e.g. in haskell-infra/hackage-trustees#179 issue Herbert edited the issue (in October 2018) adding "ghc-heap (blocked on Cabal bug)" but I cannot find any issue on Cabal tracker about that.

I also don't know whom to blame on the fact that #4857 doesn't have tests. If I had seen that kind of PR today, I'd require at least parsing code and cabal-version: annotations. So I actually could remove the parsing code, and test-suite will still pass. Maybe writing tests would trigger a review commend "should we try to actually compile a package as well", maybe not.

---

Regardless, I agree with the sentiment; merging half-finished patches is almost always something that I come to regret.

Ironically, cherry-picking #6033 to 3.0 branch, was kind of that.

I've backported the current state of this PR to the 3.0 branch via c5395b8 and this PR remains open for the express purpose to fix the outstanding issues with the preprocessor generated sources

Yet, without putting that there linking issue wouldn't be found, as that PR also doesn't have tests.

---

So my opinion (which you really don't need to pay attention of):

• reverting Herbert patch is a solution to 3.0 release problem
• But I'd remove cmm-options stuff from master, and someone could reimplement that feature properly next time.

[hvr]

@hvr, I am reasonably certain that c5395b8 is broken. The GHC build fails with linker errors when built against v3.0.0.0-rc1.

The most likely reason for this is that cabal-version: wasn't bumped to 3.0 -- i.e. in prior versions of the cabal spec this field doesn't exist and is treated as a warning or error, depending on how ghc-cabal operates. @bgamari is currently verifying this

[bgamari]

Alright, I believe we have identified the issue: ghc-heap.cabal stated it's cabal-version was 2.1, yet cmm-sources is unsupported (and consequently ignored) in versions earlier than 3.0. After fixing the cabal-version things seem to be building (although I'm still waiting for a full validation).

The warning issued by Cabal to draw attention to the invalid cmm-sources field was unfortunately drowned out in the other noise of the build. However, even once I found it it's quite unclear that the field is ignored:

Warning: ghc-heap.cabal:30:3: The field "cmm-sources" is available only since the Cabal specification version 3.0.

I will open a pull request making this message a bit clearer.

[23Skidoo]

OK, I guess you will need an rc3 tag then too, which reverts the revert?

[bgamari]

@23Skidoo that would be great, yes.

[23Skidoo]

@bgamari Created the Cabal-v3.0.0.0-rc3 tag.

[hvr]

This can obviously be closed given the existence of

• https://downloads.haskell.org/cabal/cabal-install-3.0.0.0/
• http://hackage.haskell.org/package/cabal-install-3.0.0.0

[lspitzner]

cabal-install still has a base >= 4.8 && < 4.13 bound.

[hvr]

@lspitzner This ticket is mostly about lib:Cabal for which a GHC-8.8 compilable version had been released alongside GHC 8.8.1 (otherwise there couldn't have been a GHC 8.8.1 rls to being with); note that exe:cabal is in fact compatible with GHC 8.8.1; you just can't build it with it yet.

You're probably referring to #6328 instead which is about having exe:cabal compilable by GHC 8.8.1 as well; and we're already at 3.0.1.0-rc3