Bogus build failures on HackageDB

[bos]

(Imported from Trac #598, reported by mboes on 2009-10-10)

HackageDB reports build failures for packages due to inconsistent dependencies. For instance, the dedukti package currently has a build failure because it depends on both bytestring and text. The latest version of bytestring is 0.9.1.5, but text is built using bytestring-0.9.1.4. This causes a build failure because the functions in text expect as arguments something of type ByteString? from bytestring-0.9.1.4, not something of type ByteString? from package bytestring-0,9.1.5.

If the text package were built using the latest available version of bytestring, as would happen after a "cabal install text" on a fresh system, the problem would go away. Always building packages with the latest versions of their dependencies available isn't a solution, because dedukti could, for instance, depend on some specific version of bytestring yet text might happen to be built with another version on the Hackage server.

Could there be a way to at least detect when build failures are due to inconsistent versions in dependencies, which is a problem specific to the Hackage server installation and not the package itself?

[bos]

(Imported comment by mboes on 2009-10-10)

Build log for dedukti as of 2009-10-10

[bos]

(Imported comment by mokus on 2009-10-10)

I'm getting extremely frustrated by situations very much like this one. the bytestring-0.9.1.5 update was particularly nasty and caused bogus failures in every package I uploaded for a couple months, to the point where i added hacks to one project's cabal file for the sole purpose of making it build on the hackage server. It really should not be possible for a .._.x version bump to wreak this kind of havoc.

More recently, I uploaded a new version of my 'random-fu' package and it failed due to some problem which prevented the non-negative package from linking. I have not looked into it just yet, but it looks like another of the same issues - essentially bitrot in the ghc-pkg database on whatever system does the hackage builds. This is getting out of hand, in my opinion.

I propose a few alternative solutions, varying in directness and complexity:

1) all packages be rebuilt occasionally (say, monthly) from a clean Haskell-Platform install (preferably also updating the info on the web for packages that successfully rebuild)

2) keep stats on reverse-dependency build failures, use a pagerank-like tool to propagate those stats up the graph, and rebuild packages and their dependencies when scores start to plummet (which presumably would have happened to bytestring shortly after 0.9.1.5 was uploaded and huge numbers of projects referencing it would not build)

3) Just build *every* upload in its own clean package.conf environment as a fallback (instead of or in addition to the preferred-versions fallback), rebuilding all the package's dependencies as well like "cabal install" does.

That last should be fairly simple to implement and would make a lot of sense to me, especially given that I think most users do something of the kind in their own pre-upload testing. I know I do anyway.

I would personally be satisfied if I could just get the haddock documentation to appear on the package archive page when the build fails. People can and will skim the package documentation and try building a package themselves if the package sounds interesting to them, but they're extremely unlikely to build the documentation just so that they can discover whether the package is interesting to them.

[bos]

(Imported comment by ross on 2010-01-11)

Replying to mokus:

3) Just build *every* upload in its own clean package.conf environment as a fallback (instead of or in addition to the preferred-versions fallback), rebuilding all the package's dependencies as well like "cabal install" does.

Is there a sequence of cabal-install commands to do that (including the options --html-location, --hyperlink-source, --hoogle and --haddock-option)?

[bos]

(Imported comment by @dcoutts on 2010-01-11)

The simplest solution would be to use cabal-install rather than runghc Setup. cabal-install will rebuild things as appropriate to avoid the diamond dependency problem.

[bos]

(Imported comment by ross on 2010-01-12)

Replying to @dcoutts:

The simplest solution would be to use cabal-install rather than runghc Setup. cabal-install will rebuild things as appropriate to avoid the diamond dependency problem.

It will rebuild package versions that are already installed? Probably saves trouble in the long run, though it would be annoying if the rebuild failed.

[bos]

(Imported comment by @dcoutts on 2010-01-12)

Replying to ross:

It will rebuild package versions that are already installed?

Exactly. It rebuilds them against different versions of their dependencies, so that the overall installation plan (graph) is consistent. It tries where possible to reuse existing installed package instances.

Probably saves trouble in the long run, though it would be annoying if the rebuild failed.

Hopefully it would not matter since you wouldn't be uploading the results of dependent package rebuilds, just the single target package you're interested in.

[bos]

(Imported comment by mokus on 2010-01-13)

Replying to @dcoutts:

Replying to ross:

Probably saves trouble in the long run, though it would be annoying if the rebuild failed.

Hopefully it would not matter since you wouldn't be uploading the results of dependent package rebuilds, just the single target package you're interested in.

I may be wrong, but I think he was referring to the new uploader as the one being annoyed by such a situation, because they would know that there's a version of the package on which their upload depends already built on Hackage. I'd say that this annoyance is an acceptable risk, though - if an old package won't build in a particular configuration of the dependency graph, that's a legitimate build failure of the package which induced that graph, even if the fault is in the dependency. An end user very well may run into that same failure. The depended-upon package's maintainer should probably get notified if that happens, or the build failure should at least be recorded somewhere that the maintainer could easily find it without having to dig through other packages' build logs.

[bos]

(Imported comment by ross on 2010-01-13)

Replying to @dcoutts:

The simplest solution would be to use cabal-install rather than runghc Setup. cabal-install will rebuild things as appropriate to avoid the diamond dependency problem.

To generate the documentation for a package, one must

• build, haddock and install all pre-requisite packages
• do the equivalent of

  runghc Setup.hs configure --ghc --user --haddock-option=--use-contents=http://hackage.haskell.org/package/<pkgid>
  runghc Setup.hs haddock '--html-location=http://hackage.haskell.org/packages/archive/$pkg/$version/doc/html' --hyperlink-source
  runghc Setup.hs haddock --hoogle
  

From hunting through the code, it seems the planning stuff only happens if I use the "install" subcommand, but that doesn't take the haddock options I want to use.

What are the cabal-install commands I should be using?

[bos]

(Imported comment by ross on 2010-01-13)

The client's build environment got messed up by building recent versions of a couple of GHC 6.12.1 core packages with GHC 6.10. There are two problems with that:

• some of them have undeclared dependencies on base-4.2.0.0
• Cabal and the GHC build system build packages with different flags, so that yields a binary incompatibility.

Switching to GHC 6.12.1 fixes those, leaving the diamond dependency problem. Building with cabal-install should fix that, but seems to be blocked by #517.

[bos]

(Imported comment by ross on 2010-01-16)

It's still not possible to build suitable documentation using cabal install due to #517, but the dependency version problem is avoided by building in a fresh environment, using cabal install --dry-run to get the list of packages and then building each using the above commands.