configure incorrectly reports package as broken

[tibbe]

I was working with the new sandboxing support (which uses a package DB in .cabal-sandbox/ghc-<version>-packages.conf.d instead of the user package DB) when I ran into this curious error.

First, Cabal claims that aeson is broken:

$ cabal configure --constraint="hashable == 1.1.2.5" -v --enable-benchmarks --package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d/
'/usr/local/bin/ghc' '--numeric-version'
looking for tool "ghc-pkg" near compiler in /usr/local/bin
found ghc-pkg in /usr/local/bin/ghc-pkg
'/usr/local/bin/ghc-pkg' '--version'
'/usr/local/bin/ghc' '--supported-languages'
'/usr/local/bin/ghc' '--info'
Reading available packages...
Choosing modular solver.
Resolving dependencies...
Could not resolve dependencies:
trying: unordered-containers-0.2.3.0 (user goal)
rejecting: unordered-containers-0.2.3.0:!bench (global constraint requires
opposite flag selection)
trying: unordered-containers-0.2.3.0:*bench
trying: criterion-0.6.2.0/installed-ff8... (dependency of
unordered-containers-0.2.3.0:*bench)
next goal: aeson (dependency of criterion-0.6.2.0/installed-ff8...)
rejecting: aeson-0.6.1.0/installed-d58... (package is broken)
Configuring unordered-containers-0.2.3.0...
Flags chosen: debug=False
Dependency base ==4.*: using base-4.6.0.0
Dependency bytestring -any: using bytestring-0.10.0.0
Dependency containers -any: using containers-0.5.0.0
Dependency criterion -any: using criterion-0.6.2.0
Dependency deepseq >=1.1: using deepseq-1.3.0.1
Dependency hashable >=1.0.1.1: using hashable-1.1.2.5
Dependency hashmap -any: using hashmap-1.3.0.1
Dependency mtl -any: using mtl-2.1.2
Dependency random -any: using random-1.0.1.1
'/usr/local/bin/ghc' '--info'
Using Cabal-1.17.0 compiled by ghc-7.4
Using compiler: ghc-7.6.1
Using install prefix:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0
Binaries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/bin
Libraries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/lib
Private binaries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/libexec
Data files installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/share
Documentation installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/doc
Using alex version 3.0.2 found on system at: /usr/bin/alex
Using ar found on system at: /usr/bin/ar
No c2hs found
No cpphs found
No ffihugs found
Using gcc version 4.2.1 found on system at: /usr/bin/gcc
Using ghc version 7.6.1 found on system at: /usr/local/bin/ghc
Using ghc-pkg version 7.6.1 found on system at: /usr/local/bin/ghc-pkg
No greencard found
Using haddock version 2.12.0 found on system at: /usr/local/bin/haddock
Using happy version 1.18.10 found on system at: /usr/bin/happy
No hmake found
Using hpc version 0.6 found on system at: /usr/local/bin/hpc
Using hsc2hs version 0.67 found on system at: /usr/local/bin/hsc2hs
No hscolour found
No hugs found
No jhc found
Using ld found on system at: /usr/bin/ld
No lhc found
No lhc-pkg found
No nhc98 found
No pkg-config found
Using ranlib found on system at: /usr/bin/ranlib
Using strip found on system at: /usr/bin/strip
Using tar found on system at: /usr/bin/tar
No uhc found

However, ghc-pkg disagrees:

$ ghc-pkg list --package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d
.cabal-sandbox/ghc-7.6.1-packages.conf.d
   HUnit-1.2.5.1
   abstract-deque-0.1.6
   abstract-par-0.3.1
   aeson-0.6.1.0
   attoparsec-0.10.3.0
   blaze-builder-0.3.1.0
   cereal-0.3.5.2
   criterion-0.6.2.0
   dlist-0.5
   erf-2.0.0.0
   hashable-1.1.2.5
   hashmap-1.3.0.1
   hastache-0.4.2
   ieee754-0.7.3
   math-functions-0.1.1.2
   monad-par-0.3
   monad-par-extras-0.3.2
   mtl-2.1.2
   mwc-random-0.12.0.1
   parallel-3.2.0.3
   parsec-3.1.3
   primitive-0.5.0.1
   random-1.0.1.1
   statistics-0.10.2.0
   syb-0.3.7
   text-0.11.2.3
   transformers-0.3.0.0
   unordered-containers-0.2.3.0
   utf8-string-0.3.7
   vector-0.10.0.1
   vector-algorithms-0.5.4.2

$ ghc-pkg check --package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d
# empty output

[tibbe]

The top-down solver fails too, but in a different way:

$ cabal configure --constraint="hashable == 1.1.2.5" -v3 --enable-benchmarks --package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d/ --solver=topdown
searching for ghc in path.
found ghc at /usr/local/bin/ghc
("/usr/local/bin/ghc",["--numeric-version"])
/usr/local/bin/ghc is version 7.6.1
looking for tool "ghc-pkg" near compiler in /usr/local/bin
found ghc-pkg in /usr/local/bin/ghc-pkg
("/usr/local/bin/ghc-pkg",["--version"])
/usr/local/bin/ghc-pkg is version 7.6.1
("/usr/local/bin/ghc",["--supported-languages"])
("/usr/local/bin/ghc",["--info"])
Reading installed packages...
("/usr/local/bin/ghc-pkg",["dump","--global","-v0"])
("/usr/local/bin/ghc-pkg",["dump","--user","-v0"])
("/usr/local/bin/ghc-pkg",["dump","--package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d/","-v0"])
("/usr/local/bin/ghc",["--print-libdir"])
Reading available packages...
Resolving dependencies...
applying constraint base 'installed'
applying constraint ghc-prim 'installed'
applying constraint hashable ==1.1.2.5
applying constraint unordered-containers ==0.2.3.0
cannot configure unordered-containers-0.2.3.0. It requires criterion -any
There is no available version of criterion that satisfies -any
Using internal setup method with build-type Simple and args:
["configure","--verbose=3","--ghc","--prefix=/Users/tibell/Library/Haskell/$compiler/lib/$pkgid","--libsubdir=","--datasubdir=","--docdir=$prefix/doc","--user","--package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d/","--enable-benchmarks"]
Configuring unordered-containers-0.2.3.0...
creating dist
searching for ghc in path.
found ghc at /usr/local/bin/ghc
("/usr/local/bin/ghc",["--numeric-version"])
/usr/local/bin/ghc is version 7.6.1
looking for tool "ghc-pkg" near compiler in /usr/local/bin
found ghc-pkg in /usr/local/bin/ghc-pkg
("/usr/local/bin/ghc-pkg",["--version"])
/usr/local/bin/ghc-pkg is version 7.6.1
("/usr/local/bin/ghc",["--supported-languages"])
("/usr/local/bin/ghc",["--info"])
Reading installed packages...
("/usr/local/bin/ghc-pkg",["dump","--global","-v0"])
("/usr/local/bin/ghc-pkg",["dump","--user","-v0"])
("/usr/local/bin/ghc-pkg",["dump","--package-db=.cabal-sandbox/ghc-7.6.1-packages.conf.d/","-v0"])
("/usr/local/bin/ghc",["--print-libdir"])
Flags chosen: debug=False
Dependency base ==4.*: using base-4.6.0.0
Dependency bytestring -any: using bytestring-0.10.0.0
Dependency containers -any: using containers-0.5.0.0
Dependency criterion -any: using criterion-0.6.2.0
Dependency deepseq >=1.1: using deepseq-1.3.0.1
Dependency hashable >=1.0.1.1: using hashable-1.1.2.5
Dependency hashmap -any: using hashmap-1.3.0.1
Dependency mtl -any: using mtl-2.1.2
Dependency random -any: using random-1.0.1.1
searching for alex in path.
found alex at /usr/bin/alex
("/usr/bin/alex",["--version"])
/usr/bin/alex is version 3.0.2
searching for ar in path.
found ar at /usr/bin/ar
searching for c2hs in path.
Cannot find c2hs on the path
searching for cpphs in path.
Cannot find cpphs on the path
searching for ffihugs in path.
Cannot find ffihugs on the path
searching for gcc in path.
found gcc at /usr/bin/gcc
("/usr/bin/gcc",["-dumpversion"])
/usr/bin/gcc is version 4.2.1
searching for greencard in path.
Cannot find greencard on the path
searching for haddock in path.
found haddock at /usr/local/bin/haddock
("/usr/local/bin/haddock",["--version"])
/usr/local/bin/haddock is version 2.12.0
searching for happy in path.
found happy at /usr/bin/happy
("/usr/bin/happy",["--version"])
/usr/bin/happy is version 1.18.10
searching for hmake in path.
Cannot find hmake on the path
searching for hpc in path.
found hpc at /usr/local/bin/hpc
("/usr/local/bin/hpc",["version"])
/usr/local/bin/hpc is version 0.6
looking for tool "hsc2hs" near compiler in /usr/local/bin
found hsc2hs in /usr/local/bin/hsc2hs
("/usr/local/bin/hsc2hs",["--version"])
/usr/local/bin/hsc2hs is version 0.67
searching for HsColour in path.
Cannot find HsColour on the path
searching for hugs in path.
Cannot find hugs on the path
searching for jhc in path.
Cannot find jhc on the path
searching for ld in path.
found ld at /usr/bin/ld
("/usr/local/bin/ghc",["-c","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57483.c","-o","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57483.o"])
("/usr/bin/ld",["-x","-r","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57483.o","-o","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57484.o"])
searching for lhc in path.
Cannot find lhc on the path
searching for lhc-pkg in path.
Cannot find lhc-pkg on the path
searching for nhc98 in path.
Cannot find nhc98 on the path
searching for pkg-config in path.
Cannot find pkg-config on the path
searching for ranlib in path.
found ranlib at /usr/bin/ranlib
searching for strip in path.
found strip at /usr/bin/strip
searching for tar in path.
found tar at /usr/bin/tar
searching for uhc in path.
Cannot find uhc on the path
("/usr/local/bin/ghc",["--info"])
Using Cabal-1.17.0 compiled by ghc-7.4
Using compiler: ghc-7.6.1
Using install prefix:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0
Binaries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/bin
Libraries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/lib
Private binaries installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/libexec
Data files installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/share
Documentation installed in:
/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/doc
Using alex version 3.0.2 found on system at: /usr/bin/alex
Using ar found on system at: /usr/bin/ar
No c2hs found
No cpphs found
No ffihugs found
Using gcc version 4.2.1 found on system at: /usr/bin/gcc
Using ghc version 7.6.1 found on system at: /usr/local/bin/ghc
Using ghc-pkg version 7.6.1 found on system at: /usr/local/bin/ghc-pkg
No greencard found
Using haddock version 2.12.0 found on system at: /usr/local/bin/haddock
Using happy version 1.18.10 found on system at: /usr/bin/happy
No hmake found
Using hpc version 0.6 found on system at: /usr/local/bin/hpc
Using hsc2hs version 0.67 found on system at: /usr/local/bin/hsc2hs
No hscolour found
No hugs found
No jhc found
Using ld found on system at: /usr/bin/ld
No lhc found
No lhc-pkg found
No nhc98 found
No pkg-config found
Using ranlib found on system at: /usr/bin/ranlib
Using strip found on system at: /usr/bin/strip
Using tar found on system at: /usr/bin/tar
No uhc found
("/usr/bin/gcc",["-m64","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57483.c","-o","/var/folders/00/0__yr000h01000cxqpysvccm0019bv/T/57483","-D__GLASGOW_HASKELL__=706","-Ddarwin_HOST_OS=1","-Dx86_64_HOST_ARCH=1","-Idist/build/autogen","-I.","-I/Users/tibell/Library/Haskell/ghc-7.6.1/lib/vector-algorithms-0.5.4.2/lib//include","-I/usr/local/lib/ghc-7.6.1/directory-1.2.0.0/include","-I/usr/local/lib/ghc-7.6.1/unix-2.6.0.0/include","-I/Users/tibell/Library/Haskell/ghc-7.6.1/lib/vector-0.10.0.1/lib//include","-I/Users/tibell/Library/Haskell/ghc-7.6.1/lib/primitive-0.5.0.1/lib//include","-I/usr/local/lib/ghc-7.6.1/time-1.4.0.1/include","-I/usr/local/lib/ghc-7.6.1/bytestring-0.10.0.0/include","-I/usr/local/lib/ghc-7.6.1/base-4.6.0.0/include","-I/usr/local/lib/ghc-7.6.1/include","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/hashmap-1.3.0.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/criterion-0.6.2.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/statistics-0.10.2.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/vector-algorithms-0.5.4.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/monad-par-0.3/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/parallel-3.2.0.3/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/monad-par-extras-0.3.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/random-1.0.1.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/math-functions-0.1.1.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/erf-2.0.0.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/parsec-3.1.3/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/mwc-random-0.12.0.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/hastache-0.4.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/utf8-string-0.3.7/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/ieee754-0.7.3/lib","-L/usr/local/lib/ghc-7.6.1/directory-1.2.0.0","-L/usr/local/lib/ghc-7.6.1/unix-2.6.0.0","-L/usr/local/lib/ghc-7.6.1/filepath-1.3.0.1","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/cereal-0.3.5.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/aeson-0.6.1.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/vector-0.10.0.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/primitive-0.5.0.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/unordered-containers-0.2.3.0/lib","-L/usr/local/lib/ghc-7.6.1/time-1.4.0.1","-L/usr/local/lib/ghc-7.6.1/template-haskell-2.8.0.0","-L/usr/local/lib/ghc-7.6.1/pretty-1.1.1.0","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/syb-0.3.7/lib","-L/usr/local/lib/ghc-7.6.1/old-locale-1.0.0.5","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/mtl-2.1.2/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/transformers-0.3.0.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/hashable-1.1.2.5/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/dlist-0.5/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/blaze-builder-0.3.1.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/attoparsec-0.10.3.0/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/text-0.11.2.3/lib","-L/usr/local/lib/ghc-7.6.1/bytestring-0.10.0.0","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/abstract-par-0.3.1/lib","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/abstract-deque-0.1.6/lib","-L/usr/local/lib/ghc-7.6.1/containers-0.5.0.0","-L/Users/tibell/Library/Haskell/ghc-7.6.1/lib/HUnit-1.2.5.1/lib","-L/usr/local/lib/ghc-7.6.1/deepseq-1.3.0.1","-L/usr/local/lib/ghc-7.6.1/array-0.4.0.1","-L/usr/local/lib/ghc-7.6.1/base-4.6.0.0","-L/usr/local/lib/ghc-7.6.1/integer-gmp-0.5.0.0","-L/usr/local/lib/ghc-7.6.1/ghc-prim-0.3.0.0","-L/usr/local/lib/ghc-7.6.1"])

It says There is no available version of criterion that satisfies -any, which is clearly incorrect if we look at the ghc-pkg output in the bug description.

[tibbe]

@kosmikus Do you mind taking a look. You can reproduce this using GHC 7.6.1, and empty user package DB, and the latest HEAD of unordered-containers from https://github.com/tibbe/unordered-containers.

You will have to run:

cabal install --constraint="hashable == 1.1.2.5" --only-dependencies --package-db=$PWD/.cabal-sandbox/ghc-7.6.1-packages.conf.d
cabal install --constraint="hashable == 1.1.2.5" criterion hashmap --package-db=$PWD/.cabal-sandbox/ghc-7.6.1-packages.conf.d

Before trying the above command, to make sure you have the right dependencies. The reason you have to install criterion and hashmap separately, is due to a circular dependency (cabal treats the whole package as one unit, where it ought to consider the benchmark section separately).

[kosmikus]

@tibbe Sorry for not replying earlier. I have apparently configured my mail system such that github notifications ended up in some unimportant folder. I will try to take a look.

[kosmikus]

@tibbe Ok, I know why this is happening, at least: the code in Configure.hs invokes a function called standardInstallPolicy as follows:

          standardInstallPolicy
            installedPkgIndex
            (SourcePackageDb mempty packagePrefs)
            [SpecificSourcePackage localPkg]

The final argument is a list of package ids (in this situation, containing the unordered-containers package) that are deleted from the index in the definition of standardInstallPolicy (which is in Distribution/Client/Dependency.hs):

  . hideInstalledPackagesSpecificBySourcePackageId
      [ packageId pkg | SpecificSourcePackage pkg <- pkgSpecifiers ]

and

hideInstalledPackagesSpecificBySourcePackageId :: [PackageId]
                                                  -> DepResolverParams -> DepResolverParams
hideInstalledPackagesSpecificBySourcePackageId pkgids params =
    --TODO: this should work using exclude constraints instead
    params {
      depResolverInstalledPkgIndex =
        foldl' (flip InstalledPackageIndex.deleteSourcePackageId)
               (depResolverInstalledPkgIndex params) pkgids
    }

The TODO has been left by @dcoutts a long time ago, but I think it is accurate. Deleting stuff from the index is evil. It subsequently makes the modular solver think that the reverse dependencies of the deleted package are broken (including aeson, which depends on unordered-containers), which is true according to the view of the world that that code gets. So the correct fix would be to not modify the index, but instead to add constraints that will make the solver exclude the package.

At the moment, I'm not quite sure how much work that would be, and whether there's a risk that something else breaks once this is done.

[kosmikus]

Thinking about this somewhat more, I do not think that fixing this particular issue would help in any substantial way: while unordered-containers would no longer be considered broken, you'd still get a dependency resolution failure because aeson now depends on another instance of unordered-containers than the one you are about to install; it will thus still be rejected. (BTW, the topdown solver complains about criterion, whereas modular complains about aeson which is a dependency of criterion, but the underlying problem is the same for both solvers).

[ttuegel]

Is this still an issue?

[23Skidoo]

My fix was reverted because it broke containers test suite. Now that @edsko implemented independent goals we should fix #1575 and then reapply it.