Package with custom Setup.hs "Encountered missing dependencies"

[eamsden]

New report (edited by @ezyang)

Cabal 1.23 and later #2731 allow you to skip specifying dependencies which are not part of a buildable component. cabal-install was updated to take advantage of this fact.

However, when a package has a Custom setup script, it is possible for the Setup script to be built against an old version of Cabal, which is doesn't know to ignore non-buildable dependencies. In this case, cabal-install will pass an insufficient set of dependencies, resulting in an error like this:

    setup: At least the following dependencies are missing:
    process -any, temporary >=1.1

(where these are dependencies of non-buildable components.)

A workaround is to explicitly request that all components be built. For example, if there is some flag which must be selected to make a component buildable, you should pass --constraint="package-name +flagname"

---

Original bug report:

Trying to build pandoc-citeproc (which has a custom Setup.hs with a couple of hooks) using the latest-packaged version from git in the HVR repository (Version: 1.23+git20160204.0.7aab356~wily) fails to find dependencies already installed in a sandbox (whether the dependencies are installed manually or via the dependency solver).

$ uname -a
Linux <hostname> 4.2.0-30-generic #35-Ubuntu SMP Fri Feb 19 13:52:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3
$ cabal --version
cabal-install version 1.23.0.0
compiled using version 1.23.1.0 of the Cabal library
$ cabal sandbox init
$ cabal install pandoc-citeproc
Configuring pandoc-citeproc-0.9...
setup: At least the following dependencies are missing:
process -any, temporary >=1.1
$ cabal sandbox hc-pkg list process
process-1.2.3.0
$ cabal sandbox hc-pkg list temporary
temporary-1.2.0.4

This was previously filed as jgm/pandoc-citeproc#216

[23Skidoo]

Thanks, will take a look. Probably fallout from setup-depends work.

[grayjay]

I think I see the problem. #2731 (merged in #3039) prevented cabal from requiring the dependencies of components that are not "Buildable". pandoc-citeproc has an executable, test-citeproc, that is enabled with the flag test_citeproc and depends on temporary and process. When cabal-1.23 solved for the dependencies, it chose -test_citeproc and didn't pass --dependency=temporary=... to Setup. Since Setup was built with Cabal < 1.23, Setup still required the dependencies of the non-buildable component and failed.

I don't see an easy way to fix it before the release, besides reverting the cabal-install part of #2731.

[23Skidoo]

@grayjay Thanks for the analysis. I think that the ultimate solution is to ensure that such a situation never happens and just always build setup scripts against the same version of Cabal that cabal-install was built with. The fact that a version mismatch can happen has been a never-ending source of bugs (see e.g. #3138).

For the 1.24 release, maybe we can do something similar to the filterConfigureFlags approach and switch to the old solver behaviour when the Cabal version is too old?

[grayjay]

@23Skidoo I thought custom-setup allows cabal install to choose any versions of the library that are needed to build setup scripts, so we don't know what versions will be used until after solving. The backwards-compatible behavior would also need to be in the solver, since it involves choosing more dependencies. We could do something like first running the solver with a constraint on Cabal, and then solving without the constraint but with backwards-compatible behavior if the first run fails. Or we could decide package by package in the solver, maybe by adding a flag to each package with a custom-setup, but that seems like too much complexity.

Is there any way to make Setup completely ignore dependencies? As far as I can tell, --exact-configuration only allows ignoring the version bounds.

[23Skidoo]

One other thing we can do is pass --constraint instead of --exact-configuration to setup scripts built with Cabal versions that don't support ignoring non-buildable components. This is probably the simplest solution.

@grayjay Yes, setup-depends with an upper bound on Cabal version present a problem. We may want to ban upper version bounds on Cabal, like we do for the cabal-version field (but allow them in post-release updates).

[grayjay]

@23Skidoo How can we use --constraint to ignore the dependencies?

[23Skidoo]

@grayjay Before --exact-configuration was implemented, we used to feed solver output to Setup configure using the --constraint options: --constraint=foo-$VER_FOO --constraint=bar-$VER_BAR [...]. So I think that if we use that old interface, but only pass the constraints for dependencies of buildable components, Setup configure will be able to figure out what to do using its own solver.

[grayjay]

@23Skidoo I just tried configuring the package from the "Buildable" package test, and it didn't work:

flag build-exe
  default: True

library

executable my-executable
  build-depends: base, unavailable-package
  main-is: Main.hs
  if !flag(build-exe)
    buildable: False

$ ./Setup --version
Cabal library version 1.22.4.0

$ ./Setup configure --constraint='base==4.8.1.0' -f-build-exe
Configuring BuildableField-0.1.0.0...
Setup.exe: At least the following dependencies are missing:
unavailable-package -any

I think --constraint only adds to the constraints in the .cabal file.

[23Skidoo]

Well, that's a pity.

[grayjay]

@23Skidoo Should we revert the cabal-install part of #2731 before the release?

[23Skidoo]

@grayjay Maybe we'll have to, but I don't want to do it just yet. Need to think a bit more about this.

[ezyang]

One way to solve this is to add a command to Setup scripts which lets us probe them for what features we support. Then we don't ignore non-buildable components when the Setup script doesn't support it.

[grayjay]

There actually seems to be a problem with inconsistent dependency requirements in the other direction, too. One of the integration tests failed with an invalid install plan when I tried reverting the cabal-install change.

Commit: grayjay@a6dba5c

Test suite integration-tests: RUNNING...
Integration Tests
  freeze
    should_run
      enable_benchmarks_freezes_bench_deps.sh (ignoring stdout+stderr):   FAIL (1.73s)
        Unexpected exit status.

        Exit status: ExitFailure 1
        Working directory: /tmp/cabal-install-test-22215/should_run

        <stdout> was:
        Config file path source is commandline option.
        Config file config-file not found.
        Writing default configuration to config-file
        Resolving dependencies...


        <stderr> was:
        internal error: could not construct a valid install plan.
        The proposed (invalid) plan contained the following problems:
        Package vector-algorithms-0.7.0.1 has an invalid configuration, in particular:
          the package configuration specifies mtl-2.2.1 but (with the given flag assignment) the package does not actually depend on any version of that package.
          the package configuration specifies mwc-random-0.13.4.0 but (with the given flag assignment) the package does not actually depend on any version of that package.

        Proposed plan:
        Configured Glob-0.7.5 (.fake.Glob-0.7.5)
        Configured abstract-deque-0.3 (.fake.abstract-deque-0.3)
        Configured abstract-par-0.3.3 (.fake.abstract-par-0.3.3)
        Configured aeson-0.11.1.0 (.fake.aeson-0.11.1.0)
        [...]

[grayjay]

@ezyang We would have to know the versions of the setup scripts during dependency solving in order to choose dependencies for the non-buildable components (See #3199 (comment)).

[23Skidoo]

@grayjay

We would have to know the versions of the setup scripts during dependency solving

We should probably use addDefaultSetupDependencies and restrict the Cabal version to < 1.25 for packages with build-type: Custom that don't have a custom-setup section anyway. Also, in setup wrapper code we do know the Cabal version.

[grayjay]

@23Skidoo I'm not sure I understand. Would this work? Before solving for dependencies, we decide whether each package with build-type: Custom should use Cabal < 1.24 or Cabal >= 1.24 for its setup script. Then we add that constraint to the package's setup-depends, and, if the constraint is >=1.24, transform the package so that it ignores dependencies of non-buildable components. Then we run the solver on the modified packages. I think that the setup wrapper code could just pass the dependencies from the install plan directly to the setup script. We would also need to update the install plan validation. I was hoping this would be easier to fix. Unfortunately, I won't have time to work on anything this involved before the release.

[grayjay]

@23Skidoo Do you know how we should handle this issue? I might have time to work on it now.

[ezyang]

Could we make a point release of Cabal 1.22 with the bug fixed?

[23Skidoo]

This only affects 1.24+.

[grayjay]

Would the point release contain #2731, possibly with a flag to control the behavior, so that cabal-1.24 cabal-install-1.24 can work with Cabal-1.22?

[23Skidoo]

@grayjay I think that a simple solution would be to require that the setup script is compiled against Cabal 1.24 when there are non-buildable components. In fact, we can do this for all setup scripts when there is no custom-setup section.

So I propose using addDefaultSetupDependencies to add a default set of setup deps when a custom-setup section is lacking. Dependency on Cabal should be set to 1.24 <= && < 1.25, the rest should probably consist of libs that come with GHC by default (without version bounds). You can use my all-custom-setups repo for testing: scripts that compiled with GHC 7.10 and Cabal 1.22 should continue to do so.

[grayjay]

@23Skidoo Thanks for the advice; I did some testing like you suggested

I wanted to minimize the number of packages affected by a change this close to the release, so I first tried to filter out packages that aren't affected by the bug.

All counts include package and version:
total packages on Hackage: 66347
custom setup scripts: 4491
custom setup script and at least one occurrence of Buildable: False: 860
high likelihood that a component will have dependencies when it is not "Buildable": 328

So we could add default setup dependencies in a way that only affects ~328 packages.

Then I tested the effects of calling addDefaultSetupDependencies by compiling the Setup scripts of those 328 packages using GHC 7.10.3, packages included with GHC, and a sandbox containing the latest Cabal-1.24.0.0. I also compiled with the global Cabal-1.22.5.0 by using an empty sandbox.

	Cabal-1.22.5.0	Cabal-1.24.0.0
Failed to compile:	142	154
Failed due to a missing import:	128	128

The only package that compiled with 1.22 but not 1.24 was hpqtypes (all 12 versions). Here are the missing imports. Since I didn't download full packages, some of these modules could be included in the package, like WebSetup and GetText.

Control.OldException
Data.Default
Distribution.ShellHarness
Distribution.Simple.I18N.GetText
GetText
GHC.Paths
Language.Preprocessor.Cpphs
SetupCompat
SetupWrapper
System
System.Locale
System.Time
Text.Regex
WebSetup

It seems like the biggest problem with calling addDefaultSetupDependencies is that it will prevent setup scripts from using other dependencies, even if they are installed. Is there a way to force a setup script to be built with Cabal >= 1.24.0.0 but allow it to use other installed packages?

[23Skidoo]

@grayjay Thanks for looking into this!

The only package that compiled with 1.22 but not 1.24 was hpqtypes (all 12 versions).

What was the error?

Here are the missing imports. Since I didn't download full packages, some of these modules could be included in the package, like WebSetup and GetText.

System.Locale
System.Time
Text.Regex

OK, so we need to include old-locale, old-time and regex-posix in the default set. They are a part of the HP, so I'm okay with this. Though perhaps it's better to special-case packages that depend on regex-posix, since its somewhat large.

Control.OldException

This module was removed from base quite a while ago, so the setup script would fail to compile anyway.

Data.Default
GHC.Paths

I don't think we should add data-default and ghc-paths to the default set, they are not even part of HP. Perhaps we can special-case these packages?

GetText
Distribution.ShellHarness
Distribution.Simple.I18N.GetText
Language.Preprocessor.Cpphs
SetupCompat
SetupWrapper
System
WebSetup

These look like internal dependencies.

Is there a way to force a setup script to be built with Cabal >= 1.24.0.0 but allow it to use other installed packages?

In principle yes, we could invoke ghc with -hide-package Cabal -package-id Cabal-1.24-foo (that is, without -hide-all-packages). Maybe it's the best solution.

[grayjay]

What was the error?

[1 of 1] Compiling Main             ( Setup.hs, Setup.o )

Setup.hs:35:39:
    No instance for (ConstOrId
                       (IO (Maybe FilePath))
                       (ProgramSearchPath -> IO (Maybe (FilePath, [FilePath]))))
      (maybe you haven't applied enough arguments to a function?)
      arising from a use of ‘constOrId’
    In the expression: constOrId
    In the expression:
      constOrId
      $ do { pgconfig <- findProgramLocation verbosity "pgconfig";
             pg_config <- findProgramLocation verbosity "pg_config";
             return $ pgconfig `mplus` pg_config }
    In the ‘programFindLocation’ field of a record

Is there a way to force a setup script to be built with Cabal >= 1.24.0.0 but allow it to use other installed packages?

In principle yes, we could invoke ghc with -hide-package Cabal -package-id Cabal-1.24-foo (that is, without -hide-all-packages). Maybe it's the best solution.

I tried this approach but got stuck. I was able to enforce the constraint on Cabal by setting useCabalVersion to >= 1.24 in the SetupScriptOptions used by setupWrapper. However, setupWrapper doesn't have enough information to determine whether #3199 affects the current package. The function that determines whether a package is affected takes a GenericPackageDescription, and setupWrapper only has a PackageDescription that has lost the relevant "buildable" information.

I also tried adding a dependency on Cabal >= 1.24 usingaddDefaultSetupDependencies, to ensure that the right version of Cabal is installed. That was relatively easy. But then I didn't know how to remove the default setup dependencies in setupWrapper to allow Setup to be built with Cabal >= 1.24 and other packages.

After looking into it, I realized that this change is much more complicated than I first thought, and I'm not very familiar with the code. I don't think I'll be able to finish it in time. I posted the function that I used to check whether this issue applies to a package, if someone wants to pick it up: https://gist.github.com/grayjay/c5c2627d02303bd59a3b164055233bcb

[grayjay]

@ezyang Thanks for looking into the other error.