External custom setup

[ezyang]

Specification. External Custom setups are an extension to the custom-setup stanza which permit a package to specify an external Haskell executable (ala #3708). The syntax is:

custom-setup
  setup-tool: pkgname:exename >= 0.2 && < 0.3

The intended semantics is that this specifies to build the executable exename from pkgname (under the specified version constraints), and then use this executable as a Setup script to compile the package. If the executable name is omitted, it is assumed that the executable has the same name as the package.

A setup-tool is an executable dependency, and is thus solved in the same manner as other executable dependencies (c.f. c0a4860); its dependencies are independent from the main library, and may even be compiled with an entirely different compiler toolchain than the main library.

A non setup-tool custom-setup is simply an "anonymous" executable which (1) can be built without needing a Setup script, and (2) lives inline in the same package.

To determine the supported API of a setup-tool, we look its direct dependencies and determine what version of the Cabal library is used. It is an ERROR to not depend on the Cabal library. (A future extension might lift this restriction by allowing the setup-tool executable to directly specify what version of the Cabal library it supports, or move to a more flexible command-line driven feature test interface.)

Motivation. The primary motivation is #1493: there is no way to ask Cabal to build a custom Setup stanza using a different compiler than the one that is being used to build the main library. Treating the custom Setup as an executable dependency moves us towards solving this problem: once a setup is a component by itself, we only need to ensure executable dependencies have a compiler chosen independently from the library itself. While it is true the same effect could be had by associating with every package a host compiler as well as a target compiler, there are other benefits to decoupling target/host compilers in this way: in particular, the same solution can be used to handle build-tools (and tool-depends #3708) and compiler plugin dependencies (#2965) which also need to distinguish between target and host compiler. Allowing every item in the install plan to have a distinct compiler, depending on its role in the build, is superbly natural.

There are also two minor bonus effects:

• new-build has per-component builds for non-Custom packages, but the components of a Custom package are currently built sequentially. It would be nice to also be able to atomize these packages into separate components, but a naive implementation would require each component to rebuild the Custom component from scratch. With external custom setups, we simply treat traditional custom setups as inplace, anonymous executables that can be built in a special way.
• Custom executable packages can use all of Cabal's features (including having a Custom setup for themselves!) whereas code in a custom-setup is extremely impoverished. (Though this is not a big deal in practice as you can always just put the Custom Setup code in a library.)

Implementation. Here are work items, with dependencies between them:

• PKGFORMAT: A new field setupTool :: Maybe ExeDependency needs to be added for custom-setup. The relevant data structure is SetupBuildInfo in Cabal/Distribution/Types/SetupBuildInfo.hs. You'll have to define a new ExeDependency type modeled off of Dependency (this type can be reused for tool-depends), recording PackageName, String (executable name) and VersionRange. Don't forget to add parser support (setupBInfoFieldDescrs in Cabal/Distribution/PackageDescription/Parse.hs as well as in Cabal/Distribution/PackageDescription/Parsec/FieldDescr.hs as we presently have two parser codepaths.) For completeness, it's probably a good idea to check if both setupDepends and setupTool are set in Cabal/Distribution/PackageDescription/Check.hs; if they are both set that is an error.

• SETUPWRAPPER: SetupWrapper needs a way to be told, "No, don't do anything interesting, just use this executable (supporting this Cabal interface)". We should add a new field to SetupScriptOptions which specifies when a specific file path of an executable to use as a setup script is known, as well as what version of the Cabal interface is understood. Then getSetupMethod can test if these are known, and if so just directly return them as the method to use (the stored cabal version, ExternalMethod path to setup executable, and the unmodified options).

• DEPSOLVER (depends on PKGFORMAT): We need to teach the dependency solver how to solve for setup-tool dependencies. The code for this will live in cabal-install/Distribution/Solver/Modular/IndexConversion.hs; we actually have most of the pieces in place. Look for "-- build-tools dependencies": this demonstrates how to add executable dependencies to the solver. The key function is convExeDep, which makes a qualified dependency on an executable. Note that you'll actually add the dependency on the setup executable to convSetupBuildInfo. It should be fine to put the dependency under ComponentSetup component; we'll just have to read it out from there later.

• PLANNING (depends on DEPSOLVER): The dependency solver will solve for the executable and pass the solution to the exe_deps0 argument of elaborateSolverToComponents in cabal-install/Distribution/Client/ProjectPlanning.hs; you can get out the setup executable dependency using CD.setupDeps exe_deps0. Now we need to enhance ElaboratedPlanPackage with a few more fields to track external custom setups.

  First, we need to record the unit id of our setup dependency, as well as the path to the executable. This can be done by getting the ElaboratedPlanPackage for our SolverId using elaborateExeSolverId in the same way that exe_deps0 is processed currently. elabOrderDependencies needs to be modified to ensure that we add an ordering dependency on our custom setup.

  Second, we need record the version of the Cabal interface our setup-tool supports. Probably the easiest way to do this is to unconditionally record in ElaboratedConfiguredPackage the version of a Cabal library that you depend on, if you have a direct dependency on Cabal. Then we can just read it off from the ElaboratedConfiguredPackage we get from elaborateExeSolverId (you'll need to refactor it in the same way as elaborateLibSolverId is factored, since it returns a ConfiguredId, not the ElaboratedConfiguredPackage; also, you'll need to peel it out of ElaboratedPlanPackage; there are a number of examples of this, look for PreExisting. If it helps, the pre-existing case should be impossible)

• SETUP_OPTIONS (depends on PLANNING). With all of this information in hand, we are finally ready to hook everything up. In setupHsScriptOptions in cabal-install/Distribution/Client/ProjectPlanning.hs, we feed in our newly recorded external setup dependency path and Cabal version to the returned SetupScriptOptions, so that your modifications to SetupWrapper kick in. Delete the TODO and pat yourself on the back. Write a test in integration-tests.

• PARALLEL_CUSTOM (optional). It should be possible already to enable per-component parallel builds with custom setups by modifying the assignment of eligible in cabal-install/Distribution/Client/ProjectPlanning.hs: replacing the existing code with the commented out code should work. The comment is wrong and I don't think you need PER-COMPONENT INPLACE SETUP to do this. Write a test.

• PER-COMPONENT INPLACE SETUP (optional). The big complication here is that the existing code in ProjectBuilding for building a component (buildInplaceUnpackedPackage and buildAndInstallUnpackedPackage) doesn't work on inplace custom setups, which don't have any setup script to build themselves. Instead, the code for building the custom setup lives in SetupWrapper.

  I think the easiest way to deal with this is to implement a new codepath for building custom setup (in ProjectBuilding; you'll probably test on elabPkgOrComponent to find out if it's a setup component) which just runs SetupWrapper in order to build the setup script, then we bail out and copy that setup script into the store if it's an unpacked package (use the executable path that you allocated for the "setup executable" for planning). One thing to be careful about is that elabSetupDependencies would have to return the COMPONENTS dependencies, if the component is a setup script. Look carefully at setupHsScriptOptions to make sure all the parameters are getting the correct options.

c0a4860 should provide a decent blueprint of the files you will have to touch

[ezyang]

CC @Ericson2314 who expressed interest in implementing this.

[dcoutts]

The downside of making setup stanzas more complicated is that it's not just cabal that is affected. How should distro shell scripts cope with this? The addition of dependencies of setup scripts was deliberately limited precisely to balance the utility of more control with the burden on all other tools.

If we want to add more, we need to evaluate this complexity again and figure out what distro scripts would have to do and if the tradeoff is worth it.

[dcoutts]

While it is true the same effect could be had by associating with every package a host compiler as well as a target compiler, there are other benefits to decoupling target/host compilers in this way: in particular, the same solution can be used to handle build-tools (and tool-depends #3708) and compiler plugin dependencies (#2965) which also need to distinguish between target and host compiler. Allowing every item in the install plan to have a distinct compiler, depending on its role in the build, is superbly natural.

There's no problem with more flexibility about hosts and compilers for each component, and generally better cross-compilation support. The potential problem is just with imposing extra complexity and burdens on other tools that need to be able to compile execute the Setup.hs script.

So I'd encourage us to think about the design and see if we can find a way that allows tools that want to support cross-compilation to do so, but that does not force all distros to support the ability for the Setup script for one package to actually live in another package (or other forms of additional complexity that'll make the lives of distro packaging folks difficult -- remember many of these people have to program these systems in bash so everything is hard).

[ezyang]

The addition of dependencies of setup scripts was deliberately limited precisely to balance the utility of more control with the burden on all other tools.

Arguably, setup-depends is a far more complex feature than setup-tool. If I am a distro, I can easily just package up setup-tool as a plain old executable, and then invoke it in my build scripts for the package that uses it as the Custom setup. But with setup-depends I actually have to run a dependency solver to figure out what versions to use and then actually build the Custom setup. I think in practice only cabal-install knows how to do this.

So I just simply cannot see how setup-tool would be difficult for distros to support. It's just an executable (which they have to package) which gets used during the build process of another package. In the Debian world, it would even be a compile-time only dep, so you wouldn't have to install it to use the end package.

[Ericson2314]

In any event, I think treating setup as an anonymous tool dependency makes sense internally, even if that is not exposed.

[ezyang]

Let me also add: when @Ericson2314 asked me to originally write up this ticket, he was only interested in treating setup stanzas as anonymous tool dependencies in the new-build install plan. But what I realized as I was working out the design was that, external setup dependency on an executable is much easier to implement than anonymous setup executables. They just use the existing code in a very natural way, whereas anonymous setup which needs a whole distinct codepath to be built. That is why the ticket is written this way: external setup dependencies are the easy thing, and anonymous inplace setup is the hard, optional add-on.

[Ericson2314]

Also @dcoutts as someone how contributes odds and ends to nixpkgs, custom-tool would not be hard to support, even with the current infrastructure.

[dcoutts]

Arguably, setup-depends is a far more complex feature than setup-tool. If I am a distro, I can easily just package up setup-tool as a plain old executable, and then invoke it in my build scripts for the package that uses it as the Custom setup. But with setup-depends I actually have to run a dependency solver to figure out what versions to use and then actually build the Custom setup. I think in practice only cabal-install knows how to do this.

I don't think this is really the case. For the most part it didn't change the flow for distro scripts at all. They always had to runghc Setup or compile Setup, and the setup-depends change just added more dependencies for a package. There's no need for distros to start using a solver if they were not doing so before.

This change is quite a bit more. It changes the interface from simply compiling/running the Setup.hs that comes with every package (which is what it has been since the beginning) to running another exe from another package.

And I don't see what the motivation is. We can already share code across different Setup.hs scripts by using libraries and build-depends. If the motivation is cross-compilation then we don't need this extension, we can do that more directly. Indeed surely we wouldn't want to use this just for cross-compilation? For cross-compilation we want to handle Setup.hs systematically for all packages, not make each package start using some extension before it can be cross-compiled.

Now internally it may well make a lot of sense to treat Setup.hs scripts more like independent exes, and track the compiler/platform separately for them. But I don't see why we would want or need to extend the expressiveness of the system for building the Setup.hs script.

[ezyang]

I'm willing to not actually have setup-tool not actually be an exposed feature. But I think the only design that makes sense, for handling Custom setups, is one where adding "setup-tool" would be a trivial addition (just wiring up some syntax appropriate.) I think @Ericson2314 agrees (he mostly cares about inplace Custom working, and setup-tools is a means to the end.)

EDIT: And furthermore, even if we didn't expose it in the end, I would still encourage putting in the syntax for it during development (to be removed later) as an easy mechanism to test setup executable dependencies without having rewritten SetupWrapper to factor out the special case code for building compiler. (It's just simpler!)

[Ericson2314]

Hmm, so I think the lynch-pin for this strategy on internal setups is an elaboration from SetupBuildInfo -> (BuildInfo, ExtraSetupInfo) (or ElaboratedSetupInfo, which contains a BuildInfo. Hopefully the BuildInfo alone is sufficient to describe how Setup.hs is built. This dovetails with https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/SetupWrapper.hs#L142.

[Ericson2314]

I think SolverConfig will need to be given the default version of Cabal to use when none is specified. Every other SetupScriptOptions is either a pure function of the .cabal file, or only matter at ./Setup-run-time.

[ezyang]

Yes, that sounds reasonable. The existing SetupWrapper code is the gospel!

[Ericson2314]

In #1493 I've talked about using multiple compilers/platforms when solving. One thing this elides over is when components from the same package are used in a different phase---e.g. if a package's library uses it's executable, or more common a custom setup. In this case---even with per-component solving---we often want to make sure all components in a package are consistently configured, which means teaching packages notions of a "build" vs "host" platform (in the Autoconf sense), and requiring the package be configured with every component being built for no more than one of the two platforms.

External setup dependencies do side-step this complication when the setup executable is drawn from another package.