Scoped packages (i.e. multiple libraries in a Cabal package)

[ezyang]

Currently, Cabal's internal notion of a package only supports zero or one libraries. This sticks out as a bit of a sore thumb compared to other components in a Cabal file: Cabal currently supports compiling many components per Cabal file (libraries, test suites, executables, benchmarks), but only one of these components can be a library. I propose to remove this restriction, thus making libraries treated uniformly with other components in Cabal. We can then refactor PackageDescription to have a list of components.

Use-cases:

1. For Backpack, we absolutely need the ability for a single Cabal file to result in the installation of multiple "packages" in the installed package database, because these packages are how you do modular development in Backpack. I took a detour to implement this feature, because it will serve as a good blueprint for how to make sure Cabal can handle building/registering multiple libraries.
2. This change presents a really good opportunity to substantially simplify Cabal's handling of components. Currently, benchmarks, testsuites, executables and libraries are all separately special cased in Cabal, and anything that, e.g. mucks about the BuildInfos has to be implemented FOUR times for each of these cases. If we can push the component abstraction into PackageDescription, this will be a great simplification.

This would then directly allow us to support private packages (these would be an extra library that would be installed to the installed package database, making it visible to GHC).

---

For internal packages (which may be depended upon by public packages but cannot be referenced directly), the above proposal suffices. However, you might want to export more than one package from a Cabal file. The big use-case is when you have a package which has been split into a constellation of packages in order to make it easier for users to install useful subsets of functionality without pulling in the rest of the dependencies they don't want. However, maintaining N different Cabal files can be a bit of a pain for tightly coupled packages. With scoped packages, all of these packages could be placed in one Cabal file. (We have to make sure components get depsolved separately, but @edsko has put us most of the way there.)

The big problem with this is that it breaks the invariant that there is a one-to-one mapping between exported libraries and Cabal packages. If a Cabal file can define multiple libraries (which live in the same namespace a package names), then both foo.cabal and bar.cabal could define the helperpackage (collision). Furthermore, from cabal-install's perspective, if we need the helper package, it is not obvious where to look if there is no helper.cabal. There was considerable pushback here, so it may end up being the case that this is never going to be implemented.

However, if we did want to solve this, one way would be to introduce a hierarchy of scoping to package names. For the rest of this bug, I'll assume we delimit scopes with -. (If we don't care about compatibility with older versions of GHC, we could use something less ambiguous). The invariant is that a package named p can only define libraries p and p-anysubname. (We might even further imagine publishing a package named p-anysubname, thus allowing multiple libraries to be split up from a Cabal file). If cabal-install encounters a dependency on p-subpackage, it looks at p.cabal for a description of how to build it (or for p-subpackage.cabal). There is some ambiguity since package names today can use hyphens, but the potential massive conflicts are reduced, and cabal-install only has to query a few Cabal packages to see where a package is defined, as opposed to the ENTIRE package database.

[snoyberg]

I'll quote myself from Reddit

Please, please don't do this. Our tooling is barely holding together as is. Throwing yet another curve ball is yet another failure case we all get to worry about and experience.

[ezyang]

(Updated description with some motivations.)

Reddit conversation: https://www.reddit.com/r/haskell/comments/3cu5nu/feedback_requested_for_supporting_multiple/

[DanBurton]

For Backpack, we absolutely need the ability for a single Cabal file to result in the installation of multiple "packages" in the installed package database, because these packages are how you do modular development in Backpack.

Can you elaborate on why Backpack can't just use one cabal file per package?

There are many packages which have been split it a wide constellation of packages in order to make it easier for users to install useful subsets of functionality without pulling in the rest of the dependencies they don't want. However, maintaining N different Cabal files can be a bit of a pain for tightly coupled packages.

Are there some concrete examples of this?

It would make more sense to me to have backpack sit as a layer on top of cabal, rather than inside of it.

[snoyberg]

I've read the description (and updates) and the explanations on Reddit. I'm still not sure I understand the current version of the proposal. I'll take a stab at it to at least start a conversation:

1. The .cabal file will still only support one library, which will be the same name as the package itself.
2. The Cabal library will have the concept of generating multiple installed packages in the package database based on a single .cabal file
3. Backpack will have some other metadata available somewhere allowing the user to specify "please instantiate some module with these dependencies," which will result in an extra installed package beng generated.

Presumably, cabal-install/stack will need quite a bit of smarts added to understand these new "phantom" packages in the GHC package database, but the impact will be limited to those tools, the Cabal library, and (I guess obviously) whatever changes GHC itself requires to make all of this work.

Am I on the right track with this?

[dcoutts]

@ezyang as you know, in principle I'm supportive of the backpack effort, though as you also know I sometimes struggle with the details :-)

I should point out that related to this is the idea of convenience libraries #269, which also involves the idea of multiple libraries in a single cabal package. The key thing with convenience libs however is that they're completely private to that package, are only used by other components within the package and don't contribute to the global package namespace. So totally agree on the zero / one oddity. Convenience libs would allow zero / one "public" libs and zero or more "private" libs.

The big issue here with backpack is indeed the package naming issue. Having one unit of distribution provide multiple names that can be depended on by other distribution units is a significant thing. The idea of hierarchy is interesting. Are you sure backpack could really work with that? We don't ever need to refer to backpack unit p where that could come from cabal package A or B? It's always going to be possible to say that it's A.p or B.p?

[dcoutts]

@ezyang in your revised description, I'd like to clarify what you mean/want with private libraries. The old idea of "convenience libraries" was like these private libraries except that they would not be registered into the main package db. The idea was we could get away without (and thus without namespace issues).

As far as I can see, for the private lib use case there's still no need to actually register them, though I can see that for the backpack case there is. Have I missed something or is the thing about registering them solely for the backpack use case?

[ezyang]

@dcoutts I just read over the convenience libraries proposal. It seems that:

1. The refactoring I propose will be helpful for convenience libraries, however
2. Backpack can't use convenience libraries as specified (with static linking and no registering in the main database) because the main, exported library expressly wants to refer to the convenience library.

Convenience libraries seems to be all about local use, so I don't think registering globally is useful except for Backpack. But Backpack cares about it a lot.

The big issue here with backpack is indeed the package naming issue. Having one unit of distribution provide multiple names that can be depended on by other distribution units is a significant thing.

I retract any earlier comments about this: Backpack will continue the Cabal model of only exporting a single package. Let's defer any conversation about subpackages of a Cabal file for much later.

@snoyberg

You are on track.

Presumably, cabal-install/stack will need quite a bit of smarts added to understand these new "phantom" packages in the GHC package database, but the impact will be limited to those tools, the Cabal library, and (I guess obviously) whatever changes GHC itself requires to make all of this work.

I think that the recommended strategy for cabal-install/stack is to simply ignore these packages (pretend they don't exist). They're not useful for purposes of dependency resolution. We'll need GHC changes to support this but we need GHC changes for Backpack anyway, so it's not an onerous restriction.

It would make more sense to me to have backpack sit as a layer on top of cabal, rather than inside of it.

That was the first design, but this means that Backpack cannot be used independently of Cabal, which is a steep price to pay.

[ezyang]

Closing this for now.