Cabal-install as a library

[wdanilo]

Hello! I would love to use Distribution.Client.* tree as a library - it has got a lot of useful functions, we can use to create custom behaviours. Right now it is used only by the cabal executable, but I do no think it will take much time to separate it as a library and create a simple Main.hs using this library to produce cabal executable.

I've asked about it on Haskell's IRC and I'm not the only person who finds this idea to be usefull.

[chrisdone]

I've wanted this in the past. I can only assume the reason it has been exposed as a library is to avoid caring about changing the APIs. But they seem pretty stable.

[23Skidoo]

I've actually enabled this previously, but @dcoutts reverted that patch. See 7597a17.

[wdanilo]

@23Skidoo: I've created a patch right now and wanted to create pull request ...
So I understand that when I want to use functions from cabal-install I've got to manually copy the code to my sources? I completely do not understand such decision.

I understand @dcoutts, what you are talking about (7597a17):
"At some point in the future we probably do want a library or two
but we should make that decision deliberately and think about the API
that we want to expose."

But exposing all the unctions is less evil than not exposing the library at all, at least in my opinion.

[wdanilo]

Btw - can I ask why allmost no datatypes are derriving Show in cabal-install? It makes very hard to learn whats going under the hood without the ability to print something to terminal. I've added lots of "deriving (Show) for my own purposes and its a lot easier now.

[dcoutts]

So I'm a bit concerned about having the same package be both a lib and an exe. I fear that will create problems with mixing ghc versions etc. At the moment with cabal-install having no installed footprint other than the exe itself, it's easy to mix ghc versions and cabal binary versions.

However, having much of the cabal-install functionality available as one or more libraries is certainly a good plan. I just think we ought to make them as separate libs, and in a deliberate way, not just declare the package to be both a lib and an exe.

We could do this at the same time as reorganising the Cabal lib. e.g.:

• Cabal (Distribution.*) -> cabal-lib
• Cabal (Distribution.Simple.*) -> cabal-build (or cabal-build-simple)
• cabal-install -> cabal
• cabal-install (various bits thereof, e.g. package solver etc, hackage-client api) -> (one or more other libs)

[dcoutts]

As an example, the hackage doc builder client could do with getting access to some of the cabal-install functionality. So I can certainly see the utility of it.

[ttuegel]

Two thoughts:

1. Some of the D.Client modules extend equivalent D.Simple modules. For example, cabal configure takes more options than setup configure. We may want to move some things back into D.Simple. One thing that's difficult now, for example, is that cabal build sometimes runs into trouble reconfiguring a changed package because it doesn't have access to the complete command line that was sent to cabal configure.
2. Cabal and cabal-install dump all their messages into stdout/stderr and die on certain errors. Libraries must not do this! We already have code in cabal-install to work around the fact that the Cabal library we're using might call die, but we don't want to actually exit. We need to handle errors (all kinds of messages, actually) like a library.

[fujimura]

+1 for reorganising Cabal lib. Small and well-organised library is easier to grasp, so it would be really helpful for new programmer who wants to use Cabal as a lib.

[BardurArantsson]

-1000. This is a terrible idea unless there is some specific design which considers the API very thoroughly[1] and ensured that no process-wide state (or termination and such) can possibly leak. Cabal-install is designed as and must remain an-executable-only, for reasons mentioned by @dcoutts and @ttuegel.

[1] ... and even so, it might unduly constrain what are essentially internal APIs and needlessly slow down or outright prevent global refactoring.

In addition there are just too many way cabal-install potentially affects the current process state, e.g. signals, std I/O buffering, current working dir (not currently, but potentially), etc.

We should just work to make cabal-the-executable more flexible so that you don't have to write something which uses it as a library.

[BardurArantsson]

Just to clarify: I'd consider this a WONTFIX and close it with prejudice. But maybe that's just me.

[bgamari]

@BardurArantsson I think we need to be a bit more nuanced here. There are portions of functionality currently implemented in cabal-install which can be cleanly split out and the community as a whole can easily benefit from. Take, for instance, the solver.

I agree that just adding a library section to cabal-install.cabal is probably not the right way to go, but with little effort we can end up with something much more extensible than we have now. As someone who has worked on Haskell tooling in the past, my opinion is that cabal-install currently impedes tool development where it should be helping it.

[BardurArantsson]

I agree that e.g. the solver should perhaps be split out, but then perhaps there should be an issue for separating the solver out? (Perhaps there already is, haven't looked.)

I detest abstract issues like "Make this a library" because they're not really actionable unless there's a concrete plan. It's fine to think about these things and come up with plans, but IMO this kind of discussion should probably take place on a mailing list[1] where enough people will actually have a chance to see it. I certainly know that I would never ever have discovered this particular issue (and subsequent discussion) unless I'd happened to be trawling through the issue track since I've starting developing a bit for Cabal. I was, however, lurking on the development mailing list for a long time before that (several years, I think).

[1] Since that's what this particular project seems to be using for general development discussion.

[bgamari]

@BardurArantsson indeed there is, #2768.

Fair enough, I agree that this sort of discussion of general direction should probably occur in a more visible forum.

[quasicomputational]

Another significant use-case: reading and writing .project files. Reading freeze files is a significant thing that programs are going to want to do, and there are also cases for generating .project files. Currently, the best way to get at a build plan is to grub around with plan.json, even if you've got a freeze file sitting there that has all the information anyway, which feels needlessly fragile and silly.

[hvr]

minor nitpick: no, the .freeze does not yet uniquely determine the build-plan in the presence of qualified goals (there's a ticket about which describes how to make the constraint-syntax expressive enough to achieve that). So in fact, the plan.json file is currently the most accurate description of the build-plan we have, and the plan.json has also been intended to serve as date interchange format to communicate the build-plan to other tooling.

[gbaz]

Arguably, rather than an umbrella thing, we could granularly consider which individual components of cabal-install could be migrated to Cabal-the-library on a case by case basis.

[BardurArantsson]

It's completely unrealistic to think that anything like this will happen. There's way too much inertia in the code base and in the project. (Being hamstrung by a very long backward-compat 'guarantee' for Cabal doesn't help either. For one thing it makes it very "dangerous" to move things from cabal-install to Cabal because then the project it stuck with those possibly-bad decisions, effectively forever.)

I gave it a good shot by splitting out the solver[1], but actually creating a separate "intermediate" solver package 'between' Cabal and cabal-install stalled on much hemming and hawing over (of all things) the minutiae of version numbers... It would actually have been very little extra work and would trivially have prevented the back-sliding mentioned in [1]. AFAICT this happens with almost every major change anybody who's not already "an insider" tries to do... and that wasn't even a major externally visible change. (I'm not particularly bitter though it may sound like it. Just disappointed.)

The only way forward to avoid complete ossification is disruption, I'm afraid. I have no idea where that disruption would come from, but certainly Stack has provided some of that (for better or worse, I haven't ever even used it knowingly). I also get the gut feeling that a lot of the inertia in the project is due to fear of disruption or of being rendered irrelevant.

[1] It was split into a separate module with no dependencies back to cabal-install. Of course those dependencies then crept back because there was no mechanism in place to prevent that such as keeping it in a separate package.

[23Skidoo]

I think that with the more frequent release schedule forced on us by GHC HQ (which I consider a good thing for the project) this is now more realistic. If we can't backport a patch because it introduces a breaking API change, it's less of a problem if major releases are frequent.

[phadej]

I'm against growing the API surface of Cabal-the-library. It's already too big. Splitting parts of cabal-install into auxiliary libs is OTOH not a bad idea. (Case example: hackage-security, though it has been developed as a separate lib from the beginning)

…

Sent from my iPhone

On 31 Jul 2018, at 11.57, Mikhail Glushenkov ***@***.***> wrote: I think that with the more frequent release schedule forced on us by GHC HQ (which I consider a good thing for the project) this is now more realistic. If we can't backport a patch because it introduces a breaking API change, it's less of a problem if major releases are frequent. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[hvr]

@BardurArantsson

It was split into a separate module with no dependencies back to cabal-install. Of course those dependencies then crept back because there was no mechanism in place to prevent that such as keeping it in a separate package.

But we do have that mechanism! We just couldn't use it yet as it was too new. But soon we can!

[BardurArantsson]

@hvr Which mechansim are you talking about? Multiple private libraries in a package?