Add config flag to control generation of .ghc.environment files

[hvr]

Some people are excited about .ghc.environment files, while others are mildly annoyed. I think there's probable cause to make this configurable on cabal's end.

To this end, I suggest (modulo bikeshed) to implement a flag, e.g. --pkg-environment-scope=LEVEL, where LEVEL can be one of

1. all (default & current behaviour, generate ghc env files with all transitive dependencies of project's (non-qualified) goals)
2. build-target (e.g. :pkg:Cabal or lib:Cabal or Cabal:test:parser-tests): generate ghc env files containing only the stated goal's /direct/ dependencies)
3. - (or none or off or disable?): disable generation of any ghc env files

This flag would also be persistable via cabal.project.(local).

Note: Item 2. is not the short-term goal of this feature request! I've included the 2. item mostly to motivate why it makes sense to design this to be a flag that's more than merely a boolean --{disable,enable}-package-environment-files flag.

Btw, a dual feature-request (allowing to opt-out from interpreting .ghc.environment files from GHC's side) has been filed at GHC #13753 -- however, that one will be too late for GHC 8.0.2 users.

Current status:

• Starting with GHC 8.4.3, ghc is more verbose when picking up pkg envs (see http://git.haskell.org/ghc.git/commit/00049e2dce93b1e468c3fde3287371eb988aafdc)
• Starting with GHC 8.4.4, ghc supports the special - environment which inhibits the default lookup logic (see http://git.haskell.org/ghc.git/commit/8f3c149d94814e4f278b08c562f06fc257eb3c43)

/cc @RyanGlScott

[ezyang]

So, I've recently been working in the Python/Anaconda world, where they have scripts like source activate myenv which kick you into an alternate shell where when you call python, you get the environment (interpreter, packages, etc) correspond to the particular environment you are working on. Letting this setting be done on a per shell basis seems better to me than setting it to file path location? (Yes, I know this is totally asking GHC environments to be redone, but it seems to me like it would solve the root cause of the problem you're describing above, as well as other problems.)

[hvr]

Tbh, I have been working with Python's virtualenvs for quite some time, and they were somewhat unsatisfying to me. I prefer the git-style approach which has been copied by many tools which are $CWD sensitive. I don't want to have to remember and having to explicitly perform redundant keystrokes to enter an environment other than simply cding into a project, and have my tool follow the DWIM principle. If we require to explicitly enter environments, we still have cabal exec or cabal repl; but that's a different paradigm (which has its uses) from what GHC environment files provide. Since GHC environments started working in cabal head, I've reduced my use of cabal new-repl to quite a lot less.

This ticket is primarily to help those who don't subscribe to that paradigm, and want to opt-out.

[alanz]

I just spent a significant chunk of time trying to get a project to build and run tests, using stack. I build the project using both cabal master and stack, to check that it works for both.

Having deleted the dist* directories and built with stack, the tests failed, complaining about a missing package db in ./dist-newstyle.

After re-installing everything haskell related, I eventually tracked it down to this file being silently generated, and then being picked up by other tools.

At this stage I would prefer to opt in for this generation, or at least get something in the output to say that the file has been generated, so that I can know that a potential landmine has been planted.

This is bound to trip up others as well, as the existence and use of this file is not well documented.

[ezyang]

I agree that this state of affairs is not great. @hvr, can we perhaps change this setting to be opt-in?

[23Skidoo]

Can we get stack to turn off parsing of .ghc.environment files with -hide-all-packages?

/cc @mgsloan

[cocreature]

fwiw nix also breaks in some cases due to .ghc.environment.* files so I don’t think that a workaround in stack is the right solution here.

[gbaz]

As per https://www.reddit.com/r/haskell/comments/8iyvoo/psa_for_cabal_22_new_users_regarding/ I think that the weight of opinion is that these files should not be auto-generated, but only by opt-in.

If we add this flag as a global flag that can be in ~/.cabal/config and off by default then that would help. Then if people wanted such a config, they could run cabal new-configure --package-environment-scope=ALL.

I think that would improve the situation for a whole bunch of workflows...

[hvr]

@gbaz quite honestly, I'm strongly against defaulting it off, as then nobody would even use what I consider one of the biggest improvements over the old flawed user pkg-db model, since almost nobody would know about it ;-(

Before changing anything, I'd rather want to know exactly what workflows we're talking about that people are having problems with, as I suspect the issues are of a different nature...

Fwiw, there's already patches up at

• https://phabricator.haskell.org/D4689
• https://phabricator.haskell.org/D4690

to address some of the concerns with the UI

[gbaz]

Either way, we should have the flag so it can be changed. Lots of people don't want it on, and at the minimum they should have the option. We can sort out where the actual weight of opinion lies regarding defaults in a broader discussion.

[gbaz]

To expand: my personal belief is that it is currently unexpected behavior for any new-* command to produce any changes to anything outside of dist-newstyle and the store (except for install, which obviously installs), which is what this does. So it violates least-surprise to run e.g. new-test and then suddenly find your invocation of ghci doesn't do what you expected.

[RyanGlScott]

Moreover, this affects more than just ghci. I often invoke runghc ../haskell-ci/make_travis_2.hs from within a project to generate a .travis.yml file, but have it break due to some .ghc.environment file polluting my package database with different packages than the ones that haskell-ci expects. I end up having to explicitly navigate to another directory to avoid .ghc.environment files from breaking the script.

Ironically, despite the stated goal of having GHC commands "just work" regardless of the current working directory, in practice they have the exact opposite effect. I have to be hyper-aware of which directory I'm in before I invoke runghc now, since picking the wrong directory can ruin any Haskell script I may wish to run.

[gbaz]

The opt-out flag in ghc implemented in one of the PRs linked above solves that use case. The main concern with only having that that I would have is if there's a command invoked that invokes ghc for you, and thus where it is sort of a pain to thread that flag to.

[hvr]

@gbaz First off, I consider the current behaviour the intuitive one; if I'm in a project, I expect all invocations to be operating in that context and not some other random pkg environment which has nothing to do with the project I'm in; so I'm definitely not confused at all that ghci throws you in the current CWD -- I'd rather be annoyed if this wasn't the case, and always had to remember doing some magical invocation in order to place myself into that context; and fwiw, that's also how other tools like Git operate; For tools intended to be used by humans on a shell interactively, this is a very common and established UI idiom.

Otoh, programs which invoke ghc for you will have to stop making any assumption about the implicit package environment; same goes for programs which invoke git for you. And I haven't seen anybody making any drama out of tools like Git behaving this way.

Ryan brings up the issue of runghc; Personally I've moved to using runghc in the same style I use ghci and ghc -- expecting it to be CWD sensitive. runghc-based scripting is unfortunately a weakly defined interface which operates in the old paradigm of having a stateful global+user package db in scope, which was rather fragile to use (and which new-build was designed to get rid of). It had a lot of issues already with old-build (it's integration with cabal sandboxes was also extremly sub-optimal; and which package environments elegantly adress), and for new-build we may want to redesign how scripting is done, c.f. #3843 and/or find other ways to integrate runghc with profiles. And as for haskell-ci; that script is about to drop its original goal of being a project-less single-module runghc-friendly script, as its becoming too cumbersome to maintain such a huge monolithic module that way (besides that shebang incompatiblity on some platforms...) and the need arises to start using libraries which are not GHC boot libs. So for that you'll have to switch instead to something like having a wrapper script make-travis-yml in your $PATH like I have e.g.:

#!/bin/bash
exec cabal --project-file=${HOME}/Haskell/projs/haskell-ci/cabal.project new-run exe:$(basename "$0") -- "$@"

[ElvishJerricco]

@hvr I've personally experienced many cases of users coming to me asking why their project doesn't build, only to find its because of these files that they didn't notice because they begin with .. This even broke brittany for me, a code formatter that doesn't even invoke GHC as a process or require any packaging information, which took me a nontrivial amount of time to figure out. I think people's expectation of GHC is that changes to its behavior are not to be made implicitly. This is the expectation for virtually all compilers, i.e. C compilers require every bit of configuration to be done on the command line. Every time I want to deal with package databases, I expect cabal to be involved.

Furthermore, this isn't even a particularly effective solution to the problem it's supposedly fixing, because these files can and will go stale. You have to run a cabal command anyway to update it, and who would choose to have to keep track of that in their head?

It's safe to say that in my experience, this is extremely unintuitive, and that seems to be the consensus in this thread. Having this be opt-out would still leave me with virtually just as many confused newcomers asking for help.

[hvr]

require every bit of configuration to be done on the command line
...
Every time I want to deal with package databases, I expect cabal to be involved.

I honestly doubt you even understand how utterly unusable ghc would be on the shell if we did that (and also how much GHC already doesn't meet your unrealistic ideal), and how frustratingly annoying new-build would be for those that want to use ghc/ghci w/o having to remember to setup env-vars, passing tedious cli flags, or invoking cabal. This would be even worse than the inconsistent situation we had before new-build. We're still in phase of paradigm shift, moving away from the old-build paradigm, into the new-build paradigm. This requires some new approaches, and we're still figuring out how to optimise the UI. But you're effectively suggesting throwing the baby out with the bathwater before it even had a chance, rather than trying to improve the UI to provide a next-gen seamlessly integrated and convenient shell experience for ghc. I'm dogfooding this feature as much as I can, and I'm quite happy with it; and now the time has come that we need more constructive feedback from more users. Therefore I'm more interested to hear how we can explore and improve this new paradigm, while new-build is still prefixed by new-* and can more easily justify radical UI changes.

[ElvishJerricco]

I don't think it's considered a high priority among most people that they be able to invoke ghci rather than cabal new-repl. It doesn't seem like a particularly important ability to me.

Obviously GHC isn't only configurable on the CLI. I understand that it wouldn't be realistic to uphold this standard 100% (in fact it's quite annoying that you can't configure more of the C compiler outside the CLI). But I think it's an important principle to keep in mind, and this seems like a particularly egregious deviation.

All in all, I'm willing to accept that others do consider this valuable, so you'll get no uproar from me if this goes unchanged. But it would be my preference that cabal didn't generate these files implicitly.

If we don't generate them implicitly, perhaps it could be a cabal subcommand to generate them? Users very often look at help text, so having a subcommand makes them very likely to be aware of the existence of this feature.

[RyanGlScott]

To be fair, there is an important semantic difference between invoking cabal new-repl and invoking ghci in the presence of a .ghc.environment file. The former will always rebuild the entirety of the current project in interpreted mode, whereas the latter will simply pop you into a shell with the project already built.

I want to make it clear that I find both features extremely useful, and I'm not advocating that either one of these be removed. On the contrary, I would like to be more explicit about which ones I'm using at a given time. Currently, the latter is quite easy to use by accident, which can have unintended consequences. Luckily, some progress is being made towards making this less likely, which is nice.

As I've stated (and others have stated) before, my preference would be to go from opt-out to opt-in. But that suggestion seems to have fallen on deaf ears, so I'm not going to pursue this much further.

I too would appreciate a dedicated subcommand that simply generates a .ghc.environment file and nothing else. AFAICT, the only way to do so at the moment is through a side effect of a different cabal new- command, which seems suboptimal to me.

[hvr]

appreciate a dedicated subcommand that simply generates a .ghc.environment file and nothing else

In order for a project's generated .ghc.env file to be consistent, you need to have performed a cabal new-build operation; it makes no sense to generate a .ghc.env file w/o having constructed a build-plan and having populated the referenced libraries into the nix-style stores; furthermore, in order to keep it consistent, we need to hook into every action which may result in the build-plan being changed, in order to have it regenerated. So having merely a one-off way to generate these environments for a project would be very bad UI-wise, as you'd run very easily into inconsistent states; once you go package environments, you need to keep doing it.

[RyanGlScott]

OK. In that case, it would be nice if the cabal new-build --help output mentioned that it does this (or better yet, provided a flag which toggles the generation of .ghc.environment files on/off). Currently, this information is not very discoverable (except by accident when things break).

[ElvishJerricco]

How do we reach consensus from here? It seems like we've only got one testimonial in favor of opt-out, vs several in favor of opt-in.

[lspitzner]

@hvr

I honestly doubt you even understand how utterly unusable ghc would be on the shell if we did that, and how frustratingly annoying new-build would be for those that want to use ghc/ghci w/o having to remember to setup env-vars or invoking cabal. This would be even worse than the situation we had before new-build.

Indeed it is clearly possible that I too don't understand this. And I may misunderstand what either of you meant by "configurability on the commandline/of the CLI". Could we just spell things out? What input does GHC(i)'s behaviour depend on?

From the top of my head I can think of

1. commandline args,
2. environment bindings such as PATH, GHC_PACKAGE_PATH, HOME
3. INSTALLPATH/lib/ghc-x.x.x/settings
4. the installation ("global") package db
5. .ghci files in cwd and in $HOME
6. .ghc.environment.* files

Is this list complete? Does a complete version exist (in the ghc user-guide)? I just searched for "ghc.environment" (and also just "environment") and could not find anything, yet this seems highly relevant for general reproducability.

---

But even if the environment files are just an addition to some long list, it appears to me like most other inputs are either

• explicit (1)
• very common and expectable (2, excluding perhaps GHC_PACKAGE_PATH)
• mostly constant/an implementation detail (3, 4)
• explicitly created by the user (5)

and in general i'd say 1-5 are all either are explicit or change only when the user (knowingly) fiddles with stuff. An implicit, frequently changing, new input is rather different imo.

---

@ElvishJerricco

Furthermore, this isn't even a particularly effective solution to the problem it's supposedly fixing, because these files can and will go stale. You have to run a cabal command anyway to update it, and who would choose to have to keep track of that in their head?

@RyanGlScott

To be fair, there is an important semantic difference between invoking cabal new-repl and invoking ghci in the presence of a .ghc.environment file. The former will always rebuild the entirety of the current project in interpreted mode, whereas the latter will simply pop you into a shell with the project already built.

Granted both ghci behaviours are useful, this is hardly a reason to not just have two versions of cabal (new-)repl (or a flag or whatever). This path would avoid implicits, could properly handle the staleness case, at the "cost" of requiring the user to invoke cabal for a task that in my view seems to fall under cabal responsibility, not GHC. So I don't see the downside here, apart from the annoying aspect of having to re-think the UI around this.

(GHC is not supposed to know about projects in any way, and I would not expect it to be "clever" in this direction. cabal also (indirectly) invokes gcc, but I would never expect to be able to call gcc and automagically get any haskell-specific behaviour.)

[gbaz]

How do we reach consensus from here? It seems like we've only got one testimonial in favor of opt-out, vs several in favor of opt-in.

This discussion is very clarifying about the different attitudes involved. I think the basic question is not about cabal per-se but rather how ghc should interact with in-path "config" files of various sorts (currently mainly environment files, but also ghci files, and in the future potentially others). Some people expect ghc to behave the same no matter what directory it is run in, and some people expect it to operate in a context-sensitive fashion.

Ideally we could have a solution that satisfies everybody, and ultimately I think that solution is about ghc primarily, and only secondarily about cabal. It is really just cabal auto-generating these env files that highlights the different expectations about ghc user experience.

My suggestion would be a plan and discussion regarding ghc, perhaps ultimately resulting in a ghc proposal, to clarify the principles of how the compiler interacts with various configuration files. Which is to say, if people who don't like ghc picking up env files could configure ghc to not do that, then the question of cabal generating them or not would not arise.

I have a draft proposal about dot-ghc-files (https://github.com/gbaz/ghc-proposals/blob/patch-2/proposals/0000-dot-ghc-file.rst) that I did not yet submit, in part because I hadn't worked out the interactions with or relationship with env files. I can try to work up different version that takes this discussion into account, and we could proceed from there...

[23Skidoo]

I had to release cabal-install-2.4.0.0 without this bug fixed to unblock people who want to use GHC 8.6.1, which will be out today or tomorrow.

I'll try to find time to write a patch and cut another release ASAP (hopefully before the end of the week).

[joneshf]

Since it looks like we're not getting a flag any time soon, can we get some help on the proper way to work with these files?

Here's my current situation that's breaking:

1. I use nix to bring in cabal-install and shake
2. I setup a Shakefile.hs with one rule, build, to run cabal new-build all:

   import Development.Shake (cmd_, phony, shake)
   
   main :: IO ()
   main = shake (phony "build" (cmd_ "cabal new-build all"))

3. I run shake build
4. It works once, but creates an environment file
5. I run shake build again
6. shake won't run because it picks up the environment file, but the environment file didn't list shake as a package, and it cannot find Development.Shake.

What am I doing wrong, and how am I supposed to make this work with new-build?

[ElvishJerricco]

Sounds like Shake needs to be fixed to ignore package environments when it invokes GHC, just like the various other tools that have had to make changes for this.

Personally, I see that as very frustrating. The author of every tool that invokes GHC has to be aware of the fact that they have to add the same magic lines as everyone else in order for their tool not to break with new-build. I really don't think GHC should pick these up automagically.

[phadej]

OTOH tools like liquid greatly benefit from GHC (as the lib) automatically picking up the environment.

[svenpanne]

I don't want to repeat the all the lengthy stuff I and other people have already written about the reasons why forcing tons of tools to opt-out is a bad idea compared to a structured way to opt-in (the pipenv runway), just this: The recent comments in this issue seem to enforce this point of view. :-/ I really fail to understand why people generally think that explicit is better than implict, but take the very opposite view in the case at hand...

[lspitzner]

@phadej So my time was wasted for the benefit of liquid users? Where exactly can I send my invoice for the time invested to fix my workflow?

[lspitzner]

@phadej That you keep repeating the "but it is useful" argument is just pissing me of. It is highly disrespectful to the users affected negatively. It is highly frustrating that the maintainership makes such a one-sided, private, authoritarian decision on whether breaking existing workflows is justified by convenience for certain users.

[athas]

There is no reason to become so hostile. I, too, think that the implicit loading of environment files is a misfeature and would prefer if I had to use some explicit cabal exec mechanism to pick them up. However, I do not know whether that is the majority opinion. It seems so from this thread, but the people who are happy with the current behaviour may simply not participate.

Ultimately, however, the maintainers of GHC and Cabal get to make the decision (as they should), and since they seem to prefer the implicit behaviour, we should probably just accept it, and start fixing our tools. In particular, unless someone is actually motivated enough to write code to change the current behaviour, then it will eventually just become a fait accompli due to inertia.

[ElvishJerricco]

I dunno. I'm not a fan of the idea that GHC and Cabal should service the GHC and Cabal developers before GHC and Cabal users.

[lspitzner]

There is no reason to become so hostile.

What are you referring to? Please, if you make those kind of accusations, be explicit in what and who you address.

[gbaz]

@joneshf

I think that adding a second rule to shake that deletes the env file after the build is complete might work?

I like many am waiting for the new cabal-install release including this flag, and hope it comes soon, to coincide with the pending ghc 8.6.2 bugfix release.

[joneshf]

@gbaz Thanks! I'd tried that earlier with runAfter, but it didn't seem to remove the file. After taking some time (and re-inspired by your suggestion), I ended up with:

main = shake shakeOptions $ do
  phony "build" $ do
    cmd_ "cabal new-build all"
    removeFile "" [".ghc.environment*"]

Posted for posterity/completeness. Hopefully it helps the next person that gets bit by this bug.

[mankyKitty]

I agree that some sort of flag is needed to control whether or not this file is used. However I would err on the opt out side of things, as I see myself as not really being the 'common use-case'.

For the sake of people looking for similar answers, I wrote the following function in my emacs config to help manage it:

  (defun remove-ghc-env-file-then (f)
    (shell-command-to-string (concat "rm " (projectile-project-root) ".ghc.environment.x86_64-linux-*"))
    f
    )

[nomeata]

Not sure how helpful it is to amend to this already very long thread, but I feel that this point is not addressed here:

Cabal provides the very useful flag --distdir that I can use to maintain two separate “contexts” of the same project – maybe different dependencies, different build flags etc. But this seems to be inherently incompatible with creating a GHC environment file that lives outside the distdir: It would unpredictably refer to the distdir that I used last? (The environment filename encodes the compiler used, but not any of the other ways cabal configurations could differ.)

Hence my expectation indeed is that cabal never touched files outside of dist-newstyle/ (or whatever distdir I configure), and that whenever I want to do something “in the context of a configured Cabal projects”, I expect to use one of the cabal commands.