new-exec #4722

fgaz · 2017-08-24T16:51:40Z

Merged master with dmwit/master and finished new-exec.

Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.
Basic tests.

I had some trouble with git. There are two mege commits where I should have rebased instead, and now it's kinda difficult to fix (see those new-bench commits? they are from the merge.), but I suppose I'll have to do it before merging in master?

Also there's the question of how to rewrite the ghc program name. As a temporary solution I just check if it's ``elem ["ghc", "ghcjs", "ghci"] and use it as is.
If there is no objection I'll change it to use the selected (`-w`) ghc version.

Fixes #3638 (finally!)

Although we may not want to create GHC environment files in a standard location after every build, it is still useful to be able to build them in a custom place in special circumstances. So let's disable this feature at the call-sites where it doesn't work properly rather than inside the implementation of the feature.

This implements a bare-bones skeleton for cabal new-exec. The old cabal exec gave programs access to a sandbox's package database. By analogy, cabal new-exec should give programs access to the store's package database; however, this database will be cluttered with many non-project-related packages that may confuse issues. Therefore new-exec selects just the packages that are in the current project's dependency tree and makes them available to compiler tools. Currently only very new GHCs are supported, via the GHC_ENVIRONMENT mechanism for selecting a subset of some package databases. Eventually we should probably also modify the PATH so that dependencies' executables are available.

cabal new-exec should be allowed to run any executables available in the dependencies for the current project. This patch introduces that ability: we walk the installation plan, picking out a bin/ directory for each package, then including it in the PATH if there are any executables in that directory.

Previously, whichever version of compiler tools that were on the PATH would be used. Going through the ProgramDb should let us use the appropriate versions of these tools, instead.

Previously, cabal new-exec would modify the PATH by including each package's bin directory if it contained executables. This has some drawbacks: * Lots of filesystem probing * Can have false positives due to dynamic libraries * Mildly unpredictable The new plan is to add each package's bin directory if the package includes an executable stanza. This may include some directories that have no executables in them (e.g. if the executable is not yet built or is not in the install plan), but is more reproducible and requires less filesystem access.

Previously, new-exec would use the `build` directory when looking for executables built inplace. But these executables are actually in subdirectories of `build`, so the search would fail (potentially falling back to an executable in the user's PATH instead). Now we place one directory in the PATH for each executable in each inplace-build package in the install plan.

Based on new-test.

Still in progress. The support check doesn't work yet.

The global db (note: not the user db) is needed (it contains `base`).

Edge cases first, restrict do blocks...

...rather than in the command

fgaz · 2017-08-24T16:56:37Z

cabal-install/Distribution/Client/CmdExec.hs

+              (elaboratedPlanOriginal buildCtx)
+              buildStatus
+          programIsCompiler =
+            programId program `elem` ["ghc", "ghcjs", "ghci"]


How do we handle the compiler name rewriting?
(see pr description)
EDIT: this will change in the next push

23Skidoo · 2017-08-25T14:39:20Z

I'm on vacation in Norway right now, but I'll review this when I get back next week.

cocreature · 2017-08-28T08:23:00Z

FWIW, as a user I would like cabal new-exec to not rewrite commands and only set environment variables. Rewriting commands is bound to cause confusion and will always be really fragile, e.g., cabal new-exec -- ghc might work but cabal new-exec -- time ghc won’t. I can see a case for a way to execute ghc regardless of how the executable is called but I would prefer for that to be a separate cabal new-ghc command.

fgaz · 2017-08-28T09:20:50Z

@cocreature the rewriting only happens as a fallback when using older versions of ghc that don't support .ghc.environment files, and eventually (well, after the 5 or so years of support...) will disappear. Other compilers will probably need rewriting though (I'm changing that code to be more general).

I agree that the naming is unfortunate, and I actually would prefer a new-ghc too, but new-exec is meant to be kinda compatible with old exec, and cabal -w "some-ghc" new-exec -- "some-ghc" will be completely equivalent to the hypotetical cabal -w "some-ghc" new-ghc.

Of course if there is enough support we can always split the command, it's only a few changes.

cocreature · 2017-08-28T09:23:22Z

@fgaz There will be an option to disable .ghc.environment.* files #4542 (which I am definitely going to use) so I don’t think you can rely on this ever going away.

fgaz · 2017-08-28T10:35:31Z

@cocreature that flag only applies to the .ghc.env file that every new- command generates in the project root. new-exec uses a separate and temporary .ghc.env file which is always generated regardless of that flag.

Alter the compiler flags only if the exe path matches the configured compiler's one. Ex. `cabal -w ghc-8.0 new-exec ghc-8.0 Main.hs` The flags depend on the compiler flavor. Only GHC and GHCJS are supported right now.

fgaz · 2017-09-01T20:16:35Z

cabal-install/Distribution/Client/ProjectPlanOutput.hs

+  case compilerId compiler
+  of CompilerId GHC   _ -> argsEquivalentOfGhcEnvironmentFileGhc
+     CompilerId GHCJS _ -> argsEquivalentOfGhcEnvironmentFileGhc
+     CompilerId _     _ -> error "Only GHC and GHCJS are supported"


Is error appropriate? ImplInfo does it like this too.

fgaz · 2017-09-01T20:17:56Z

cabal-install/Distribution/Client/CmdExec.hs

+matchCompilerPath elaboratedShared program =
+  programPath program
+  `elem`
+  (programPath <$> configuredCompilers)


Should i canonicalize those paths?

Not 100% sure, probably not. Aren't the configured progs paths already absolute?

No idea. They seem absolute but maybe it's only the general case. I'll probably leave it like this and add a comment about it.

OK. I think they should be absolute after the configure progs step.

23Skidoo · 2017-09-19T18:30:28Z

OSX failure seems to be spurious.

23Skidoo

So this looks fine, nicely self-contained and ready to go in. Would be nice with a changelog note and a section in the user's guide, but that (and minor fixes I suggested) can be part of a separate PR.

23Skidoo · 2017-09-19T19:54:30Z

cabal-install/Distribution/Client/ProjectOrchestration.hs

-                            elaboratedPlanOriginal
-                            elaboratedShared
-                            postBuildStatus
+    _ <- writePlanGhcEnvironment (distProjectRootDirectory


Also can use void here.

23Skidoo · 2017-09-19T20:00:33Z

cabal-install/Distribution/Client/CmdExec.hs

+import qualified Data.Map as M
+
+execCommand :: CommandUI (ConfigFlags, ConfigExFlags, InstallFlags, HaddockFlags)
+execCommand = installCommand


This is a bit hacky. installCommand is not obviously related to new-build and will be going away. Also we don't actually want to pull in everything from there. So I'd be more explicit and use commandOptions = commandOptions CmdBuild.buildCommand, .... I understand that CmdBuild.buildCommand is giving a bad example, but we should fix it too.

23Skidoo · 2017-09-19T20:13:06Z

cabal-install/Distribution/Client/ProjectPlanOutput.hs

+  case compilerId compiler
+  of CompilerId GHC   _ -> argsEquivalentOfGhcEnvironmentFileGhc
+     CompilerId GHCJS _ -> argsEquivalentOfGhcEnvironmentFileGhc
+     CompilerId _     _ -> error "Only GHC and GHCJS are supported"


23Skidoo · 2017-09-19T20:26:18Z

cabal-install/Distribution/Client/ProjectPlanOutput.hs

+     CompilerId GHCJS _ -> argsEquivalentOfGhcEnvironmentFileGhc
+     CompilerId _     _ -> error "Only GHC and GHCJS are supported"
+
+-- remove this when we drop support for non-.ghc.env ghc


Forgotten TODO?

23Skidoo · 2017-09-19T20:29:01Z

cabal-install/Distribution/Client/CmdExec.hs

+matchCompilerPath elaboratedShared program =
+  programPath program
+  `elem`
+  (programPath <$> configuredCompilers)


Not 100% sure, probably not. Aren't the configured progs paths already absolute?

fgaz · 2017-09-26T21:05:17Z

All done.
EDIT: oh wait, the docs

...except that git rerere failed me (or I failed it) and now we have three more commits that could have been fixupd. Can I merge anyway once ci passes?

[ci skip]

23Skidoo · 2017-09-26T21:31:51Z

There's a merge conflict.

23Skidoo · 2017-09-26T23:23:54Z

Merged, thanks!

dmwit and others added 30 commits February 24, 2017 15:10

enviroment -> environment

b41505a

factor out the logic for choosing a directory for binaries

c5f0052

use the ProgramDb in new-exec

43893e4

Previously, whichever version of compiler tools that were on the PATH would be used. Going through the ProgramDb should let us use the appropriate versions of these tools, instead.

write help text for new-exec

1d4533e

backwards compatibility

603897f

comment on a tricky line of code

e49880b

accept all new-build flags in new-exec

a6df3e8

Make new-bench actually run the benchmarks

b13bcbf

Based on new-test.

Add brief description of mew-bench in docs

ae9ae09

Add new-bench to changelog

f7c2b18

Merge branch 'master' of github.com:dmwit/cabal into new-exec/1

b7f726c

Adapt new-exec to the new new-build structure

57763ca

Merge branch 'master' into new-exec/1

b57f9bd

Fix some warnings

ce2518d

Re-enable .ghc.env temp directory for new-exec

4ee6da9

Add args fallback when we can't use .ghc.env files

33a8b51

Still in progress. The support check doesn't work yet.

Add basic test for new-exec

e923107

Add global package env to new-exec ghc invocations

5e454c6

The global db (note: not the user db) is needed (it contains `base`).

Reorder things a bit

194453e

Edge cases first, restrict do blocks...

Check if the compiler is supported in the buildCtx

6b9d462

...rather than in the command

Don't use ghc env files if they aren't supported

e0b085e

Exec args are defaults, not overrides

9298ad8

Make withTempEnvFile its own function

8e050a6

Add test: new-exec can run an exe

5fd32dc

Remove useless comment

d0ba72d

fgaz mentioned this pull request Aug 24, 2017

implement a new-exec command #4304

Closed

fgaz commented Aug 24, 2017

View reviewed changes

Match the configured compiler in new-exec

630634a

Alter the compiler flags only if the exe path matches the configured compiler's one. Ex. `cabal -w ghc-8.0 new-exec ghc-8.0 Main.hs` The flags depend on the compiler flavor. Only GHC and GHCJS are supported right now.

fgaz commented Sep 1, 2017

View reviewed changes

fgaz requested a review from 23Skidoo September 1, 2017 21:07

23Skidoo added this to the 2.2 milestone Sep 19, 2017

23Skidoo added the cabal-install: nix-local-build label Sep 19, 2017

23Skidoo approved these changes Sep 19, 2017

View reviewed changes

Use void instead of _ <-

f99f169

fgaz force-pushed the new-exec/1 branch from 8912fc8 to f99f169 Compare September 26, 2017 20:43

fgaz added 2 commits September 26, 2017 22:53

Fix comment

ca31578

Don't base execCommand on installCommand

8945ebd

fgaz added 2 commits September 26, 2017 23:12

Add new-exec to changelog

cb9a355

[ci skip]

Add new-exec to the docs

442bb91

[ci skip]

fgaz force-pushed the new-exec/1 branch from a51cf25 to 442bb91 Compare September 26, 2017 21:27

Merge branch 'master' into new-exec/1

28c9fd8

23Skidoo merged commit 87a9dfa into haskell:master Sep 26, 2017

fgaz deleted the new-exec/1 branch July 9, 2018 12:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new-exec #4722

new-exec #4722

fgaz commented Aug 24, 2017 •

edited

Loading

fgaz Aug 24, 2017 •

edited

Loading

23Skidoo commented Aug 25, 2017

cocreature commented Aug 28, 2017

fgaz commented Aug 28, 2017

cocreature commented Aug 28, 2017

fgaz commented Aug 28, 2017

fgaz Sep 1, 2017

23Skidoo Sep 19, 2017

fgaz Sep 1, 2017

23Skidoo Sep 19, 2017

fgaz Sep 23, 2017

23Skidoo Sep 23, 2017

23Skidoo commented Sep 19, 2017

23Skidoo left a comment •

edited

Loading

23Skidoo Sep 19, 2017

23Skidoo Sep 19, 2017

23Skidoo Sep 19, 2017

23Skidoo Sep 19, 2017

23Skidoo Sep 19, 2017

fgaz commented Sep 26, 2017 •

edited

Loading

23Skidoo commented Sep 26, 2017

23Skidoo commented Sep 26, 2017

new-exec #4722

new-exec #4722

Conversation

fgaz commented Aug 24, 2017 • edited Loading

fgaz Aug 24, 2017 • edited Loading

Choose a reason for hiding this comment

23Skidoo commented Aug 25, 2017

cocreature commented Aug 28, 2017

fgaz commented Aug 28, 2017

cocreature commented Aug 28, 2017

fgaz commented Aug 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

23Skidoo commented Sep 19, 2017

23Skidoo left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgaz commented Sep 26, 2017 • edited Loading

23Skidoo commented Sep 26, 2017

23Skidoo commented Sep 26, 2017

fgaz commented Aug 24, 2017 •

edited

Loading

fgaz Aug 24, 2017 •

edited

Loading

23Skidoo left a comment •

edited

Loading

fgaz commented Sep 26, 2017 •

edited

Loading