Move build products out of the ghc tree #113

angerman · 2016-01-06T05:30:10Z

We've discussed this in #32 already when debating how to properly reset the build. So far we have depended on the build products being built inside of the ghc tree, such that we could potentially use makes build artifacts during the build. As we now can build on Windows, OS X and Linux.

snowleopard · 2016-01-06T11:27:47Z

Agreed, it's time to start moving things around. Let me take care of this.

See #113.

snowleopard · 2016-01-10T05:03:01Z

I moved most build artefacts to buildRootPath = ".build" which can be changed in Settings/User.hs. An example command invocation:

/-----------------------------------------------\
| Run Ghc Stage0 (package = ghc)                |
|     input: compiler/main/GHC.hs               |
| => output: .build/stage0/compiler/build/GHC.o |
\-----------------------------------------------/

This is still work in progress and there are a few temporary hacks. I'll probably drop the intermediate build to make paths shorter.

The current progress is:

gmp
program binaries
package database inplace/lib/package.conf.d
generated files in includes
files produced by ghc-cabal (e.g. package-data.mk)

Should be able to finish this tomorrow, although the last one seems tricky. We may first need to tackle the ghc-cabal issue #18.

See #113 and #148.

See #113.

snowleopard · 2016-01-11T02:11:38Z

Coming across some limitations unfortunately. For example:

Exit code: 1
Stderr:
In file included from compiler\nativeGen\AsmCodeGen.hs:14:0:

compiler\HsVersions.h:23:0:
     fatal error: ../includes/ghcautoconf.h: No such file or directory
compilation terminated.

For some reason compiler\HsVersions.h includes a generated file directly ../includes/ghcautoconf.h, not relying on -I paths. This forces me to keep a copy of generated file in includes folder.

Does anyone know the reason for #include "../includes/ghcautoconf.h"? Shall this be fixed in GHC?

/cc: @bgamari @hvr

snowleopard · 2016-01-11T02:16:02Z

And some more :-(

Exit code: 1
Stderr:
compiler\main\DynFlags.hs:127:0:
     fatal error: ../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs: No such file or directory
compilation terminated.

Exit code: 1
Stderr:
compiler\main\DynFlags.hs:4191:0:
     fatal error: ../includes/dist-derivedconstants/header/GHCConstantsHaskellWrappers.hs: No such file or directory
compilation terminated.

Exit code: 1
Stderr:
compiler\main\PlatformConstants.hs:13:0:
     fatal error: ../includes/dist-derivedconstants/header/GHCConstantsHaskellType.hs: No such file or directory
compilation terminated.

snowleopard · 2016-01-11T03:11:15Z

Another mystery. When invoking the following command:

inplace/bin/ghc-stage1.exe
-hisuf hi
-osuf o
-hcsuf hc
-static
-hide-all-packages
-no-user-package-db
-this-package-key rts
-i
-i.build/stage1/rts/build
-i.build/stage1/rts/build/autogen
-I.build/stage1/rts/build
-I.build/stage1/rts/build/autogen
-optP-include
-optP.build/stage1/rts/build/autogen/cabal_macros.h
-odir .build/stage1/rts/build
-hidir .build/stage1/rts/build
-stubdir .build/stage1/rts/build
-rtsopts
-H32m
-O2
-c rts/PrimOps.cmm
-o .build/stage1/rts/build/PrimOps.o
-Irts
-Iincludes
-I.build/includes

I get the following error:

ghc-stage1.exe: ghcversion.h missing

Even though folder .build/includes contains ghcversion.h. The command goes ahead happily when I copy ghcversion.h to includes directory. No idea why this happens despite -I.build/includes.

Could this be a GHC bug too?

angerman · 2016-01-11T03:25:16Z

ghc-stage1.exe: ghcversion.h missing

I guess you are hitting this:

-- | Find out path to @ghcversion.h@ file
getGhcVersionPathName :: DynFlags -> IO FilePath
getGhcVersionPathName dflags = do
  dirs <- getPackageIncludePath dflags [rtsUnitId]

  found <- filterM doesFileExist (map (</> "ghcversion.h") dirs)
  case found of
      []    -> throwGhcExceptionIO (InstallationError ("ghcversion.h missing"))
      (x:_) -> return x

from compiler/main/DriverPipeline.hs

snowleopard · 2016-01-11T03:26:43Z

@angerman Thanks! I think you are right.

snowleopard · 2016-01-11T03:27:41Z

OK, it looks like moving generated files from includes to .build/includes is just too cumbersome due to the above issues. I don't think this this is worth it. We'll keep mutating includes folder just like the old build system did. I suggest we do the same with inplace for now, to avoid further complications.

The only remaining bit then is to move files produced by ghc-cabal (e.g. package-data.mk). Hopefully I'll have more luck with this. #108 looks scary, but maybe I need to give it a try.

angerman · 2016-01-11T07:19:55Z

@hvr, just mentioned https://ghc.haskell.org/trac/ghc/ticket/8040 on irc.

snowleopard · 2016-01-11T12:22:14Z

@angerman Thanks. Maybe we need to link to this issue from that ticket.

angerman · 2016-01-12T01:50:47Z

@snowleopard done. Just linked this issue from trac.

@snowleopard could you leave a compiled list of all the files we had to leave in the ghc tree and could not move out of it? So we can track that separately and potentially fix it upstream?

snowleopard · 2016-01-12T01:56:17Z

@angerman Thanks! Here is the list of what will stay in the tree for now:

All generated files in includes.
All binaries/generated files placed into inplace. I'm not sure we can change any conventions here, because people may depend on them.
Everything else, I hope, will end up in .build.

bgamari · 2016-09-15T19:03:12Z

@snowleopard, I think I have fixes for most of the issues above in D2530. This leaves only the ghcversion.h issue. It seems like the correct way to deal with this would be for Hadrian to ensure that the build directory is included in the rts library's include path. Would this be possible?

snowleopard · 2016-09-16T00:24:12Z

@bgamari Great, many thanks!

This leaves only the ghcversion.h issue.

Just to make sure I understand: are you saying we no longer need to generate files into includes and can in fact move all generated files from there to _build/includes?

If yes, I'll give it a try! Adding build directory to rts's include path should be easy (although judging by this comment: #113 (comment), this has already been attempted).

bgamari · 2016-10-02T02:39:27Z

Just to make sure I understand: are you saying we no longer need to generate files into includes and can in fact move all generated files from there to _build/includes?

I believe so. D2530 has been merged so you should now be able to move ahead with this. Let me know if you have any trouble.

See #113.

snowleopard · 2016-10-02T15:53:40Z

@bgamari Thanks! The above commit moves all generated includes into _build/generated.

There are still a few remaining bits, see the todo list in #113 (comment). The biggest one is the files generated by ghc-cabal. As we discussed previously, the easiest solution would be to modify ghc-cabal so it is possible to specify a folder where to put generated files (currently it creates them in the package directory).

snowleopard · 2016-10-02T16:27:23Z

@bgamari Ah, there is still one remaining relative path in DynFlags.hs:

#include "../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs

bgamari · 2016-10-02T17:02:14Z

Andrey Mokhov [email protected] writes:

@bgamari Ah, there is still one remaining relative path in DynFlags.hs:
#include "../includes/dist-derivedconstants/header/GHCConstantsHaskellExports.hs
Bah, sorry about that. Opened D2559 with a fix.

See #113

See #113.

snowleopard · 2016-10-02T20:00:47Z

@bgamari Thanks for fixing!

There is one more problem: ghcversion.h seems to be baked in somehow -- if I move it to _build/generated, I get ghc-stage1.exe: ghcversion.h missing from GHC even though I pass the directory via -I. Looking at the code, it seems that GHC looks for ghcversion.h in RTS package include directories for some reason, which point to includes.

I temporarily copy ghcversion.h to includes, but I hope there is a better solution.

bgamari · 2016-10-02T20:07:44Z

There is one more problem: ghcversion.h seems to be baked in somehow

Yes, it is. In an earlier message I suggested you deal with this by adding your build directory to the include-path of the rts package.

See #113.

snowleopard · 2016-10-02T20:52:19Z

@bgamari Ah, indeed, I confused filenames -- sorry!

As I understand, you refer to the following line in RTS package.conf.in file:

include-dirs:           TOP"/rts/dist/build" TOP"/includes" TOP"/includes/dist-derivedconstants/header" FFI_INCLUDE_DIR

In a way, we'd like to change it to

include-dirs:           TOP"/rts/dist/build" TOP"/includes" TOP"/_build/generated" FFI_INCLUDE_DIR

This is a bit tricky, because _build is not fixed but is a user setting. Anyway, the above commit seems to do the trick.

bgamari · 2016-10-02T23:43:41Z

Anyway, the above commit seems to do the trick.

Heh, oh that is devious ;)

See #113.

ezyang · 2016-11-16T18:40:51Z

What's the progress on this? When I run a Hadrian build with the master branches, I still get a pile of stage0/stage1/stage2 directories.

snowleopard · 2016-11-17T00:35:29Z

@ezyang The progress is tracked here: #113 (comment)

The pile of stage0/1/2 directories is generated by ghc-cabal #18, and noone yet was brave/persistent enough to tackle this ticket.

See #113

snowleopard · 2016-11-27T17:20:56Z

The above commits move the directories generated by ghc-cabal into the build directory.

I believe we can now close this issue, because moving program binaries outside inplace/bin and Stage1 package database outside inplace/lib is probably beyond the scope of Hadrian (and it can be easily done if/when we decide to do that).

Note: ghc-cabal still runs the configure scripts locally for some packages, generating some files in the tree, but I don't think we can do anything about it before tackling #18.

If you spot any other build artefacts that should be moved outside of the GHC tree, feel free to reopen this issue, or even better create a new one.

See #113

[skip ci]

angerman added the enhancement label Jan 6, 2016

snowleopard self-assigned this Jan 6, 2016

snowleopard mentioned this issue Jan 8, 2016

Add a clean check #131

Open

snowleopard added a commit that referenced this issue Jan 10, 2016

Move build artefacts to .build/ directory.

eda85ff

See #113.

angerman mentioned this issue Jan 10, 2016

Build fails: compiler/prelude/primops.txt.pp:76:10: fatal error: 'ghc_boot_platform.h' file not found #145

Closed

snowleopard added a commit that referenced this issue Jan 10, 2016

Postprocess inplace-pkg-config files.

f84ee22

See #113 and #148.

This was referenced Jan 10, 2016

Define a rough roadmap? #147

Closed

General refactorings #144

Merged

snowleopard added a commit that referenced this issue Jan 10, 2016

Move gmp build results to buildRootPath.

a850455

See #113.

snowleopard added a commit that referenced this issue Jan 11, 2016

Move bootstrapping.conf to .build.

24e6c28

See #113.

snowleopard mentioned this issue Jan 11, 2016

[WIP] Use Cabal for package data #108

Closed

snowleopard added this to the first-shake milestone Jan 12, 2016

snowleopard mentioned this issue Jan 12, 2016

Get rid of ghc-cabal and package-data.mk #18

Closed

snowleopard modified the milestones: tree-tremble, first-shake Jan 22, 2016

This was referenced Jan 25, 2016

First Attempt Log #194

Closed

Add validation support #187

Closed

snowleopard mentioned this issue Mar 4, 2016

Successful build error #206

Closed

snowleopard mentioned this issue Apr 17, 2016

$srcdir != $builddir support? #226

Closed

snowleopard removed this from the tree-tremble milestone May 1, 2016

snowleopard mentioned this issue May 11, 2016

hadrian/build.sh clean continues building Hadrian, instead of cleaning stuff up #249

Closed

snowleopard added a commit that referenced this issue Oct 2, 2016

Move generated includes to build directory

f2cff6f

See #113.

snowleopard added a commit that referenced this issue Oct 2, 2016

Drop TODOs

456a10b

See #113

snowleopard added a commit that referenced this issue Oct 2, 2016

Fix generated includes

c6cb106

See #113.

snowleopard added a commit that referenced this issue Oct 2, 2016

Fix path to ghcversion.h header

0ff86b4

See #113.

snowleopard added a commit that referenced this issue Oct 23, 2016

Don't generate files into the source tree

7303fcf

See #113.

snowleopard added a commit that referenced this issue Nov 26, 2016

Don't generate in-tree directories by ghc-cabal

94c88da

See #113

snowleopard added a commit that referenced this issue Nov 27, 2016

Drop redundant clean up after ghc-cabal

e93f7a4

See #113

snowleopard closed this as completed Nov 27, 2016

snowleopard added a commit that referenced this issue Nov 27, 2016

Drop old TODOs

92b5c35

See #113

snowleopard added a commit that referenced this issue Nov 27, 2016

Drop references to #113

0412d60

[skip ci]

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move build products out of the ghc tree #113

Move build products out of the ghc tree #113

angerman commented Jan 6, 2016

snowleopard commented Jan 6, 2016

snowleopard commented Jan 10, 2016 •

edited

Loading

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 11, 2016

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 12, 2016

snowleopard commented Jan 12, 2016

bgamari commented Sep 15, 2016

snowleopard commented Sep 16, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

ezyang commented Nov 16, 2016

snowleopard commented Nov 17, 2016

snowleopard commented Nov 27, 2016

Move build products out of the ghc tree #113

Move build products out of the ghc tree #113

Comments

angerman commented Jan 6, 2016

snowleopard commented Jan 6, 2016

snowleopard commented Jan 10, 2016 • edited Loading

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 11, 2016

snowleopard commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 11, 2016

snowleopard commented Jan 11, 2016

angerman commented Jan 12, 2016

snowleopard commented Jan 12, 2016

bgamari commented Sep 15, 2016

snowleopard commented Sep 16, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

snowleopard commented Oct 2, 2016

bgamari commented Oct 2, 2016

ezyang commented Nov 16, 2016

snowleopard commented Nov 17, 2016

snowleopard commented Nov 27, 2016

snowleopard commented Jan 10, 2016 •

edited

Loading