Manage external source trees as Git submodules #2487
Conversation
If an example directory exists but does not have a CMakeLists.txt file then assume it is not complete and ignore it because add_subdirectory will fail anyway.
This makes `git status` in the nlopt source tree report a clean tree by ignoring an executable that appears in its source tree when building with `octave` installed.
The DOWNLOAD_DIR option is not used when GIT_REPOSITORY is used.
Source trees currently placed in our source tree by ExternalProject build rules will soon be converted to Git submodules, so we should no longer ignore them.
The CMake ExternalProject module produces a `mkdir` step that makes
relevant directories before other steps, including the build directory.
For `director` we set the BINARY_DIR to be inside the source tree
because that is where we want `make` to run. When we later consolidate
our `download-${proj}` targets into their main `${proj}` targets then
this will cause a subdirectory to be made in the director source tree
before its `download` step. This will cause `git clone` to fail
complaining that the clone destination directory is a non-empty
directory. Avoid this problem by adding a `chdir` step to our in-source
`director` BUILD_COMMAND so that we can use a BINARY_DIR that does not
sit in a subdirectory of the source tree.
…dule Later we may optionally convert the submodule configuration to follow a branch to restore current behavior.
Hack the `CMakeLists.txt` file to convert all of our
<proj>_GIT_REPOSITORY
<proj>_GIT_TAG
pairs into Git submodule configuration.
Use Git submodules to track external source trees and manage checkouts. This will allow Git, our version control tool, to manage downloads and updates of both our source tree and its external project source trees. For example, a single `git status` from the top will now report the state of all source trees used to build the project. Convert our table of `<proj>_GIT_REPOSITORY`/`<proj>_GIT_TAG` pairs from the CMake code into Git submodule configuration in the source tree. Perform the conversion automatically by running CMake once. Our temporarily hacked `CMakeLists.txt` file configures the `.gitmodules` file and adds the submodule entries to the Git index.
…odules Remove from `CMakeLists.txt` the hack previously added to perform the conversion.
|
Some CI builds fail with: I suspect this is caused by parallel downloads colliding on submodule initialization. Will fix. |
|
|
That could be; I always run the top level build as In fact, since we require CMake 3.5 which I believe is new enough, can we just make the externals use ninja's console pool for all steps? (I guess we'll still need a fix for other generators, but this would be nice anyway...) |
I've revised the topic to fix this, and the CI builds are now clean. The fix was simply to perform
Perhaps, but that is beyond the scope of this change/PR. |
The topic commit history shows how this was done automatically to preserve the values. Basically I added temporary changes to |
I was expecting you would need to explicitly serialize the download steps, likely making changes very similar to having the externals use the console pool also. Since you ended up doing something quite different, such a change would indeed be much more orthogonal than I was anticipating. Anyway, LGTM... Reviewed 1 of 1 files at r3. Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion. drake/doc/from_source.rst, line 137 [r3] (raw file):
Am I correctly understanding the following?
If so: is it possible to make the generated build files re-run Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion. drake/doc/from_source.rst, line 137 [r3] (raw file): We could have CMake create this alias for the local Git work tree, or provide a "developer setup" script.
We could make the build do the update by setting the external project Either way we will still have gained whole-tree management by Git (e.g. Comments from Reviewable |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion. drake/doc/from_source.rst, line 137 [r3] (raw file):
|
|
Review status: 34 of 35 files reviewed at latest revision, 1 unresolved discussion. drake/doc/from_source.rst, line 137 [r3] (raw file):
|
|
Looks good to me. @RussTedrake, did you want to review this change as well?
|
|
Reviewed 33 of 34 files at r1, 1 of 1 files at r2, 1 of 1 files at r4. Comments from Reviewable |
|
We actually already had this discussion almost two years ago: #1169 and came to exactly the opposite conclusion. We had been using git submodules within drake for a long time, and ultimately decided to abandon them for several reasons:
|
|
@rdeits thanks for the link and the summary of previous discussion. I've found it difficult to work with Drake externals without submodules. It is very hard to tell whether any external sources have been modified. In fact while working on this I found that our builds were making some source trees dirty and no one noticed (I added
With the approach proposed in this PR no users actually have to deal with submodules. The workflow is the same as before for users. The needed subset of externals is automatically selected by the CMake configuration and used to configure a subset of submodules. Builds automatically update them unless
Currently we have no non-Git externals. We can handle them by importing them into Git ourselves in a dedicated support repository. Even before this change the externals logic hard-coded use of
Turn off |
|
|
@rdeits -- thanks for clearly articulating the concerns that I had immediately, too. I'm still a bit worried because the current system has been much better / cleaner than any submodule approach we had before. a few clarification questions to make sure I understand:
We should also add some documentation in the developers section telling them how to update the submodule hash since I know we'll get questions about it.
|
Yes.
Yes.
Yes.
The Without the Review status: all files reviewed at latest revision, 3 unresolved discussions. drake/doc/from_source.rst, line 137 [r4] (raw file):
|
It will, as will I believe |
|
thx. i trust that you've thought it through. i'm willing to give it a shot.
|
|
@drake-jenkins-bot retest this please |
|
This is ready to merge other than some Reviewable discussions not marked as resolved. |
|
Review status: all files reviewed at latest revision, 1 unresolved discussion. CMakeLists.txt, line 316 [r4] (raw file):
|
|
Review status: all files reviewed at latest revision, 1 unresolved discussion. drake/doc/from_source.rst, line 137 [r4] (raw file):
|
Our external source trees are now managed by Git as submodules. Teach
our external projects to download and update their source trees via
git submodule update --init --recursive $path
Users that turn off the AUTO_UPDATE_EXTERNALS option may now run
git submodule update --recursive
to update submodule checkouts after changing versions of the host
project (e.g. `git pull`).
Combine the `${proj}` and `download-${proj}` external project targets
into a single `${proj}` external project target that contains all steps.
Add a separate `${proj}-update` target that depends on the update step
of `${proj}` (and transitively on the download step). Then make
`download-all` depend on all of those.
|
Review status: all files reviewed at latest revision, all discussions resolved. drake/doc/from_source.rst, line 137 [r4] (raw file):
|
Drake's top-level
CMakeLists.txtfile has a table of Git repository URLs and commit SHA-1s for the external source trees. These two pieces of information are exactly what Git submodules record (in a.gitmodulesfile and a tree-object entry, respectively). Drake also tells CMake's ExternalProject module to create build rules that clone the Git repositories into subdirectories within its own source tree. This is exactly what Git submodules are meant to do.Drake's external source trees are actually part of its overall source tree from the perspective of local development and build operations. This is why we place them in subdirectories of the main source tree. They just happen to be externally maintained and version-controlled separately. Using the build system to manage parts of the source tree is sub-optimal:
AUTO_UPDATE_EXTERNALSoption to help developers update their external source trees after performinggit pullon the main source tree while still providing a way to do local development in the external source trees without erasing the changes on the next build..gitignoretheexternals/*directories (and a few others) since they are placed in the source tree but not managed by Git as part of the main source tree. This means that top-level operations likegit statusonly see the main source tree rather than the whole source tree.A solution to these problems is to use Git to manage the whole source tree by using Git submodules to manage the external source trees underneath our main source tree. The only involvement we need from the build system is to determine which external source trees need to be downloaded for the chosen build options. After the needed external source trees (submodules) have been downloaded (cloned) all further source tree status and update operations can be managed cleanly by the version control tool (Git).
This change is