docker: Use system GIT_CACHE_DIR if available #9651

cladmi · 2018-07-31T13:13:51Z

Contribution description

If GIT_CACHE_DIR is a directory make it available to docker.

I used the same naming convention I used in #9646

Optional

I did not remove the default value for GIT_CACHE_DIR from dist/tools/git/git-cache but duplicated it in the build system to minimize changes to the script. I could do it in this PR or another if wanted.

Testing

You must have ${HOME}/.gitcache directory created to use git-cache.

# initialize the package if it is not in git-cache
make BOARD=native -C examples/ccn-lite-relay/ clean all
...
# Then build in docker
make BOARD=native -C examples/ccn-lite-relay/ BUILD_IN_DOCKER=1 DOCKER="sudo docker"  clean all`
make: Entering directory '/home/harter/work/git/RIOT/examples/ccn-lite-relay'
make[1]: Nothing to be done for 'Makefile.include'.
make[1]: Nothing to be done for 'Makefile.include'.
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/RIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/RIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/RIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
    -e 'BOARD=native' \
    -w '/data/riotbuild/riotproject/examples/ccn-lite-relay/' \
    'riot/riotbuild:latest' make  'BOARD=native'
...
GIT_CACHE_DIR=/data/riotbuild/gitcache /data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/cn-uofbasel/ccn-lite/" "48b17ebee7600b2e79b3acf0728217473d7a4ee8" "/data/riotbuild/riotproject/examples/ccn-lite-relay/bin/pkg/native/ccn-lite"
git-cache: cloning from cache. tag=commit48b17ebee7600b2e79b3acf0728217473d7a4ee8
...

You can notice the -e GIT_CACHE_DIR=/data/riotbuild/gitcache and git-cache: cloning from cache.

Issues/PRs references

Fixes #8267 if it is ok that I do not create the cache dir by default but only use the existing one.

jnohlgard · 2018-07-31T13:26:19Z

pkg/pkg.mk

@@ -31,7 +31,7 @@ endif
 $(PKG_BUILDDIR)/.git-downloaded:
 	rm -Rf $(PKG_BUILDDIR)
 	mkdir -p $(PKG_BUILDDIR)
-	$(GITCACHE) clone "$(PKG_URL)" "$(PKG_VERSION)" "$(PKG_BUILDDIR)"
+	GIT_CACHE_DIR=$(GIT_CACHE_DIR) $(GITCACHE) clone "$(PKG_URL)" "$(PKG_VERSION)" "$(PKG_BUILDDIR)"


Do you need to write this environment variable on the command line if it is already exported (vars.mk)?

That is a remnant of when I tried to not export the variable, but it failed because of sub makefiles.

I left it to show that the script was using the variable but I can remove it.

I removed it, it is an internal script anyway and git grep GIT_CACHE_DIR finds it.

cladmi · 2018-08-06T08:54:28Z

I removed the trailing slash for $(wildcard $(GIT_CACHE_DIR)/) otherwise it would match / for an empty variable…

I could go for a $(filter-out /,$(wildcard $(GIT_CACHE_DIR)/)) but I am not sure if I really need to handle GIT_CACHE_DIR being set to a file.

cladmi · 2018-10-21T16:54:28Z

@jia200x It would be great to have this for the release to be able to test building in docker in proper conditions.
If it is not included, I would merge it in my test branch anyway.

jia200x · 2018-10-22T17:56:31Z

without this PR:

Compile OpenThread for FTD device
$OPENTHREAD_ARGS is [--enable-cli-app=ftd --enable-application-coap]
rm -Rf /data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread
mkdir -p /data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/openthread/openthread.git" "thread-reference-20170716" "/data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread"
Cloning into '/data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread'...
remote: Enumerating objects: 2, done.
remote: Counting objects: 100% (2/2), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 38470 (delta 1), reused 0 (delta 0), pack-reused 38468
Receiving objects: 100% (38470/38470), 52.24 MiB | 18.96 MiB/s, done.
Resolving deltas: 100% (28721/28721), done.
Checking connectivity... done.
HEAD is now at a8496c8... [configure]: update mac_filter help string (#2004)

with this PR:

$OPENTHREAD_ARGS is [--enable-cli-app=ftd --enable-application-coap]
rm -Rf /data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread
mkdir -p /data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/openthread/openthread.git" "thread-reference-20170716" "/data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread"
git-cache: cloning from cache. tag=415f21b1e830a8ac8f26ccbba68ec539183586a5/thread-reference-20170716
touch /data/riotbuild/riotproject/examples/openthread/bin/pkg/samr21-xpro/openthread/.git-downloaded

So, it works for me

jia200x · 2018-10-22T17:58:38Z

makefiles/docker.inc.mk

@@ -83,6 +83,12 @@ DOCKER_OVERRIDE_CMDLINE := $(strip $(DOCKER_OVERRIDE_CMDLINE))
 # Overwrite if you want to use `docker` with sudo
 DOCKER ?= docker

+# Mounted volumes and exported environment variables
+
+# Add GIT_CACHE_DIR if the direcotry exists


typo here:
s/direcotry/directory

jia200x · 2018-10-23T12:32:20Z

ACK on my side. Please squash!

Currently the default value was set by `dist/tools/git/git-cache`. This moves/duplicates the default value in the build system.

If GIT_CACHE_DIR is a directory make it available to docker. This will allow using the system git_cache also in the docker container.

cladmi · 2018-10-23T13:02:57Z

Squashed, I put some more description in the commit messages.
Waiting for murdock.

jia200x · 2018-10-23T16:25:29Z

ok, all good. GO

cladmi · 2018-10-23T16:42:18Z

Thank you for reviewing.

jia200x · 2018-10-23T16:48:16Z

thanks to you for the feature. It saves tons of time

cladmi added Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … labels Jul 31, 2018

cladmi requested review from jnohlgard, jcarrano and kaspar030 July 31, 2018 13:13

cladmi mentioned this pull request Jul 31, 2018

docker: Default settings should provide a volume for git-cache #8267

Closed

jnohlgard reviewed Jul 31, 2018

View reviewed changes

cladmi added the CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR label Aug 2, 2018

cladmi mentioned this pull request Aug 8, 2018

Release 2018.07 - RC2 RIOT-OS/Release-Specs#69

Closed

69 tasks

cladmi added this to the Release 2018.10 milestone Sep 19, 2018

cladmi mentioned this pull request Oct 17, 2018

Update to Ubuntu Bionic RIOT-OS/riotdocker#42

Merged

cladmi force-pushed the pr/make/docker/add_gitcache_support branch from 635c281 to eee0740 Compare October 21, 2018 16:53

jia200x requested changes Oct 22, 2018

View reviewed changes

jia200x approved these changes Oct 23, 2018

View reviewed changes

cladmi added 2 commits October 23, 2018 15:00

makefiles: make the build system aware of GIT_CACHE_DIR

357a4a0

Currently the default value was set by `dist/tools/git/git-cache`. This moves/duplicates the default value in the build system.

docker: Use system GIT_CACHE_DIR if available

c603247

If GIT_CACHE_DIR is a directory make it available to docker. This will allow using the system git_cache also in the docker container.

cladmi force-pushed the pr/make/docker/add_gitcache_support branch from 2be71cd to c603247 Compare October 23, 2018 13:02

jia200x merged commit d386c97 into RIOT-OS:master Oct 23, 2018

cladmi deleted the pr/make/docker/add_gitcache_support branch October 24, 2018 09:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker: Use system GIT_CACHE_DIR if available #9651

docker: Use system GIT_CACHE_DIR if available #9651

cladmi commented Jul 31, 2018 •

edited

Loading

jnohlgard Jul 31, 2018

cladmi Jul 31, 2018

cladmi Jul 31, 2018

cladmi commented Aug 6, 2018

cladmi commented Oct 21, 2018

jia200x commented Oct 22, 2018

jia200x Oct 22, 2018

cladmi Oct 23, 2018

jia200x commented Oct 23, 2018

cladmi commented Oct 23, 2018

jia200x commented Oct 23, 2018

cladmi commented Oct 23, 2018

jia200x commented Oct 23, 2018

docker: Use system GIT_CACHE_DIR if available #9651

docker: Use system GIT_CACHE_DIR if available #9651

Conversation

cladmi commented Jul 31, 2018 • edited Loading

Contribution description

Optional

Testing

Issues/PRs references

jnohlgard Jul 31, 2018

Choose a reason for hiding this comment

cladmi Jul 31, 2018

Choose a reason for hiding this comment

cladmi Jul 31, 2018

Choose a reason for hiding this comment

cladmi commented Aug 6, 2018

cladmi commented Oct 21, 2018

jia200x commented Oct 22, 2018

jia200x Oct 22, 2018

Choose a reason for hiding this comment

cladmi Oct 23, 2018

Choose a reason for hiding this comment

jia200x commented Oct 23, 2018

cladmi commented Oct 23, 2018

jia200x commented Oct 23, 2018

cladmi commented Oct 23, 2018

jia200x commented Oct 23, 2018

cladmi commented Jul 31, 2018 •

edited

Loading