Different build behavior between murdock and riot/riotbuild:latest image

[cladmi]

Description

Compilation behavior is different between murdock the docker image with BUILD_IN_DOCKER=1 even with RIOT_CI_BUILD=1.

It makes things hard to debug and understand the root cause of problems.

Like for the following PRs:

#9362 Warnings only on murdock
#9605 Warnings only on murdock with llvm
#9507 Enough memory on murdock but not in the container

Sub pull requests

• pkg/libfixmath: replace FILE by RIOT_FILE_NOPATH pkg/libfixmath: replace __FILE__ by RIOT_FILE_NOPATH #9643
• makefiles/docker.inc.mk: export RIOT_CI_BUILD to docker makefiles/docker.inc.mk: export RIOT_CI_BUILD to docker #9895
• makefiles/docker.inc.mk: handle building in git worktree makefiles/docker.inc.mk: handle building in git worktree #10303
• makefiles/docker.inc.mk: Use directories in RIOT when possible makefiles/docker.inc.mk: Use directories in RIOT when possible #9646
  • Makefile.include: Fix BUILDRELPATH when RIOTPROJECT is CURDIR. Makefile.include: Fix BUILDRELPATH when RIOTPROJECT is CURDIR. #10302
• tests/*boot: handle building in docker tests/mcuboot: handle building in docker  #11083

Steps to reproduce the issue

To take one as an example:

Building tests/pkg_fatfs_vfs for BOARD=wsn430-v1_4 works in murdock as it is not complaining but this does not

make -C tests/pkg_fatfs_vfs/  BOARD=wsn430-v1_4 clean all RIOT_CI_BUILD=1 BUILD_IN_DOCKER=1 DOCKER="sudo docker"

Expected results

It has the same behavior in docker as in murdock, so either both fail or both succeed.

Actual results

Murdock is ok, but in docker it fails with:

/data/riotbuild/riotproject/tests/pkg_fatfs_vfs/bin/wsn430-v1_4/fatfs.a(ff.o):(.debug_loc+0x1656): relocation truncated to fit: R_MSP430_16_BYTE against `no symbol'
collect2: ld returned 1 exit status
/data/riotbuild/riotbase/Makefile.include:411: recipe for target '/data/riotbuild/riotproject/tests/pkg_fatfs_vfs/bin/wsn430-v1_4/pkg_fatfs_vfs.elf' failed
make: *** [/data/riotbuild/riotproject/tests/pkg_fatfs_vfs/bin/wsn430-v1_4/pkg_fatfs_vfs.elf] Error 1

Versions

Building with docker in master.

[cladmi]

@kaspar030 @smlng I think you are the persons who would know the most about it

[cladmi]

First thing I found, is that RIOT_CI_BUILD is not given to docker:

Without docker: -DRIOT_VERSION=\"buildtest\"
With Docker: -DRIOT_VERSION=\"2018.10-devel-104-g8832-9f7d272ad793\"

When building from a worktree, as the .git is not the real directory I even get:

With Docker: -DRIOT_VERSION=\"'UNKNOWN (builddir: /data/riotbuild/riotbase)'\"

So maybe the RIOT_VERSION should be parsed by the main repo and passed to docker.

[cladmi]

Other issue, RIOT_FILE_RELATIVE is not really relative in docker:

-DRIOT_FILE_RELATIVE=\"/data/riotbuild/riotproject/tests/libfixmath_unittests/bin/pkg/stm32f0discovery/libfixmath/libfixmath/fix16_str.c\"

Outside I get:

-DRIOT_FILE_RELATIVE=\"tests/libfixmath_unittests/bin/pkg/stm32f0discovery/libfixmath/libfixmath/fix16_str.c\"

Which can easily explain build size difference when RIOT_FILE_RELATIVE is used.

[cladmi]

Also if it's possible to have an archive of the currently used image. I could diff with my local docker image and try to find the reason.

[miri64]

At least for the LLVM thing @kaspar030 found a reason: #9398 (comment)

[ZetaR60]

Any updates on this? The new Docker image in RIOT-OS/riotdocker#42 is waiting on this to be resolved.

[miri64]

It appears ccache was also the reason for the erroneous behavior in #9362. Can't say anything specific about RIOT-OS/riotdocker#42 though.

[cladmi]

@ZetaR60 yes we can remove the blocking dependency.

There I have this PR here that handles some of the case for relative directories but it is also currently breaking working things: #9646 I will try to find time to address them this week.

[cladmi]

RIOT_CI_BUILD was not exported to docker which looks like the real reason for wsn430 not building some examples in docker.

[miri64]

The difference between the riot/riotbuild:latest and murdock worker image is now documented in https://github.com/RIOT-OS/riotdocker/tree/master/murdock-worker. Is this sufficient to close this issue?

[kaspar030]

The board doesn't exist anymore, "RIOT_CI_BUILD was not exported to docker which looks like the real reason for wsn430 not building some examples in docker.", all tasks marked done.

Also, the murdock-worker container was always based on riotdocker. If there are differences how murdock builds wrt a local build, that should be independent of Docker. If Docker builds differently than a local build, that should be independent of Murdock.

So let's close this, feel free to re-open if an issue with Murdock vs. Docker remains.