Failure cross-compiling for ARM

[fedenusy]

What version of rules_go are you using?

0.20.2

What version of gazelle are you using?

0.19.1

What version of Bazel are you using?

1.2.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

darwin/amd64

Any other potentially useful information about your toolchain?

I adopted the toolchain from Bazel's cc_toolchain test.

Made slight modifications to fit the toolchain into my repo, and to match rules_go cross-compile recommendations. Specifically I:

• Placed the toolchain under //toolchain.
• Simplified armeabi-v7a to arm so that rules_go would pick up the toolchain.

What did you do?

.bazelrc:

build:myconfig --platforms=@io_bazel_rules_go//go/toolchain:linux_arm_cgo
build:myconfig --cpu=arm
build:myconfig --crosstool_top=//toolchain/arm_compiler:toolchain
build:myconfig --host_crosstool_top=@bazel_tools//tools/cpp:toolchain
build:myconfig --action_env=GOARM=7

Then:

> bazel build --config=myconfig //mytarget --sandbox_debug

What did you expect to see?

Cross-compiled binary success.

What did you see instead?

INFO: Analyzed target //mytarget:mytarget (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/external/io_bazel_rules_go/BUILD.bazel:8:1: GoStdlib external/io_bazel_rules_go/linux_arm_stripped/stdlib%/pkg failed (Exit 1) sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/execroot/com_github_softwaremotor_smc && \
  exec env - \
    CC=toolchain/arm_compiler/linaro_linux_gcc/arm-linux-gnueabihf-gcc \
    CGO_CFLAGS='--sysroot=external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc -mfloat-abi=hard -nostdinc -isystem external/org_linaro_components_toolchain_gcc_5_3_1/lib/gcc/arm-linux-gnueabihf/5.3.1/include -isystem external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/include -isystem external/org_linaro_components_toolchain_gcc_5_3_1/lib/gcc/arm-linux-gnueabihf/5.3.1/include-fixed -isystem external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/include -U_FORTIFY_SOURCE -fstack-protector -fPIE -fdiagnostics-color=always -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='--sysroot=external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc -lstdc++ -latomic -lm -lpthread -Ltools/arm_compiler/linaro_linux_gcc/clang_more_libs -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/lib -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/lib -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/lib -Bexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/bin -Wl,--dynamic-linker=/lib/ld-linux-armhf.so.3 -no-canonical-prefixes -pie -Wl,-z,relro,-z,now' \
    GOARCH=arm \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=toolchain/arm_compiler/linaro_linux_gcc:/bin:/usr/bin \
    TMPDIR=/var/folders/v8/pgq2fh216w997cls2brzl2vc0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/sandbox.sb /var/tmp/_bazel_fedenusy/install/3c8c3e061eca79d4c8e8c05bb2875b69/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/go_sdk/builder stdlib -sdk external/go_sdk -installsuffix linux_arm -out bazel-out/arm-fastbuild/bin/external/io_bazel_rules_go/linux_arm_stripped/stdlib%)
# runtime/cgo
/private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/execroot/com_github_softwaremotor_smc/toolchain/arm_compiler/linaro_linux_gcc/arm-linux-gnueabihf-gcc: line 3: bazel-out/arm-fastbuild/bin/external/io_bazel_rules_go/linux_arm_stripped/stdlib%/src/runtime/cgo/external/org_linaro_components_toolchain_gcc_5_3_1/bin/arm-linux-gnueabihf-gcc: No such file or directory
/private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/execroot/com_github_softwaremotor_smc/toolchain/arm_compiler/linaro_linux_gcc/arm-linux-gnueabihf-gcc: line 3: exec: bazel-out/arm-fastbuild/bin/external/io_bazel_rules_go/linux_arm_stripped/stdlib%/src/runtime/cgo/external/org_linaro_components_toolchain_gcc_5_3_1/bin/arm-linux-gnueabihf-gcc: cannot execute: No such file or directory
stdlib: error running subcommand: exit status 2
Target //mytarget:mytarget failed to build
ERROR: /Users/fedenusy/dev/go/src/smc/mytarget/BUILD.bazel:18:1 GoStdlib external/io_bazel_rules_go/linux_arm_stripped/stdlib%/pkg failed (Exit 1) sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/execroot/com_github_softwaremotor_smc && \
  exec env - \
    CC=toolchain/arm_compiler/linaro_linux_gcc/arm-linux-gnueabihf-gcc \
    CGO_CFLAGS='--sysroot=external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc -mfloat-abi=hard -nostdinc -isystem external/org_linaro_components_toolchain_gcc_5_3_1/lib/gcc/arm-linux-gnueabihf/5.3.1/include -isystem external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/include -isystem external/org_linaro_components_toolchain_gcc_5_3_1/lib/gcc/arm-linux-gnueabihf/5.3.1/include-fixed -isystem external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/include -U_FORTIFY_SOURCE -fstack-protector -fPIE -fdiagnostics-color=always -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='--sysroot=external/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc -lstdc++ -latomic -lm -lpthread -Ltools/arm_compiler/linaro_linux_gcc/clang_more_libs -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/lib -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/lib -Lexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/libc/usr/lib -Bexternal/org_linaro_components_toolchain_gcc_5_3_1/arm-linux-gnueabihf/bin -Wl,--dynamic-linker=/lib/ld-linux-armhf.so.3 -no-canonical-prefixes -pie -Wl,-z,relro,-z,now' \
    GOARCH=arm \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=toolchain/arm_compiler/linaro_linux_gcc:/bin:/usr/bin \
    TMPDIR=/var/folders/v8/pgq2fh216w997cls2brzl2vc0000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_fedenusy/82761fbb391b03a602abc1324ff91dda/sandbox/darwin-sandbox/1972/sandbox.sb /var/tmp/_bazel_fedenusy/install/3c8c3e061eca79d4c8e8c05bb2875b69/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/host/bin/external/go_sdk/builder stdlib -sdk external/go_sdk -installsuffix linux_arm -out bazel-out/arm-fastbuild/bin/external/io_bazel_rules_go/linux_arm_stripped/stdlib%)
INFO: Elapsed time: 9.846s, Critical Path: 9.55s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

Looks like rules_go is not playing nicely with the cc_toolchain (gcc is run from the wrong directory?)

Help would be much appreciated!

[yicm]

Tips for Bazel 3.3.1: attributes 'cpu' and 'compiler' have been deprecated, please remove them. See bazelbuild/bazel#7075 for details

Hi, @jayconrod Does rules_go support platform-based construction? This configuration has no effect on Go compilation, only useful for C++ compilation.

If Go is cross-compiled separately, you can use CC to specify the cross-compiler,eg:

$ CC=arm-linux-gnueabihf-gcc GOOS=linux GOARCH=arm GOARM=6 CGO_ENABLED=1 go build -v -o myprogram -ldflags="-extld=$CC".

[jayconrod]

@yicm rules_go does support platform-based construction. The document you linked is out of date, as noted at the top.

Not sure how that go build command is related to this issue. CC is set to a relative path that is not valid in the directory where commands are run.

[yicm]

@jayconrod The question I want to ask is not clearly expressed, so I should make it clear here #2089 (comment), Thank you for your reply.

[kalcutter]

The issue is GoStdLib (go install) runs the C compiler from a directory that is not the execroot. At the same time, rules_go uses the cc_toolchain from bazel. This is just broken behavior. It will not work with any toolchain that uses a cc wrapper script with relative paths (or any toolchain that uses cc flags with relative paths that rules_go doesn't know how to make absolute). I think this should be fixed in rules_go.

Maybe the best way to solve this is to add a cc-wrapper mode to the builder. When invoking go, CC should be set to use this wrapper. The wrapper could change the current directory back to the bazel execroot and transform input/output parameters and possibly any includes added by the go compiler on the fly to absolute paths. This is the opposite of what is attempted currently, but a more robust solution that could support all toolchain implementations and flags.

[bsirang]

I ran into this issue today, and took a path that was inspired by @kalcutter 's advice.

However, rather than changing directory and transforming the input/output parameters, I constructed a relative path to the toolchain binaries that is relative to the path of the wrapper scripts themselves rather than relative to the current working directory.

[kalcutter]

@bsirang Yes that is the best strategy when writing a toolchain. My LLVM toolchain does exactly the same thing: https://github.com/kalcutter/bazel-llvm-toolchain/commit/360c56833e45f39ab08a2a9f638726a95bf404e9.