Cannot do cross-compilation? (--target flag not recognized)

[umbertov]

Hello, I am already using gclang/gclang++ successfully on a project for which LLVM bytecode is extracted, processed and finally lowered to (native) machine code.

I need to perform the exact same task, but in a cross-compilation setting. I expected this to be rather easy since by Clang's design, it is sufficient to pass a --target=my_architecture as a flag when building. As it turns out, gclang does not recognize this flag, and ends up with a bad call to objcopy.

Minimal reproducible example

You just need the simplest possible C program to reproduce this. I wrote the program in the file a.c:

int main() { return 0; }

Then we run clang on a.c and verify that an aarch64 executable is successfully created:

clang --target=aarch64-linux-gnu a.c
file a.out
# outputs:
# a.out: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped

Doing the exact same with gclang results in an error:

gclang --target=aarch64-linux-gnu a.c

I paste the error below:

WARNING:Did not recognize the compiler flag: --target=aarch64-linux-gnu
objcopy: Unable to recognise the format of the input file `.a.c.o'
WARNING:attachBitcodePathToObject: objcopy [--add-section .llvm_bc=/tmp/gllvm2318839526 .a.c.o] failed because exit status 1
/usr/bin/ld: .a.c.o: Relocations in generic ELF (EM: 183)
/usr/bin/ld: .a.c.o: error adding symbols: file in wrong format
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

If i run gclang with the -v flag, i get:

WARNING:Did not recognize the compiler flag: --target=aarch64-linux-gnu
clang version 12.0.1
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /llvm/bin
Found candidate GCC installation: /usr/lib/gcc-cross/aarch64-linux-gnu/8
Selected GCC installation: /usr/lib/gcc-cross/aarch64-linux-gnu/8
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/llvm/bin/clang-12" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name a.c -mrelocation-model static -mframe-pointer=non-leaf -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /llvm/lib/clang/12.0.1 -internal-isystem /usr/local/include -internal-isystem /llvm/lib/clang/12.0.1/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir /sunset/SUNSET_private/platforms/aarch64 -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/a-6bd7ff.o -x c a.c
clang -cc1 version 12.0.1 based upon LLVM 12.0.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /llvm/lib/clang/12.0.1/include
 /usr/include
End of search list.
 "/usr/bin/aarch64-linux-gnu-ld" -EL --hash-style=both --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o a.out /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib/crt1.o /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib/crti.o /usr/lib/gcc-cross/aarch64-linux-gnu/8/crtbegin.o -L/usr/lib/gcc-cross/aarch64-linux-gnu/8 -L/lib/../lib64 -L/usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib -L/usr/lib/gcc-cross/aarch64-linux-gnu/8/../../.. -L/llvm/bin/../lib -L/lib -L/usr/lib /tmp/a-6bd7ff.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc-cross/aarch64-linux-gnu/8/crtend.o /usr/lib/gcc-cross/aarch64-linux-gnu/8/../../../../aarch64-linux-gnu/lib/crtn.o
objcopy: 'a.o': No such file
WARNING:attachBitcodePathToObject: objcopy [--add-section .llvm_bc=/tmp/gllvm1182547957 a.o] failed because exit status 1

Is there an easy user-side fix to this problem? Or does it need a source code modification? This problem is mentioned in #63 , and it is hinted to have a look at parser.go, but the author didn't follow up, and i didn't find relevant pull requests.

[woodruffw]

Or does it need a source code modification?

I suspect it needs a source modification, specifically in Parse:

gllvm/shared/parser.go

Line 159 in 49c0709

func Parse(argList []string) ParserResult {

That function is aware of a bunch of flags; it probably needs to be made aware of --target.

[umbertov]

it probably needs to be made aware of --target

That's what i thought as well, I am unfamiliar with the codebase though, and I am not sure which callback i should assign it to: it might be compileLinkBinaryCallback, but I am not sure about that. Do you have a suggestion? Or perhaps i just go with trial & error?

[woodruffw]

Do you have a suggestion? Or perhaps i just go with trial & error?

I'd trial and error it 🙂 -- compileLinkBinaryCallback sounds correct, but I haven't made any contributions to that bit in a while so I can't state that confidently.

[ianamason]

Yes my guess is that it is both compile and link. If you do fix it, can you please make a pull request?

[umbertov]

Yes, I'm planning to have a look at it today, if I'm successful you'll find a pull request 😄

[umbertov]

EDIT: Just found a workaround, see next message

Hello, I made the modification in this commit. I added the following code:

		"-target": {1, pr.compileLinkBinaryCallback},

Note: the flag is -target here instead of the previously mentioned --target. Clang accepts either one of them, but i had some error when using --target and didn't want to lose focus troubleshooting.

It is almost there, meaning that an aarch64 ELF executable is created, but no .llvm_bc section is added to the executable! This is because the final objcopy still fails.

gclang -target aarch64-linux-gnu /tmp/prova/a.c
# outputs:
objcopy: Unable to recognise the format of the input file `.a.c.o'
WARNING:attachBitcodePathToObject: objcopy [--add-section .llvm_bc=/tmp/gllvm3259057631 .a.c.o] failed because exit status 1

You can see that a warning is printed. The a.out file is however created, and it is a valid aarch64 executable:

file a.out 
a.out: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped

But it won't contain the .llvm_bc section, in fact get-bc fails:

get-bc a.out
ERROR:Error reading the .llvm_bc section of ELF file a.out.

it is verifiable independently via readelf:

readelf -S a.out | grep .llvm_bc # no output

I suspect that the -target flag in parser.go should be marked with the callback objectFileCallback in addition to compileLinkBinaryCallback.
Is there a way to do this callback composition easily?

[umbertov]

Just found out that for some reason, objcopy (GNU toolchain) is the default, not llvm-objcopy from the LLVM toolchain.

In order to get proper cross-compilation to work end-to-end, i need to set the environment variable GLLVM_OBJCOPY to llvm-objcopy.

So this works on my fork:

GLLVM_OBJCOPY=llvm-objcopy gclang -target aarch64-linux-gnu a.c

If for some reason you want to use the GNU objcopy, you'll need to install a cross GCC toolchain and do:

GLLVM_OBJCOPY=aarch64-linux-gnu-objcopy gclang -target aarch64-linux-gnu a.c

I found out that the decision is carried in injectPath() inside the shared/compiler.go file, in lines 211-224:

	if runtime.GOOS == osDARWIN {
		if len(LLVMLd) > 0 {
			attachCmd = LLVMLd
		} else {
			attachCmd = "ld"
		}
		attachCmdArgs = []string{"-r", "-keep_private_externs", objFile, "-sectcreate", DarwinSegmentName, DarwinSectionName, tmpFile.Name(), "-o", objFile}
	} else {
		if len(LLVMObjcopy) > 0 {
			attachCmd = LLVMObjcopy
		} else {
			attachCmd = "objcopy"
		}
		attachCmdArgs = []string{"--add-section", ELFSectionName + "=" + tmpFile.Name(), objFile}
	}

In particular, the culprit seems to be attachCmd = "objcopy" as the fallback case when the OS is not Mac, and LLVMObjcopy is not set, which in turn is set in shared/environment.go and corresponds to the GLLVM_OBJCOPY environment variable.

I may be opinionated here, but i think that you should default to llvm-objcopy instead of objcopy, because the former comes from LLVM toolchain, supports all targets and it is more than reasonable to assume that it is going to be present on the system. The GNU objcopy in turn needs to be built for each specific architecture, so you'd have to install it explicitly along with a cross toolchain, and call the prefixed executable (i.e. aarch64-linux-gnu-objcopy).

[ianamason]

I'll look at this more closely later today and tomorrow. I am fine with the objcopy thing, but the main thing is it will need to be documented in the README.md, especially the environment variable. But thanks for all your effort!

[ianamason]

So it wouldn't hurt to go the extra yards to support the --target=ARCHSTRING would it? That would have to be done
with a pattern like:

	{`^--target=.+$`, flagInfo{0, pr.compileLinkUnaryCallback}},

[umbertov]

Thanks Ian! It is actually my first time writing real Go code, so i didn't quite know what Go's idiomatic way was, I just went with trial & error and found -target to work immediately.

Tomorrow morning I'll get back at the pull request & add this change & remove the llvm-objcopy default.

Perhaps we should also add a couple lines in the README.md mentioning that cross compilation is supported, but needs the GLLVM_OBJCOPY env var as well?

On a side note (again, first time Golang user), i noticed that when i forked the project, the change in URL messed up all the import github.com/SRI-CSL/* statements, and i had to sed 's/SRI-CSL/umbertov/g' all the files in the repo to make go install not complain about missing imports. Is there a handy way to avoid that?

[ianamason]

Yes the url thing is a pain in the ass. I know of no way around it, but my go-fu is on the wain.

Yes mentioning cross-compiling requires GLLVM_OBJCOPY would be good and may save someone else some pain!