bazel run/test ignore segmentation violations

[andrej-eisfeld]

Description of the bug:

bazel run hides segmentation violations, e.g. it doesn't print them on the commandline. It likely just filters them out for some reason. This way bugs in the codebase might go unnoticed for some time and end up on the main development branch. It makes it also harder to figure out the root cause of crashes

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

main.cc

#include <iostream>
using namespace std;
 
int main()
{
    char* str;
 
    // Stored in read only part of data segment //
    str = "GfG";
 
    // Problem:  trying to modify read only memory //
    *(str + 1) = 'n';
    return 0;
}

BUILD

cc_binary(
     name = "main",
     srcs = ["main.cc"],
)

When we "bazel run" the cc_binary here's what we get:

Running the binary from bazel-bin instead we clearly see the application segfaults (return code 139)

Which operating system are you running Bazel on?

Ubuntu 22.04

What is the output of bazel info release?

release 7.0.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No regression. Apparently this bug exists for years already and hasn't been fixed yet. See #1304

Have you found anything relevant by searching the web?

#1304

Any other information, logs, or outputs that you want to share?

No response

[tjgq]

Bazel isn't technically ignoring anything. All bazel run does is run the binary, propagate its stdout/stderr, and return its exit code. When you invoke the binary in a shell, the Segmentation fault (core dumped) message is printed by the shell, not by the binary, after inspecting the exit code (see https://stackoverflow.com/questions/32564735/what-actually-prints-segmentation-fault).

If the cc_binary exits abnormally, you should still be able to tell by inspecting its exit code (which will also be bazel run's exit code). Would that be sufficient? Otherwise, this would be a feature request for bazel run to implement shell-like exit status handling.

[andrej-eisfeld]

Thanks for the quick response and the shared link. Does bazel change the return code then? I don't understand why I get 255 with bazel run but 139 when I manually run it. If bazel were to return 139 then the shell would be able to print the error message as well, right?

I also found this regarding return codes in general

[andrej-eisfeld]

Or maybe bazel catches the SIGSEGV signal and/or doesn't forward it? Or a hardcoded exit(-1) in case of an abnormal exit of the application?

[tjgq]

Actually, ignore what I said above. I wrongly assumed bazel run executed the binary as a child process and waited for it to terminate. But actually, it execs the binary. So there's no exit code forwarding; bazel run simply becomes the binary to be run (technically, /bin/bash -c /path/to/binary, but that doesn't matter since the inner shell also execs). Thus the abnormal termination detection by the outer shell should work.

In addition, I can't reproduce this issue when running Bazel 7.0.2 directly; I can only reproduce it when running it through Bazelisk. Are you also running Bazelisk?

[tjgq]

For Bazelisk, it makes total sense that the abnormal termination wouldn't be detected by the shell: Bazelisk does in fact run Bazel as a child process and waits for it to terminate, then forwards its error code to the shell invoking Bazelisk. But the shell looks at the full process termination status (see wait(2) manpage), not just the exit code, so from its point of view Bazelisk terminated normally.

I suspect the 139->255 exit code transformation has to do with how WaitStatus.ExitStatus() works in Go for an abnormal process termination (that's how Bazelisk determines which code to exit with).

So the solution here would be to either have Bazelisk exec Bazel (which is going to be awkward, since exec doesn't exist on Windows), or have Bazelisk detect the abnormal termination and print the message itself.

[andrej-eisfeld]

wow thanks for the in-depth analysis. Indeed I am using bazelisk to ease version updates. It's been quite some time since I switched to bazelisk so I already forgot that I am not using bazel natively :)
I guess a patched version of bazelisk with exec bazel should work in my case (or go back to native bazel)

thank you

[fmeum]

@andrej-eisfeld It would make sense to file an issue on the Bazelisk repo. Bazelisk is supposed to be an "install once and forget" wrapper around Bazel, so discrepancies like this could very well be considered a bug.