Compilation of pe_heuristics.cpp on Ubuntu 16.04 fails with "internal compiler error: Segmentation fault"

[Laerte]

I trying to compile retdec on Ubuntu 16.04, but i get this error after Built target retdec-llvmir2hll

[ 72%] Built target retdec-llvmir2hll
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

I also recorded the compilation on asciinema:
https://asciinema.org/a/Db9sKKSowNTNiG5LPgGDXQ6eM

What should i do? What more info should i provide? Thanks!

[metthal]

The problem seems to be caused by this particular line https://github.com/avast-tl/retdec/blob/master/src/cpdetect/compiler_detector/heuristics/pe_heuristics.cpp#L811. Judging by the error message which can be seen at 8:28, the compiler version you use contains some bug and that line of code I linked triggers internal compiler error.

For now, you can use this workaround which should hopefully work. (Just save it to some file issue_231.patch and then run git apply issue_231.patch or patch -p1 < issue_231.patch.) We'll further investigate what else we can do about it.

diff --git a/src/cpdetect/compiler_detector/heuristics/pe_heuristics.cpp b/src/cpdetect/compiler_detector/heuristics/pe_heuristics.cpp
index 3bd8112..1c80908 100644
--- a/src/cpdetect/compiler_detector/heuristics/pe_heuristics.cpp
+++ b/src/cpdetect/compiler_detector/heuristics/pe_heuristics.cpp
@@ -808,14 +808,13 @@ void PeHeuristics::getNetHeuristic()
 			addPacker(source, strength, "CliSecure");
 		}
 
-		if (std::any_of(dotNetShrinkPatterns.begin(), dotNetShrinkPatterns.end(),
-			[&] (const auto &str)
+		for (const auto& str : dotNetShrinkPatterns)
+		{
+			if (search.findUnslashedSignature(str, start, end))
 			{
-				return this->search.findUnslashedSignature(str, start, end);
+				addPacker(source, strength, ".netshrink", "2.01 (demo)");
+				break;
 			}
-		))
-		{
-			addPacker(source, strength, ".netshrink", "2.01 (demo)");
 		}
 	}
 }

[Laerte]

It's work! Thanks!

[s3rvac]

@Laerte Can you please run gcc --version and post the output here?

[Laerte]

@s3rvac

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

I will try later on vm and then report here, i also have problem with default cmake that official repo of ubuntu install that is version 5.4. So i have to install manually cmake:

wget https://cmake.org/files/v3.6/cmake-3.6.3-Linux-x86_64.sh
chmod +x cmake-3.6.3-Linux-x86_64.sh
./cmake-3.6.3-Linux-x86_64.sh
ln -s /opt/cmake-3.6.3-Linux-x86_64/bin/* /usr/local/bin

[s3rvac]

I was able to reproduce the problem on vanilla Ubuntu 16.04 inside Docker with GCC 5.4.0, which is apparently the latest GCC that Ubuntu 16.04 provides by default.

[Laerte]

@s3rvac So the problem relies on version of gcc?

[s3rvac]

Yes, this appears to be a GCC 5 bug, as I was able to build RetDec on Ubuntu 16.04 by using GCC 6 from this PPA:

apt-get install software-properties-common
add-apt-repository ppa:jonathonf/gcc-6
apt-get update
apt-get install gcc-6 g++-6
# ...
CC=gcc-6 CXX=g++-6 cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/retdec-install

I guess the only thing we can do on our side is to incorporate the patch from @metthal.

[Laerte]

I see, thanks @s3rvac @metthal for quick response!

[s3rvac]

In 5471aee, I have applied the patch by @metthal.