Bug in bin2llvmir Decoder

[seviezhou]

I try to translate the following PE file:

pe-Windows-x86-cmd.zip

But in the decoder phase, the retdec just gets an error and exits:

The problem is in file src/bin2llvmir/optimizations/decoder/ir_modifications.cpp, function Decoder::canSplitFunctionOn, line 426 and line 445, two portions of code:

...
			auto up = fncStarts.upper_bound(bAddr);
			--up;
			Address bFnc = *up;

...
				auto up = fncStarts.upper_bound(pAddr);
				--up;
				Address pFnc = *up;

...

The problem here is, if up equals to fncStarts.begin(), the --up will crash, the possible fix is:

            auto up = fncStarts.upper_bound(bAddr);
            if (up == fncStarts.begin()) {
                return false;
            }
            --up;
            uint64_t bFnc = *up;

After this fix, the Decoder works well:

[xkubov]

Hi, thank you for the report. Indeed it looks like an issue as the boundaries are not checked in those parts of the code. If you want, you can open a pull request to fix this issue.

[PeterMatula]

• Fixed by Try to fix issue #637 #641.
• Added CHANGELOG.md entry in a894749.
• Since this was not a decompilation quality bug, but a crash bug, I added the binary to the internal nightly tests suite that checks that RetDec does not crash.