Crash in yara rule matching seemingly due to compiled rule word-size mismatch #48
Comments
|
Funnily enough I am currently looking at either this exact issue or a very similar one... Yesterday @RaduEmanuel92 posted a reply to our previous thread with a deadlock in RtlLookupFunctionEntry which looks to be the same as your (2) above: #12 (comment) I haven't yet replied to this as I thought I would see if I could solve it first. But this issue occurs in standalone mode with a single process and no external yara rules... It looks very much like the old hang but this time the location via yara and subsequent check of LdrpInvertedFunctionTableSRWLock in our_stackwalk appears to be working, so I'm currently not quite sure why it's happening. This leads me to be all the more intrigued by the change in behaviour you are seeing with your change to compiled rules handling. Perhaps there are two issues here. Would it not be straightforward to just delete all the yara rules for your tests? This should leave the internal yara scan in tact but prevent the external rules being loaded, which might in turn allow you to narrow it down a bit further. Thanks for bringing this to my attention. |
|
I saw that comment as well and thought it may also be some form of what I reported as #49. That's why I rushed to get both reports out today. I have disabled compilation of Here's the diff for reference: diff --git a/CAPE/YaraHarness.c b/CAPE/YaraHarness.c
index c7d2e0b..bcfe009 100644
--- a/CAPE/YaraHarness.c
+++ b/CAPE/YaraHarness.c
@@ -345,11 +345,11 @@ BOOL YaraInit()
PathRemoveFileSpec(analyzer_path);
PathRemoveFileSpec(analyzer_path);
sprintf(yara_dir, "%s\\data\\yara", analyzer_path);
-#ifdef _WIN64
- sprintf(compiled_rules, "%s\\capemon64.yac", yara_dir);
-#else
+//#ifdef _WIN64
+// sprintf(compiled_rules, "%s\\capemon64.yac", yara_dir);
+//#else
sprintf(compiled_rules, "%s\\capemon.yac", yara_dir);
-#endif
+//#endif
yr_initialize();
@@ -376,6 +376,7 @@ BOOL YaraInit()
goto exit;
}
+#if 0
char FindString[MAX_PATH];
WIN32_FIND_DATA FindFileData;
sprintf(FindString, "%s\\*.yar", yara_dir);
@@ -422,6 +423,7 @@ BOOL YaraInit()
}
else
DebugOutput("YaraInit: Found no Yara rules in %s\n", yara_dir);
+#endif
// Add 'internal' yara
if (yr_compiler_add_string(Compiler, InternalYara, NULL) != 0) |
|
In order to hopefully be able to investigate further, I need to try and recreate it. I have never seen this exception myself, as far as I know. Would you be able to share the xls file? As these detonations are quite dependent on Office flavours, and I don't have 2013 on my Windows 10 x64 vms, is there any other more straightforward file which triggers this exception? |
|
The XLS is here: https://github.com/scVENUS/PeekabooAV/blob/master/tests/test-data/office/CheckVM.xls I was also just today pondering how to make headway with this issue and will try to come up with a testcase, e.g. a 32 bit |
|
So I've been running below as a c:\windows\syswow64\cmd.exe /k c:\windows\sysnative\cmd.exe /k for /l %%i in (1,1,1000) do echo %%iThere must be something else going on. (note2self: |
|
So I grabbed I'm also sometimes seeing a SIGSEGV happen just a bit earlier in which case the match itself points to inaccessible memory (what!?): The crash happens with a 64 bit yara 4.0.2 cli compiled on Windows 10 as well as a 32 bit and 64 bit yara 4.0.2 cli compiled on Ubuntu 22.04. So the compiled rule file seems to be corrupt somehow. I'm wondering what the best way is to find out in what way exactly it is corrupt. I have been trying to compare the files but since they're binary and contain the rules in highly pre-processed form nothing obvious jumped out right away. Other than the corrupt one being smaller than when I compile the rules myself using m@cape:~/yara$ ./yarac /opt/CAPEv2/analyzer/windows/data/yara/*.yar foo.yac
/opt/CAPEv2/analyzer/windows/data/yara/Guloader.yar(12): warning in rule "Guloader": $antihook is slowing down scanning
/opt/CAPEv2/analyzer/windows/data/yara/Guloader.yar(26): warning in rule "GuloaderB": $antihook is slowing down scanning
m@cape:~/yara$ ls -la ../capemon.yac foo.yac
-rw-rw-r-- 1 m m 30616 Nov 16 2022 ../capemon.yac
-rw-rw-r-- 1 m m 30734 Nov 16 14:30 foo.yacI expect the same to happen if I try to walk the code flow using a debugger since I'd first need to understand how the compiled yara rules and matches actually work. Am I missing some simple and obvious approach? When I divert the rules-saving into bitness-specific files it appears that the m@cape:~/yara$ sha256sum ../capemon* foo.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 ../capemon32.yac
163ad433dcdebce8897abe35fe35bb702882a58b6a2394b34b46ab5a5b79c4a0 ../capemon64.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 ../capemon.yac
c2a9dbe280467c7c73902daf949f21955f2451ed27e3d6b90f93ef030f2f4d31 foo.yac
m@cape:~/yara$ ls -la ../capemon* foo.yac
-rw-rw-r-- 1 m m 30616 Nov 16 2022 ../capemon32.yac
-rw-rw-r-- 1 m m 31872 Nov 16 2022 ../capemon64.yac
-rw-rw-r-- 1 m m 30616 Nov 16 2022 ../capemon.yac
-rw-rw-r-- 1 m m 30734 Nov 16 14:30 foo.yac
|
|
It appears, the 32 bit Windows version of yara generates different output than all the other versions: m@cape:~/yara$ ls -la iys*
-rw-rw-r-- 1 m m 5501 Nov 17 08:15 iys32-nl.yac
-rw-rw-r-- 1 m m 5485 Nov 17 08:35 iys32-windows.yac
-rw-rw-r-- 1 m m 5501 Nov 17 08:36 iys32.yac
-rw-rw-r-- 1 m m 5501 Nov 17 08:35 iys64-windows.yac
-rw-rw-r-- 1 m m 5501 Nov 17 08:37 iys64.yac
-rw-rw-r-- 1 m m 413 Nov 17 08:35 iys.yar
m@cape:~/yara$ sha256sum iys*
fb10b85cb016d27773042a7de3238095529f7cc147255bc51bee6d827d8e730c iys32-nl.yac
d9afe817be52918be8ef1b662e90e970e6674ab76ee475041ad5f47b9132ddf7 iys32-windows.yac
fb10b85cb016d27773042a7de3238095529f7cc147255bc51bee6d827d8e730c iys32.yac
fb10b85cb016d27773042a7de3238095529f7cc147255bc51bee6d827d8e730c iys64-windows.yac
fb10b85cb016d27773042a7de3238095529f7cc147255bc51bee6d827d8e730c iys64.yac
05b39cab9c22be0a7df08f3f7499e979d744d6e63a14780ee9dbee5c850d8703 iys.yar
For validation I compiled all the yara rules including the internal yara scan rule using 32 and 64 bit This produced files identical to what m@cape:~/yara$ ls -la capemon32-windows.yac capemon64-windows.yac ../capemon32.yac ../capemon64.yac
-rw-rw-r-- 1 m m 30616 Nov 17 09:11 capemon32-windows.yac
-rw-rw-r-- 1 m m 30616 Nov 16 22:41 ../capemon32.yac
-rw-rw-r-- 1 m m 31872 Nov 17 09:11 capemon64-windows.yac
-rw-rw-r-- 1 m m 31872 Nov 16 22:56 ../capemon64.yac
m@cape:~/yara$ sha256sum capemon32-windows.yac capemon64-windows.yac ../capemon32.yac ../capemon64.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 capemon32-windows.yac
163ad433dcdebce8897abe35fe35bb702882a58b6a2394b34b46ab5a5b79c4a0 capemon64-windows.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 ../capemon32.yac
163ad433dcdebce8897abe35fe35bb702882a58b6a2394b34b46ab5a5b79c4a0 ../capemon64.yac
This discrepancy also happens with my small I'll validate with the latest version of yara and if it persists I'll look into the cause. BTW: If I try to replace the Do you have a special build procedure or local changes for the yara builds you include in capemon? |
|
Current yara master now consistently produces different files depending on word size of the compiler independent of the operating system but a 64 bit So I'd like to update yara in capemon next but run into above error messages. Can you help? m@cape:~/yara$ ls -la ../capemon32.yac ../capemon64.yac capemon32-windows.yac capemon64-windows.yac capemon32-*master.yac capemon64-*master.yac
-rw-rw-r-- 1 m m 29543 Nov 17 09:31 capemon32-master.yac
-rw-rw-r-- 1 m m 29543 Nov 17 09:25 capemon32-windows-master.yac
-rw-rw-r-- 1 m m 30616 Nov 17 09:11 capemon32-windows.yac
-rw-rw-r-- 1 m m 30616 Nov 16 22:41 ../capemon32.yac
-rw-rw-r-- 1 m m 30571 Nov 17 09:30 capemon64-master.yac
-rw-rw-r-- 1 m m 30571 Nov 17 09:25 capemon64-windows-master.yac
-rw-rw-r-- 1 m m 31872 Nov 17 09:11 capemon64-windows.yac
-rw-rw-r-- 1 m m 31872 Nov 16 22:56 ../capemon64.yac
m@cape:~/yara$ sha256sum ../capemon32.yac ../capemon64.yac capemon32-windows.yac capemon64-windows.yac capemon32-*master.yac capemon64-*master.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 ../capemon32.yac
163ad433dcdebce8897abe35fe35bb702882a58b6a2394b34b46ab5a5b79c4a0 ../capemon64.yac
b5a42ca6e02b23445f2189c87d58745ff68b9fcbe726cd9374a96998dad2a5a0 capemon32-windows.yac
163ad433dcdebce8897abe35fe35bb702882a58b6a2394b34b46ab5a5b79c4a0 capemon64-windows.yac
c006a745feda0bab77b9b37144c383e9c15dbea81a9cbf56b0e5f4aa9278c7e6 capemon32-master.yac
c006a745feda0bab77b9b37144c383e9c15dbea81a9cbf56b0e5f4aa9278c7e6 capemon32-windows-master.yac
123301d86b23ba1225c0660eb95ec091110a4f0f0ff8a066dbcaecc4406e8bc7 capemon64-master.yac
123301d86b23ba1225c0660eb95ec091110a4f0f0ff8a066dbcaecc4406e8bc7 capemon64-windows-master.yac
m@cape:~/yara$ ./yara --version
4.2.1
m@cape:~/yara$ file yara .libs/yara
yara: Bourne-Again shell script, ASCII text executable
.libs/yara: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3fb6fffc3f3c855ddcea6f946b60ccbc94db3037, for GNU/Linux 3.2.0, with debug_info, not stripped
m@cape:~/yara$ ./yara -m -C capemon64-windows-master.yac capemon32-windows.yac
Guloader [author="kevoreilly",description="Guloader bypass",cape_options="bp0=$trap0,bp0=$trap1+4,action0=skip,bp1=$trap2+11,bp1=$trap3+19,action1=skip,bp2=$antihook,action2=goto:ntdll::NtAllocateVirtualMemory,count=0"] capemon32-windows.yac
UrsnifV3 [author="kevoreilly",description="Ursnif Config Extraction",cape_options="br0=$crypto32-73,dumpsize=eax,action1=dump:ebx,typestring1=UrsnifV3 Config,bp2=$timing_trap-6,action2=setesi:9,count=1"] capemon32-windows.yac
m@cape:~/yara$ ./yara -m -C capemon32-windows-master.yac capemon32-windows.yac
Guloader [author="kevoreilly",description="Guloader bypass",cape_options="bp0=$trap0,bp0=$trap1+4,action0=skip,bp1=$trap2+11,bp1=$trap3+19,action1=skip,bp2=$antihook,action2=goto:ntdll::NtAllocateVirtualMemory,count=0"] capemon32-windows.yac
UrsnifV3 [author="kevoreilly",description="Ursnif Config Extraction",cape_options="br0=$crypto32-73,dumpsize=eax,action1=dump:ebx,typestring1=UrsnifV3 Config,bp2=$timing_trap-6,action2=setesi:9,count=1"] capemon32-windows.yac
m@cape:~/yara$ git checkout -f v4.0.2
[...]
m@cape:~/yara$ ./configure --enable-debug && make clean && make -j
[...]
m@cape:~/yara$ ./yara -m -C capemon64-windows-master.yac capemon32-windows.yac
rules were compiled with a different version of YARA
m@cape:~/yara$ ./yara -m -C capemon64-windows.yac capemon32-windows.yac
Guloader [author="kevoreilly",description="Guloader bypass",cape_options="bp0=$trap0,bp0=$trap1+4,action0=skip,bp1=$trap2+11,bp1=$trap3+19,action1=skip,bp2=$antihook,action2=goto:ntdll::NtAllocateVirtualMemory,count=0"] capemon32-windows.yac
UrsnifV3 [author="kevoreilly",description="Ursnif Config Extraction",cape_options="br0=$crypto32-73,dumpsize=eax,action1=dump:ebx,typestring1=UrsnifV3 Config,bp2=$timing_trap-6,action2=setesi:9,count=1"] capemon32-windows.yac
m@cape:~/yara$ ./yara -m -C capemon32-windows.yac capemon32-windows.yac
error scanning capemon32-windows.yac: error: 4
m@cape:~/yara$ strace ./yara -m -C capemon32-windows.yac capemon32-windows.yac 2>&1 | grep SEGV
rt_sigaction(SIGSEGV, {sa_handler=0x55c7088ccc70, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fa9236d8520}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fa9236d8520}, NULL, 8) = 0
rt_sigaction(SIGSEGV, {sa_handler=0x55c7088ccc70, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7fa9236d8520}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fa9236d8520}, 8) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fa9236d8520}, NULL, 8) = 0 |
|
Hi Michael, thank you for this thorough investigation. I will have a look at how I compiled yara into capemon previously - I recall it was a bit of a pain, alas I didn't write detailed notes, but hopefully we can get it updated. |
|
Hey Kev, thanks! Just to be clear: I'm by no means certain a newer yara will solve the issue. It's just the next logical thing to try so we don't waste time debugging a problem that has maybe been solved by yara upstream already. And being able to update and debug yara seems a worthwhile goal in itself to me. My test of scanning the yac file with itself is by no means conclusive. In fact I was surprised I was able to reproduce a crash and even the exact same crash by doing so. I read the link errors as libyara using msvcrt functions like read() and strtod() and msvcrt not being linked into capemon.dll. I'm just unsure what the correct solution is with regards to capemon: Avoid/replace the function calls somehow to avoid a dependency on msvcrt, just link it dynamically (maybe causing a need to install the visual c++ runtime redistributable) or link a static version of it (maybe causing symbol conflicts with other modules doing the same). I'm not enough of a Windows programmer to know the best practice here and capemon likely has a very special set of requirements beyond that. |
|
No worries even if the upgrade doesn't solve this, it will be one less chore and excluded from the possibilities. I spent some time on Friday trying to 'quickly' compile updated lib but this failed. I have some vague memories of having to compile the dependencies (OpenSSL and Janssen) separately the first time then manually combining three lib files... I wish I had documented it better but I'll keep plugging away until I get it working and make sure to document it this time! |
|
Looking at https://stackoverflow.com/questions/68330445/linker-can-not-find-imp-clock-and-imp-time64-when-building-a-sample-of-gtsam and recompiling libyara with /MT in line with capemon, I'm now down to these errors: 2>------ Build started: Project: capemon, Configuration: Release Win32 ------
2>LINK : warning LNK4075: ignoring '/INCREMENTAL' due to '/LTCG' specification
2> Creating library C:\Users\u\source\repos\capemon\Release\capemon.lib and object C:\Users\u\source\repos\capemon\Release\capemon.exp
2>LINK : warning LNK4098: defaultlib 'MSVCRT' conflicts with use of other libs; use /NODEFAULTLIB:library
2>libyara32.lib(value.obj) : error LNK2001: unresolved external symbol __imp___dclass
2>libyara32.lib(strconv.obj) : error LNK2001: unresolved external symbol __imp__strtod
2>MSVCRT.lib(chandler4gs.obj) : error LNK2001: unresolved external symbol __except_handler4_common
2>C:\Users\u\source\repos\capemon\Release\capemon.dll : fatal error LNK1120: 3 unresolved externals
2>Done building project "capemon.vcxproj" -- FAILED.
========== Build: 1 succeeded, 1 failed, 1 up-to-date, 0 skipped ==========These look they stem from libcrypto or libjansson included in libyara because there's no |
|
I managed to build jannson libs with private runtime which I was able to link into yara HEAD and that again into capemon without errors. Here are my build commands for jansson v2.9 (which is what yara uses regardless of the confusing nuget package version 1.0.1 which might refer to an sover): md build32
cd build32
cmake -G "Visual Studio 17" -A win32 -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded ..
cd ..
md build64
cd build64
cmake -G "Visual Studio 17" -A x64 -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreaded ..
cd ..
cmake --build build32 --config Release
cmake --build build64 --config Release
In a first few quick tests, 64bit |
|
It appears the remaining analysis problems seem to have been due to #52. With this resolved my tests run smoothly now. |
|
But the yara issue persists? |
|
Not at all. That seems to have been cured by the upgrade to yara devel HEAD with libjansson 2.9. |
|
Is there any chance you could share the lib files, or outline the steps you took to get there? I've found compiling this really difficult so any help you can offer would be appreciated. |
|
Hello Kev, happy new year! First off: I have just verified that #52 is not enough to fix the yara rule miscompilation issue. So the yara update is needed as well to make everything work. I have also verified that updating to yara release 4.2.3 is enough to make the issue go away - devel HEAD is not required. I'll try to list all the steps I took to upgrade the libs:
|
|
EDIT - seems I forgot to factor in the break and sleep required to obtain a successful result. Turns out my Cmake was slightly too old for those options. I updated it and it now works. Hopefully I will publish a new monitor with Yara 4.3.2 soon and we can close this issue. Thanks for your help |
|
Not sure if this is the best place for this but I've noticed that if there are any exceptions in capemon arising from within the yara code, there are assertion dialogs produced on the desktop. The setting we need for capemon is for it to die in silence - I've tried googling for the |
Hi,
I'm observing crashes caused by Yara rule matching when analysing program call chains containing both 32 and 64 bit programs on Windows 10. My test case is Microsoft Excel 2013 with a small XLS file containing a macro that does a WMI query. This causes (amongst lots of other things)
svchost.exeandwmiprvse.exeprocesses being spawned which are 64 bit. These I have seen fail in three ways:Access violation when trying to find the SRW lock using InternalYaraScan:
Hang in RtlLookupFunctionEntry as if the SRW lock could not be found using InternalYaraScan (but didn't crash either):
Access violation in caller_dispatch():
On a hunch I had
capemon.dllwrite/load the compiled rules to/from separate files like so:That made these problems go away.
It appears, the initial hooking of the 32 bit
excel.exewrites out rules which confuse a 64 bit yara in programs spawned later.What's peculiar is that the yara docs promise exactly this not to be the case: https://yara.readthedocs.io/en/stable/capi.html#saving-and-retrieving-compiled-rules
What could be the cause for me to see this behaviour?
Thanks!
Michael
The text was updated successfully, but these errors were encountered: