DrMemory 2.4 and 2.5 Unable to load client library on Windows Server 2008 R2 Enterprise

[cedric-ca]

Describe the bug
When I use DrMemory 2.3 it work.
When I upgrade to 2.4 or 2.5, it fails with following message:
Unable to load client library: import MoveFileExW not found in KERNELBASE.dll.
I saw an issue that is now closed, where you had the same problem, but with previous version:
DynamoRIO/dynamorio#2658

To Reproduce
Steps to reproduce the behavior:

1. Pointer to a minimized application (ideally the source code for it and instructions on which toolchain it was built with).
   I didn't try with a simple example, but I think it will fails with the same message.

2. Precise command line for running the application.
   cd /d c:\dev\jenkins\workspace\qlib_diagnostics\Sources\qlibtester\projects & c:\dev\Dr.Memory\bin\drmemory.exe -logdir c:\dev\jenkins\workspace\qlib_diagnostics\qlib_5 -batch -brief -light -suppress c:\svn\Tools\trunk\Scripts..\drmemory-suppress.txt -- c:\dev\jenkins\workspace\qlib_diagnostics\Sources\qlibtester_exe32\qlibtester_vc19_q.exe -split 5/134 -selftest > c:\dev\jenkins\workspace\qlib_diagnostics\drmemory.qlib_5.win32.purify.drmemory.3.txt 2>&1

3. Exact output or incorrect behavior.
   <Application c:\dev\jenkins\workspace\qlib_diagnostics\Sources\qlibtester_exe32\qlibtester_vc19_q.exe (8776). Unable to load client library: import MoveFileExW not found in KERNELBASE.dll.>
   <Application c:\dev\jenkins\workspace\qlib_diagnostics\Sources\qlibtester_exe32\qlibtester_vc19_q.exe (8776). Unable to load client library: drmemorylib.dll: unable to process imports of client library..>
   Dr.M WARNING: unable to locate results file: can't open c:\dev\jenkins\workspace\qlib_diagnostics\qlib_5/resfile.8776 (code=2).
   Dr. Memory failed to start the target application, perhaps due to
   interference from invasive security software.
   Try disabling other software or running in a virtual machine.
   WARNING: Examine the following unusual libraries in this process to help identify
   invasive software that may have affected the target application:

   C:\Windows\system32\api-ms-win-core-localization-l1-2-0.dll
   C:\Windows\system32\api-ms-win-core-file-l1-2-0.dll
   C:\Windows\system32\api-ms-win-crt-time-l1-1-0.dll
   C:\Windows\system32\api-ms-win-crt-runtime-l1-1-0.dll
   C:\Windows\system32\api-ms-win-core-processthreads-l1-1-1.dll
   C:\Windows\system32\api-ms-win-core-timezone-l1-1-0.dll
   C:\Windows\system32\api-ms-win-crt-string-l1-1-0.dll
   C:\Windows\system32\api-ms-win-core-file-l2-1-0.dll
   C:\Windows\system32\api-ms-win-crt-locale-l1-1-0.dll
   C:\Windows\system32\api-ms-win-crt-private-l1-1-0.dll
   C:\Windows\system32\api-ms-win-core-synch-l1-2-0.dll

Please file a bug about this at http://drmemory.org/issues
Dr.M WARNING: application exited with abnormal code 0xffffffff

Please also answer these questions drawn from https://drmemory.org/page_help.html#sec_narrow :

• Does the problem go away when running in light mode (pass -light to Dr. Memory)?
  No
• Does the problem go away when running with the options -leaks_only -no_count_leaks -no_track_allocs?
  Didn't try
• Does the problem go away when running under plain DynamoRIO? Do this by running dynamorio/bin32/drrun -- <application and args> or dynamorio/bin64/drrun -- <application and args> depending on the bitwidth of your applicaiton. (Ignore warnings about "incomplete installation".)
  Yes Application runs well
• What happens with the debug version of Dr. Memory and of its underlying engine DynamoRIO? Try this by passing -debug -dr_debug -pause_at_assert. Are any messages reported?
  Didn't try
  Expected behavior
  DrMemory to run as with version 2.3

Versions

• What version of Dr. Memory are you using?
  DrMemory-Windows-2.4.0-2.msi and DrMemory-Windows-2.5.0.msi

• Does the latest build from
  https://drmemory.org/page_download.html#sec_latest_build solve the problem?
  No with DrMemory-Windows-2.5.18954.msi. DrMemory-Windows-2.3.0-1.msi is the last version where it works

• What operating system version are you running on? ("Windows 10" is not sufficient: give the release number.)
  OS Name: Microsoft Windows Server 2008 R2 Enterprise
  OS Version: 6.1.7601 Service Pack 1 Build 7601
  OS Build Type: Multiprocessor Free
  System Model: Virtual Machine
  System Type: x64-based PC
  Processor(s): 1 Processor(s) Installed.
  [01]: Intel64 Family 6 Model 85 Stepping 4 GenuineIntel ~2095 Mhz
  BIOS Version: American Megatrends Inc. 090006 , 28/04/2016

• Is your application 32-bit or 64-bit?
  32-bit

[derekbruening]

As discussed in DynamoRIO/dynamorio#2658 please find out which DLL on your system exports MoveFileExW.
On Win10 kernel32.dll imports from api-ms-win-core-file-l2-* including for MoveFileExW as shown below.
Is it different on your system?

% dumpbin /imports kernel32.dll
    api-ms-win-core-file-l2-1-2.dll
              6B881114 Import Address Table
              6B8A1688 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    2 CopyFileW
                    4 CreateHardLinkA

    api-ms-win-core-file-l2-1-0.dll
              6B8810DC Import Address Table
              6B8A1650 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    0 CopyFile2
                    D ReadDirectoryChangesW
                    1 CopyFileExW
                    7 GetFileInformationByHandleEx
                    8 MoveFileExW
                    9 MoveFileWithProgressW
                    3 CreateDirectoryExW
                    E ReplaceFileW
                    B ReOpenFile
                    6 CreateSymbolicLinkW
                    5 CreateHardLinkW

    api-ms-win-core-file-l2-1-3.dll
              6B881120 Import Address Table
              6B8A1694 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    C ReadDirectoryChangesExW

    api-ms-win-core-file-l2-1-1.dll
              6B88110C Import Address Table
              6B8A1680 Import Name Table
                     0 time date stamp
                     0 Index of first forwarder reference

                    A OpenFileById

% dumpbin /exports kernelbase.dll | egrep -i 'movefileexw|core-file|createhardlink|copyfile2|readdirectorych|openfilebyid'
        174   A6 001587D0 CopyFile2
        200   C0 0015B940 CreateHardLinkA
        201   C1 0015B9B0 CreateHardLinkW
       1038  40B 0012B100 MoveFileExW
       1066  427 0015AB30 OpenFileById
       1340  53A 001566E0 ReadDirectoryChangesExW
       1341  53B 00156820 ReadDirectoryChangesW

Xref https://groups.google.com/g/drmemory-users/c/V_43cs7Ys4I/m/qICdvPNSAgAJ

Xref https://groups.google.com/g/dynamorio-users/c/2X4SYFrOMgk/m/qFFqLeSNBgAJ

[derekbruening]

If your machine has MoveFileExW only in kernel32.dll and not in kernelbase.dll, again as pointed out in DynamoRIO/dynamorio#2658 if you change this line https://github.com/DynamoRIO/dynamorio/blob/master/core/win32/loader.c#L1640 for API-MS-Win-Core-File-L2-1 to return kernel32.dll does that solve the problem?

[cedric-ca]

Hi Derek,

So I build drmemory-release_2.5.0 from source code with dynamorio-cronbuild-8.0.18915 sources without any changes in the code on my Windows 10 with VS 2017 (doesn't build with VS 2019).

• Generate dynamorio files with cmake
• generate drmemory files with cmake
• Copy dynamorio files in drmemory\builddynamorio dir
• Build drmemory with cmake
• Build the solution with VS 2017

When I launch DrMemory I got a system Error popup with following message:
The program can't start because api-ms-win-downlevel-kernel32-l2-1-0. dll is missing from your computer.

I looked on the net and found out that the first version of api-ms-win-downlevel-kernel32-l2-1-0.dll was released for the Windows 8.1 Operating System on 10/18/2013, while Windows Server 2008 R2 was released on July 22, 2009.

On my Win 10 (WinVer=105;Rel=1909;Build=18363;Edition=Enterprise):
DrMemory 2.5.0 release fails with: Failed to take over all threads after multiple attempts
Home build: Dr. Memory internal crash at PC 0x0573fcf2.

I don't know if I'm doing something wrong.

[derekbruening]

As requested above I was hoping you could provide confirmation that kernel32.dll exports MoveFileExW but kernelbase.dll does not for 2008 R2 as I do not have access to such a machine.

I did get access to some Win7 libraries and we have:

% dumpbin /exports kernelbase.dll/6.1.7601.18229-x64/kernelbase.dll  |  egrep -i 'movefileexw|core-file|createhardlink|copyfile2|readdirectorych|openfilebyid'
% dumpbin /exports kernel32.dll/6.1.7601.18229-x64/kernel32.dll  |  egrep -i 'movefileexw|core-file|createhardlink|copyfile2|readdirectorych|openfilebyid'
        146   91 0005B9D0 CreateHardLinkA = CreateHardLinkA
        147   92 0005BB30 CreateHardLinkTransactedA = CreateHardLinkTransactedA
        148   93 0005BA70 CreateHardLinkTransactedW = CreateHardLinkTransactedW
        149   94 0005B6D0 CreateHardLinkW = CreateHardLinkW
        868  363 00003060 MoveFileExW = MoveFileExW
        891  37A 00050380 OpenFileById = OpenFileById
        964  3C3 00002F80 ReadDirectoryChangesW = ReadDirectoryChangesW

I guess we just assume that matches 2008 R2.

Re: other problems hit in #2440 (comment): I don't know; have not seen the first and the second would need a lot more information. I cannot reproduce them but again have no access to 2008 R2. In any case the import problem can be fixed. I would guess that it was never actually fixed in DynamoRIO/dynamorio#2658 and something else made it not matter.

[derekbruening]

The exports are there in 6.2 == Win8 so it's just Win7:

kernelbase.dll/6.2.9200.16384-x64/kernelbase.dll
        114   71 00083CD0 CopyFile2 = CopyFile2
        133   84 000671EC CreateHardLinkW = CreateHardLinkW
        735  2DE 0002D4F0 MoveFileExW = MoveFileExW
        933  3A4 0001C910 ReadDirectoryChangesW = ReadDirectoryChangesW
kernelbase.dll/6.2.9200.16384-x86/kernelbase.dll
        118   71 00071E0E CopyFile2 = _CopyFile2@12
        137   84 00057AC9 CreateHardLinkW = _CreateHardLinkW@12
        741  2E4 00040CAD MoveFileExW = _MoveFileExW@12
        939  3AA 0003A412 ReadDirectoryChangesW = _ReadDirectoryChangesW@32