How to debug a 16-bit Windows program running in OTVDM? #1000
Comments
|
I ran the program in Win98 VM and it worked well, so I know there is something wrong with OTVDM. By static disassembling, I know the error happened after: CREATEBITMAP --> CREATECOMPATIBLEDC --> SELECTOBJECT --> DELETEDC. But without the ability to debug real-time, I have no idea what exactly went wrong. Could somebody help? |
|
I use windbg to debug winevdm itself and watcom wdw for programs running in it. You really should run a trace though, see #226 (comment) . |
|
Thanks! That really helps. It is probably worth putting that info into wiki, so that more people know the steps. |
|
Maybe not a bug of OTVDM itself? It seems the failure is either the "GLOBALALLOC" or the "DOSVM_Int21Handler" before that. 0df4:0c28:trace:selector:SELECTOR_FreeBlock 170: (11bf,1) |
|
Wait, even though it says "GLOBALALLOC() retval=none", but AX=1476 has a value? Maybe I misunderstood this "retval"? |
|
Retval is none in this case because the function takes a full register context so the value in AX is correct, see Line 747 in 705e733 |
|
By the way, does OTVDM call GLOBALFLAGS() constantly? I've seen millions of such calls over and over again, as if enumerating every possible handle. 0c28:Call KERNEL.22: GLOBALFLAGS(0006) ret=14af:01aa ds=14df |
|
In win3 global handles really are global so that would enum all the global handles on the system. In winevdm (and ntvdm), it'll only get the global handles in the current vm. The program would have to be disassembled to know why. It could be a winevdm bug causing it to do that but I don't know. |
|
Thanks for your help so far! How can I see and break on the program's code inside OpenWatcom Debugger? I can see the execution is always inside vm86.dll, but it seems it never reaches the 16-bit program under debug. |
|
You need to use the win16 wdw. |
|
I see. I have to install win16 wdw as if my host OS were win16. Even the OpenWatcom installer has to be emulated...... It never occurred to me. I thought the 16-bit installer wouldn't run. |
|
With wdw16, I can't even reach the original error. It seems exception is generated in ntdll.dll. version: 2117 17: vm86.dll!disassemble_debug+0x48e - 0x73da3610 0x73da3a9e (null):0 address=778256E8 Interrupt 0D #GP (1597:07DD) flags 3246 err 1D40 |
|
I managed to run wdw16 inside Win98 VM, and found the program actaully sends out tons of GLOBALFLAGS(). The program enumerates all handles between 0 - 0x8000, in a loop of GLOBALFLAGS() -> GLOBALSIZE() -> GLOBALPAGELOCK() -> GLOBALLOCK() -> GLOBALUNLOCK() -> GLOBALPAGEUNLOCK(). If any of them can proceed to GLOBALPAGELOCK() retval = NON_ZERO, it goes into the GOOD route. It seems the problem is at GLOBALSIZE(). None of the global handles has a size greater or equal than 0x004B0000 (4MB), so it never even reaches GLOBALPAGELOCK(). In Win98 VM, we do have 0c28:Call KERNEL.22: GLOBALFLAGS(1006) ret=14af:01aa ds=14df |
|
Does this work in ntvdm? It's looking for a >64K global handle that's arg_8 in size. I suspect that it's win95 specific behavior that ntvdm doesn't replicate. |
|
Unfortunately I can't run NTVDM. I am on Win8.1 x64.
|
|
It's easy to create an arbitrary large ghandle but without knowing why it's searching for a particular one rather than just allocating it itself I'm not optimistic that it'll fix it. |
|
This is weird. The program does not call GLOBALALLOC() itself, but tries to find existed global handle that's bigger than 4MB. Thanks anyway! |
|
I traced a bit more in Win98 VM, and found the program actually calls GLOBALALLOC() not far before that with a size request of 0x004B0000! It is calculating the display buffer(?) of 1280 x 960 x 32-bit color = 0x0012C000 x 4 bytes = 0x004B0000. It seems we only get 8-bit color (1 byte) in OTVDM? |
|
The software isn't set for 256 color compat mode is it? Can you zip and post the whole trace? |
|
Sorry, scratch that. That's because I wasn't able to run it, so I set the compatibility mode to 8-bit color for another try. After I removed that compatibility mode, it is allocating 4MB display buffer. |
|
Winevdm returns 0x4000 for freed handles which is wrong but fixing it won't help here because it's actually freeing the handle. What happens there in win98? If it frees the handle but still finds it I don't know the correct way to handle that. |
|
In Win98 VM, GLOBALALLOC() of 4MB succeeds, then it calls GLOBALLOCK(), and goes on to call GLOBALUNLOCK(), and finally calls GLOBALFREE().
FYI: 0x4000 == GMEM_DISCARDED. I am confused by this behavior. In OTVDM, GLOBALLOCK() succeeds, |
|
Ignore the upper 16 bits for the return from globalunlock. There are programs that expect the handle to be there but the lower 16 bits being 0 means it succeeded. Wait, it's calling createbitmap(1280, 960, 1, 32). I bet it expects the bitmap to be placed into a global handle which would explains why later it's searches for a handle that size. This could be emulated by using createdibsection with global handle that points to the dib. |
|
Another strange thing (not sure if related): |
|
I don't think the CREATEBITMAP() is related. Because GLOBALUNLOCK() was called before it and succeeded, which means you don't have its actual memory address any more. I think the key is that GLOBALFREE() succeeded in OTVDM;
|
|
Is the globallock return you are looking at eax or dx:ax? The first isn't correct because win16 doesn't use 32bit registers for any return. |
|
Thanks, I just noticed that a minute ago. GLOBALLOCK() returns EAX=0, EDX=0x****7AFA Then the program saves AX and DX. later GLOBALUNLOCK() returns EAX=0, EDX=0x****7AF6 When it calls CREATEBITMAP(), it pushes 1280, 960, 1, 32, saved DX (0x7AFA), saved AX (0x0), returns a valid AX. Finally GLOBALFREE() returns EAX=0x00100000, EDX=0 (not sure whether this is success or failure) Later GLOBALFLAGS() on this handle (7AF6) returns EAX=0x00010000, EDX=0x004B6D87 and passes the AX != 0x4000 test. So it seems the handle is freed in Win98 but not discarded? Is that the difference we can't meet? |
This can't be quite right because I just tested on win311, win98 and ntvdm and on all of them globalfree freed the block immediately.
You sure about this? Selectors in win98 always have the lowest 3 bits set so need to end with 7 or f. |
|
Oops, it is 7AF7. (I guess I typed it wrong.) Here is the screenshot, running in Win98SE VM. Anyway, I patched the program to skip that GLOBALFREE(), then the program actually runs in OTVDM (meaning later it passed the GLOBALFLAGS() test on that (not freed) handle, but the client region of main window is black, nothing in it. If I copy that patch into Win98, it shows the same thing--black window. (It seems this GLOBALFREE() is still needed?)
|
|
I checked in win311 and it does allocate a global handle for the bitmap when using createbitmap so I'm 99% sure that's what is needed. When you use that patch it finds the buffer rather than the bitmap and so nothing gets copied to the screen. |
|
In Win98, it runs like this: In OTVDM, not a single handle passed the size check. |
|
My theory is that the program uses some hack to locate the screen buffer for fast drawing. It first allocates a global handle exactly the size of the screen, locks it to get its address. It goes on to create a bitmap using this global handle address as the pointer to pixel color initialization values, frees the global handle, then selects this bitmap into DC. (Now the bitmap contains the system tick signature as if it were some color value.) Now it starts to enumerate every possible global handle that is not discarded, also as large as the previously freed global handle. If no handle passes all these tests, it reports "out of memory". This explains why it enumerates all handles and tries to find the tick signature in those handles, because if there is one, that must be the current screen buffer in use. This can also explain why nothing is drawn on screen if I skip that GLOBALFREE, because then graphics is drawn into that global handle, which is not used by Windows as screen buffer (I think), because that handle is still owned by our program, not by Windows. Somehow in Win98, the handle found is exactly the same handle the program freed earlier. (The program freed it, but Windows immediately picked it up and used it as screen buffer.) |
|
Right, that's basically what I was thinking too (other than the timer part which I didn't notice in the picture above). If this is correct then it's effectively doing a createdibsection which didn't exist in win31. Also, that is basically what WinG does, if they had just used it this issue wouldn't exist. |
|
Thank you for your help all the way! |
|
You didn't have to close this, if the theory is correct then it can be made to work without too much difficulty. |
|
Thanks. I'll try and let you know. |
|
Great news! Your method works. However it crashes when I close the program. I will look deeper and try to compare it in Win98 later. |
|
I think I know what was wrong. When the program exits, it shows another "goodbye" window for a few seconds. This "goodbye" window is created in the same hackish way as the main window. (actually using the same code routine). At this point, the original main window and its bitmap should have been destroyed. Crash point:
[bp+6h] stores the return DWORD value (i.e. DX:AX) of GLOBALLOCK(), which should be the memory address of a global handle. It seems GLOBALLOCK() returned an invalid address (seg==0) even though this global handle already passed GLOBALFLAGS() "not discarded" test and GLOBALSIZE() "larger than the screen" test. Not sure if your modification can support more than 1 of such screen buffer created by CREATEBITMAP(). This problematic handle must be the bitmap handle of the original main window, since we never mark it as "discarded", it will be found again, but the selectors are freed areadly at this stage. I think we need to mark the handle "discarded" or change its size to 0, when its selector is freed. (Maybe not at GLOBALFREE(), but at segptr free) |
But the program still finds this global handle "not discarded" and try to load its selector (which is 0) into GS, and crash. I haven't read all code of OTVDM, but I guess arena entry is different from the return value of GLOBALFLAGS()? Or is it because of this bug? |
ayuanx
changed the title
|
If you don't mind me asking, how would you go about bytepatching a variable within the emulated program during runtime? This is for a different program, not the one specified by OP, but since you discussed debugging with wdw at length here I thought I might ask. I found the variable with ghidra and then looked for it with wdw, the variable being at DS:19F7 OFFSET:05C4 (that's the address when debugged with wdw), but I don't know how to translate it to an address within the otvdm.exe process. Is there any example of runtime bytepatching? I tried it with Or is the segment address different than the one reported in wdw? The offset should be correct, ghidra shows an identical one. edit: I can get the module handle's address with edit2: my bad, |
|
Use windbg and break the program then use ".call krnl386!mapsl(segoff)" where segoff is the selector and offset as a single dword. That will give the linear address in the winevdm address space. I don't think wdw can provide selector base addresses. |
I've got "out of memory" error.
There is no output in OTVDM window except "version 2117".
What debugger can I use? Or is there any OTVDM internal debugging mechanism I can use?
The text was updated successfully, but these errors were encountered: