Oct 142014
 

Introduction to MiniDumpWriteDump

In this post, let me show you some sample code to create a full memory dump of any given process using the Windows API MiniDumpWriteDump.

Creating Full Memory Dumps using MiniDumpWriteDump

Here’s some sample code to create a full memory dump of any given process. Just call WriteFullDump(hProcessHandle). This process handle must have PROCESS_QUERY_INFORMATION and PROCESS_VM_READ access to the process. If handle information is to be collected then PROCESS_DUP_HANDLE access is also required.

Please test out before production use. Just some quick code that I was writing up for a customer.

#include <tchar.h>
#include <windows.h>
#include <DbgHelp.h>

using namespace std;

#pragma comment (lib, "dbghelp.lib")

void WriteFullDump(HANDLE hProc)
{
   const DWORD Flags = MiniDumpWithFullMemory |
   MiniDumpWithFullMemoryInfo |
   MiniDumpWithHandleData |
   MiniDumpWithUnloadedModules |
   MiniDumpWithThreadInfo;

   HANDLE hFile = CreateFile(_T("F:\\main.dmp"), GENERIC_ALL, 0, nullptr, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr);
   if (!hFile)
   {
     std::cerr << _T("Failed to write dump: Invalid dump file");
   }
   else
   {
     BOOL Result = MiniDumpWriteDump( hProc,
                                      GetProcessId(hProc),
                                      hFile,
                                      (MINIDUMP_TYPE)Flags,
                                      nullptr,
                                      nullptr,
                                      nullptr );

     CloseHandle(hFile);

     if (!Result)
     {
        std::cerr << _T("Looks like an error: MiniDumpWriteDump failed") ;
     }
   }// End if

   return;
}

Significance of MiniDumpWriteDump flags

What gets into a process’ memory dump is determined by the flags we pass in to MiniDumpWriteDump API. The code that I’ve given produces the biggest dumps possible for a process. The output dump file in this case will have the entire process’ memory, modules, registers, thread information, unloaded module information, and process handle related information.

For our case we’ve given these five flags, this will just basically copy everything from a process’ virtual address space into a dump. Comment out any flags if you’re not interested in that information, for e.g. if you don’t need information on the handles in a process just comment out: MiniDumpWithHandleData.

MiniDumpWithFullMemory |
MiniDumpWithFullMemoryInfo |
MiniDumpWithHandleData |
MiniDumpWithUnloadedModules |
MiniDumpWithThreadInfo;

References

http://msdn.microsoft.com/en-us/library/windows/desktop/ms680360(v=vs.85).aspx

Sep 102014
 

Issue

Recently a colleague asked this question. They had a customer who was experiencing a heap corruption so as expected we enabled PageHeap but there was a catch. The application had to run for a long time (around 30 days) in order to reproduce the crash and we had no idea what’s causing the crash.

How do we enable PageHeap?

We can enable standard PageHeap using following command run from an admin command prompt: gflags /p /enable ImageFileName
To enable full PageHeap use the following: gflags /p /enable ImageFileName /full 

(MSDN) Use care in interpreting the Enable page heap check box for an image file in the GFlags dialog box. It indicates that page heap verification is enabled for an image file, but it does not indicate whether it is full or standard page heap verification. If the check results from selecting the check box, then full page heap verification is enabled for the image file. However, if the check results from use of the command-line interface, then the check can represent the enabling of either full or standard page heap verification for the image file.

Why application hung?

So customer enabled PageHeap and went home. Came back next day to see that the application has stopped responding and is hung. The application hung apparently after enabling PageHeap and as we know of PageHeap: every allocation is paged to the page file. So guess why would the hang take place? PageFile size!

Resolution

The customer had set PageFile to its default size which apparently was not enough in this case. We suggested to increase the PageFile size and the hang went away. This resolved the issue. Note that if you enable PageHeap and then go home no matter what’s the PageFile size eventually the result will be unpredictable as the PageFile size is finite. You might need to tweak your PageHeap settings and make it per module or non-full standard page heap.

Conclusion

Please note there are different variants of PageHeap. In this case we needed a full PageHeap so please note this will be pretty heavy on the PageFile.

Dec 092013
 

Recently a colleague of mine asked where’s the length of CString string stored in memory. Hmm so lets dig around. Please note I’ve declared the following CString object in my code…

CString TestCString = _T("Nibu is testing CString");

If you dump CString type in the debugger we see following…

0:000> dt TestCString
Local var @ 0xb4fcd4 Type ATL::CStringT<wchar_t,StrTraitMFC_DLL<wchar_t,ATL::ChTraitsCRT<wchar_t> > >
   +0x000 m_pszData        : 0x00dfa2f8  "Nibu is testing CString"

From above dump of type CString we see that CString class defines just one variable: m_pszData. I don’t see a length variable here so where is the length stored for CString string?

Length of a CString string is stored at a negative offset from m_pszData. The data structure that resides at the negative offset is: ATL::CStringData

0:000> dt mfc100ud!ATL::CStringData
   +0x000 pStringMgr       : Ptr32 ATL::IAtlStringMgr
   +0x004 nDataLength      : Int4B
   +0x008 nAllocLength     : Int4B
   +0x00c nRefs            : Int4B

CStringData is retrieved via a call to function: GetData()

CStringData* GetData() const throw()
{
    return( reinterpret_cast< CStringData* >( m_pszData )-1 );
}

The above code is bit of pointer arithmetic, first m_pszData is cast to a pointer to CStringData and then the casted type is deducted by –1 (which will equate to -sizeof(CStringData). So lets see while debugging if we can get to the CStringData located at a negative offset. First lets get the size of ATL::CStringData in memory.

0:045> ?? sizeof(ATL::CStringData)
unsigned int 0x10

Size of ATL::CStringData comes to 0x10 bytes. So in my test application lets find out what is located at a negative offset of 0x10 bytes. In my current frame I’ve the following locals. My CString object is called TestCString, highlighted in bold in the below code snippet.

0:000> dv
           this = 0x00ef6ba8
        cmdInfo = class CCommandLineInfo
       ttParams = class CMFCToolTipInfo
      InitCtrls = struct tagINITCOMMONCONTROLSEX
   pDocTemplate = 0xcccccccc
    TestCString = class ATL::CStringT<wchar_t,StrTraitMFC_DLL<wchar_t,ATL::ChTraitsCRT<wchar_t> > > 
     pMainFrame = 0xcccccccc

Deduction of 0x10 bytes from address of m_pszData (0x00dfa2f8) gives us the address: 00dfa2e8

0:000> ? 0x00dfa2f8-0x10
Evaluate expression: 14656232 = 00dfa2e8

Lets try dumping out CStringData located at the address: 00dfa2e8. See below

0:000> dt 00dfa2e8 TestStack!ATL::CStringData
   +0x000 pStringMgr       : 0x786cb8e4 ATL::IAtlStringMgr
   +0x004 nDataLength      : 0n23
   +0x008 nAllocLength     : 0n23
   +0x00c nRefs            : 0n1

Dump type says, length of string is: 0n23 which is correct. The length of string “Nibu is testing CString” is indeed 23.

Code documentation of CStringData says this about its member variables…

struct CStringData
{
    IAtlStringMgr* pStringMgr;  // String manager for this CStringData
    int nDataLength;  // Length of currently used data in XCHARs (not including terminating null)
    int nAllocLength;  // Length of allocated data in XCHARs (not including terminating null)
    long nRefs;     // Reference count: negative == locked
    // XCHAR data[nAllocLength+1]  // A CStringData is always followed in memory by the actual array of character data

Difference between nDataLength and nAllocLength is quite evident from the above documentation. Hope this helps.

Aug 232013
 

While stepping through disassembly code you might have wondered if there is a way to jump directly to the next branching statement or the next call or the next return statement instruction. The answer is: Yes there are some very useful ones, the following table of commands is taken from WinDbg documentation.

p (Step)

clip_image001[4]

Debug | Step Over

F10

Target executes one instruction. If this instruction is a function call, that function is executed as a single step.

pa (Step to Address)

     

Target executes until it reaches the specified address. All steps in this function are displayed (but steps in called functions are not).

pc (Step to Next Call)

     

Target executes until the next call instruction. If the current instruction is a call instruction, this call is executed completely and execution continues until the next call.

pct (Step to Next Call or Return)

     

Target executes until it reaches a call instruction or a return instruction.

ph (Step to Next Branching Instruction)

     

Target executes until it reaches any kind of branching instruction, including conditional or unconditional branches, calls, returns, and system calls.

pt (Step to Next Return)

     

Target executes until it reaches a return instruction.

t (Trace)

clip_image002[4]

Debug | Step Into

F11

F8

Target executes one instruction. If this instruction is a function call, debugger traces into that call.

ta (Trace to Address)

     

Target executes until it reaches the specified address. All steps in this function and called functions are displayed.

tb (Trace to Next Branch)

     

(All modes, except kernel mode, only on x86-based systems) Target executes until it reaches the next branch instruction.

tc (Trace to Next Call)

     

Target executes until the next call instruction. If the current instruction is a call instruction, the instruction is traced into until a new call is reached.

tct (Trace to Next Call or Return)

     

Target executes until it reaches a call instruction or return instruction. If the current instruction is a call instruction or return instruction, the instruction is traced into until a new call or return is reached.

th (Trace to Next Branching Instruction)

     

Target executes until it reaches any kind of branching instruction, including conditional or unconditional branches, calls, returns, and system calls. If the current instruction is a branching instruction, the instruction is traced into until a new branching instruction is reached.

tt (Trace to Next Return)

     

Target executes until it reaches a return instruction. If the current instruction is a return instruction, the instruction is traced into until a new return is reached.

wt (Trace and Watch Data)

     

Target executes until the completion of the whole specified function. Statistics are then displayed.

Apr 272013
 

What’s NonInvasive debugging?

Non-Invasive debugging is a useful technique to debug hung processes. When NonInvasive debugging is going on the debugger suspends all threads in the process and has access to all threads, memory and register’s of the process. The debugger when NonInvasive debugging is in progress cannot modify process memory, cannot instruct the process to run as well.

NonInvasive debugging via WinDbg

To do non-invasive debugging via windbg/cdb check this link out:
http://msdn.microsoft.com/en-in/library/windows/hardware/ff552274(v=vs.85).aspx

Here let me show you how to do NonInvasive debugging via WinDbg UI.

Open WinDbg, press F6 or File->Attach to a Process. Please make sure you check “Noninvasive” check box in the “Attach to Process” dialog.

Enable NonInvasive debugging

While non-invasive debugging is in progress we can have another instance of the debugger attached to the debuggee. This proves that when non-invasive debugging is in progress the debugger is not attached to the debuggee. Note on Windows only one debugger can be attached at any time to a process. Also while debugging non-invasively common windbg commands like ‘g’ won’t work because this debugger is not attached to the debuggee invasively hence cannot instruct the process to resume execution. A debugger invasively attached to a debuggee manipulates it via a thread created in the remote process.

Thread manipulation during NonInvasive debugging

As written already: for non-invasive debugging the debugger suspends all the threads in the process so this also means that we can resume execution of these threads too. The command to do this is as follows…

0:000> ~*m

~m resumes a thread while ~n suspends a thread. If we don’t resume the threads we won’t see the process UI as the UI thread is also in a suspended state.

Viewing debuggee process memory during NonInvasive debugging

Now with two debuggers monitoring the process we can view the process’ memory via the non-invasive debugger as well. For e.g. when you set a breakpoint via the second debugger  (attached invasively) its interesting to see how the function code is modified by the debugger to get the breakpoint to work, see below e.g.

This is how code for ntdll!ntopenfile will look like before a breakpoint is set…

0:001> uf ntdll!ntopenfile
ntdll!ZwOpenFile:
000007f9`25972f10 4c8bd1          mov     r10,rcx           <<<<---- This three byte instruction is replaced, see below
000007f9`25972f13 b831000000      mov     eax,31h
000007f9`25972f18 0f05            syscall
000007f9`25972f1a c3              ret

This is how code for ntdll!ntopenfile will look like after a breakpoint is set…

0:000> uf ntdll!ntopenfile
ntdll!ZwOpenFile:
000007f9`25972f10 cc              int     3  <<<<------- Single byte instruction cc, and followed by
000007f9`25972f11 8bd1            mov     edx,ecx <<--- 8bd1: remaining two bytes of the above three bytes instruction: 4c8bd1. 4c replaced by cc.
000007f9`25972f13 b831000000      mov     eax,31h
000007f9`25972f18 0f05            syscall
000007f9`25972f1a c3              ret

In effect the original three byte instruction (4c8bd1) is replaced by (cc8bd1). The only change: 4c –> cc. cc evaluates to int 3. When the breakpoint is hit (or when we press Ctrl + Break) the breakpoint instruction is replaced by original op code i.e. 4c8bd1.

We could figure this out via the non-Invasive debugger. If a process is hung we can in effect go through the call stacks and find out potential hang scenarios, for e.g. a process waiting on a network drive.

Apr 102012
 

kf is a useful command to find out stack memory taken by a frame. See below…
I have three functions which looks like this…

#pragma auto_inline(off)
void TestStack2()
{
       printf("hello");
       return;
}
void TestStack1()
{
       TestStack2();
       char bytes[0x190] = {9};
       printf("hello: %s", bytes);
}
void TestStack()
{
       TestStack1();
       char bytes[0x90] = {9};
       printf("hello: %s", bytes);
}

// Check out the frame sizes…
  Memory  ChildEBP RetAddr 
          0024f000 00291578 TestMFC1!TestStack2+0x5
      19c 0024f19c 002915d8 TestMFC1!TestStack1+0x18
       9c 0024f238 002916ea TestMFC1!TestStack+0x18
       28 0024f260 7856f282 TestMFC1!CTestMFC1Dlg::OnInitDialog+0xca
        8 0024f268 752c62fa mfc100!AfxDlgProc+0x31
       2c 0024f294 752ef9df USER32!InternalCallWinProc+0x23
       7c 0024f310 752ef784 USER32!UserCallDlgProcCheckWow+0xd7
       <snip…>

Alternatively we can take difference of child ebp and current esp to know frame size.

Mar 062012
 

Why should we force symbol loading in Windbg

Sometimes we could have a dump which does not load .pdb files even though they are present in the dump folder. The reason for the load failure is not necessarily every time a code change but could be just a rebuild of the source code. In such cases if you force load the .pdb file you should get a call stack that makes sense but you got to be good at API’s and libraries to make sure the stack makes sense. So until you get a proper .pdb file you can force load a .pdb file and work on the dump.
——————————————————-
Remember: Always insist on correct pdb file.
——————————————————-
So the command to enable this feature is: ‘.symopt’. Lists out the current symbol loading options. On my machine this is what I get…

0:000> .symopt
Symbol options are 0x30377:
0x00000001 – SYMOPT_CASE_INSENSITIVE
0x00000002 – SYMOPT_UNDNAME
0x00000004 – SYMOPT_DEFERRED_LOADS
0x00000010 – SYMOPT_LOAD_LINES
0x00000020 – SYMOPT_OMAP_FIND_NEAREST
0x00000100 – SYMOPT_NO_UNQUALIFIED_LOADS
0x00000200 – SYMOPT_FAIL_CRITICAL_ERRORS
0x00010000 – SYMOPT_AUTO_PUBLICS
0x00020000 – SYMOPT_NO_IMAGE_SEARCH

These flags determine how and what symbols will be loaded. These options also determine whether line number information should be loaded or not.

How to force load debugging symbols

So in our debugging scenario if we want to load symbols in a loose manner, i.e., without strict mapping of .pdb with .exe we will have to enable the following option…

0x00000040 – SYMOPT_LOAD_ANYTHING

In windbg we do this via…

0:000> .symopt+ 0x40
Symbol options are 0x30377:
0x00000001 – SYMOPT_CASE_INSENSITIVE
0x00000002 – SYMOPT_UNDNAME
0x00000004 – SYMOPT_DEFERRED_LOADS
0x00000010 – SYMOPT_LOAD_LINES
0x00000020 – SYMOPT_OMAP_FIND_NEAREST
0x00000040 – SYMOPT_LOAD_ANYTHING <———– Prevents validation of .pdb file
0x00000100 – SYMOPT_NO_UNQUALIFIED_LOADS
0x00000200 – SYMOPT_FAIL_CRITICAL_ERRORS
0x00010000 – SYMOPT_AUTO_PUBLICS
0x00020000 – SYMOPT_NO_IMAGE_SEARCH

To re-enable strict mapping between .exe and .pdb use

0:000> .symopt- 0x40
Symbol options are 0x30377:
0x00000001 – SYMOPT_CASE_INSENSITIVE
0x00000002 – SYMOPT_UNDNAME
0x00000004 – SYMOPT_DEFERRED_LOADS
0x00000010 – SYMOPT_LOAD_LINES
0x00000020 – SYMOPT_OMAP_FIND_NEAREST
0x00000100 – SYMOPT_NO_UNQUALIFIED_LOADS
0x00000200 – SYMOPT_FAIL_CRITICAL_ERRORS
0x00010000 – SYMOPT_AUTO_PUBLICS
0x00020000 – SYMOPT_NO_IMAGE_SEARCH

Note the +/- in the above command. ‘+’ enables, ‘-‘ disables.

Alternative way

Another way or maybe a better way to do this is to do this as and when necessary i.e. via .reload command. So if you see a PDB file not loading up due to a mismatch you can just use .reload and ask the debugger to load up the symbols even when they mismatch. This is how we do it.

Following example shows how to load a mismatched PDB/symbol file for test.exe

.reload /f /i test.exe

The /i in above command tells the debugger to ignore any symbol mismatch and just load up the PDB/Symbol file.

Feb 272012
 

Use ‘lml’ to list all dlls whose symbols has been loaded/failed to load, the list will also include dlls which failed symbol loading. See sample…

0:000> lml
start end module name
00000000`03d90000 00000000`040e3000 Test1 T (no symbols
00000000`77d40000 00000000`77eb3000 kernel32 (private pdb symbols) c:\sym\kernel32.pdb\F0EC676938D745549823C7204D03B07B2\kernel32.pdb
00000000`77ec0000 00000000`77ffc000 ntdll (private pdb symbols) c:\sym\ntdll.pdb\C5666A2C21444EFAA53EB4F1CFBE56D22\ntdll.pdb
00000001`55600000 00000001`55801000 Test2 (export symbols) Test2.dll
00000001`80000000 00000001`806c2000 Test3 T (no symbols
000007ff`57040000 000007ff`57071000 iphlpapi (private pdb symbols) c:\sym\iphlpapi.pdb\487BEF7A066A4E5DB7C0230E9B8564CA2\iphlpapi.pdb
000007ff`57140000 000007ff`573c5000 ole32 (private pdb symbols) c:\sym\ole32.pdb\7DDC15A822B0415CAD8C3BC2BF86C3082\ole32.pdb
000007ff`77310000 000007ff`77340000 ws2_32 (private pdb symbols) c:\sym\ws2_32.pdb\89321AB9C6CD443FB74F07BA57B507452\ws2_32.pdb
000007ff`7fd30000 000007ff`7fed0000 rpcrt4 (private pdb symbols) c:\sym\rpcrt4.pdb\8852BC6255D84F43A64A5FA4BE7D74162\rpcrt4.pdb

I’ve highlighted the dlls which failed symbol loading.

Use ‘lme’ if you want to list only those dlls which failed symbol loading. See sample…
0:000> lme
start end module name
00000000`03d90000 00000000`040e3000 Test1 T (no symbols
00000001`55600000 00000001`55801000 Test2 (export symbols) Test2.dll
00000001`80000000 00000001`806c2000 Test3 T (no symbols)

Feb 242012
 

-> Please note for demo purpose we are using current thread stack range as address range: poi(@$teb+8) poi(@$teb+4) <-

Search for an ascii string beginning with "Rtl"
s -a poi(@$teb+8) poi(@$teb+4) "Rtl"
//Output
0fd3d906 52 74 6c 47 65 74 50 72-6f 64 75 63 74 49 6e 66 RtlGetProductInf

Search for a unicode string "AgentService"
s -u poi(@$teb+8) poi(@$teb+4) "AgentService"
//Output
0fd3ed7c 0041 0067 0065 006e 0074 0053 0065 0072 A.g.e.n.t.S.e.r.
0fd3edec 0041 0067 0065 006e 0074 0053 0065 0072 A.g.e.n.t.S.e.r.


Display all ascii strings which are at least 8 in length

s -[l8]sa poi(@$teb+8) poi(@$teb+4)
// Output
0fd3d0d4 "mscorlib.pdb"
0fd3d906 "RtlGetProductInfo"


Display all unicode strings which are at least 58 in length

s -[l58]su poi(@$teb+8) poi(@$teb+4)
// Output
0fd3bc08 "謬矐뻬࿓ilC:\Windows\WinSxS\x86_micr"
0fd3bc48 "osoft.vc80.crt_1fc8b3b9a1e18e3b_"
0fd3bc88 "8.0.50727.6195_none_d09154e04427"
0fd3bcc8 "2b9a"

Jan 062012
 

You have a managed application crash dump and you would like to load sos.dll, to use the powerful commands it provides to help with managed debugging, but the load of sos.dll always fails. The command that you are executing for loading sos.dll is… 0:015> .loadby sos clr Unable to find module ‘clr’ On enter you see the above error and you are not sure what is going on. So in order to understand the error message it is important to understand what .loadby command does. “.loadby sos clr” means load sos.dll from where clr.dll is loaded. So as you might have guessed by now if clr.dll is not loaded then sos.dll cannot be found. This is what the error means when it says “Unable to find module ‘clr’”. All it means is clr.dll is not in the loaded module list which means there is no path to locate clr.dll. So how can I fix this error? In order to fix this error you must understand that clr.dll has been introduced into .net applications from .net 4.0 and prior to that we used to have mscorwks.dll. So if your .net application is not a 4.0 app then clr.dll won’t be loaded, yes you will have to use mscorwks.dll instead… 0:015> .loadby sos mscorwks But you are damn sure that the application that crashed is a .net 4.0 application but you still see the error! All I can say is clr.dll is not loaded. Wait for it to load or break on its load and then execute .loadby sos clr. So how to break on a module load… 0:015> sxe ld clr This will for sure help!