bits and bytes

Why?

Well debugging exe is quite easy, but ever wondered how to debug a dll. Well why? Simple, you write a dll and you wanna debug a bug just like you would do with an executable.

Can you give us some examples?

Well so what is a dll? I think of it as a parasite because just like a parasite it’s dependent on an executable to get it running. Some examples follow…

1. A good example to think of will be rundll32.exe. Ever thought about that name “rundll32.exe”, it just means “run a 32 bit dll using this exe”, so how does this achieve this, remember the command line that we pass to it, i.e. name of the dll followed by a procedure name exported from our dll.
2. Another example to think of is regsvr32.exe, does the same stuff i.e. registers a COM dll but this time with a difference, we don’t have to give a procedure name. Because COM programming protocol defines mandatory functions to be exported from a COM dll and regsvr32.exe knows exactly which function to call based on the command line switches passed in.
3. Another example that comes to mind is the windows surrogate process “dllhost.exe”, I’ve heard people say that it’s a virus . Naah it’s a COM surrogate process which helps in hosting a COM dll to make the dll behave like a process hence providing the flexibility to behave like an executabable and at the same time like a dll, vice versa is not possible. Well it can at times be hosting a malicious dll too. It’s quite rare though.

What has this got to do with debugging a DLL?

The reason why I’ve given above examples is to help you understand that a DLL cannot run by itself, it needs a host process to help it run.

Exactly! same for debugging it needs a process to get it running so that execution reaches our breakpoint that we’ve set. Aha! so there must be some option to tell the debugger about the host process. Oh yeah, goto Project settings->Debugging->Executable for Debug Session. Select any process which is using your dll.

Think that you are writing a window’s shell extension, which are dlls, think on how to debug them. So for this case your target process will be explorer.exe. So you will specify explorer.exe as your process. Or imagine writing an activex control, so for this case the target process will be an activex test container.

This is a shot from VC6, it’s quite similar in VC7, VC8 and VC9, instead of tabs there you have a tree menu holding the “Debugging” item right at the top of the tree.

Well now just press F5 and see what happens, you will see your target process running and just try doing a button or menu click that could invoke the function in which you’ve set a breakpoint.  If you are writing a shell extension for e.g. extending shell context menu, then you would right click on a file in the explorer window.

Also note that if your active project is a ‘dll project’ then you’ll get prompted to enter a target process name to get the dll hosted as soon as you press F5. So no need to go to project properties.

Hope this helps you?