Writing a basic Windows debugger

Preamble

All of us have used some kind of debugger while programming in some language. The debugger you used may be in C++, C#, Java or another language. It might be standalone like WinDbg, or inside an IDE like Visual Studio. But have you been inquisitive over how debuggers work?

Well, this article presents the hidden glory on how debuggers work. This article only covers writing debugger on Windows. Please note that here I am concerned only about the debugger and not about compilers, linkers, or debugging extensions. Thus, we'll only debug executables (like WinDbg). This article assumes a basic understanding of multithreading from the reader (read my article on multithreading).

1. How to Debug a Program?

Two steps:

Starting the process with DEBUG_ONLY_THIS_PROCESS or DEBUG_PROCESS flags.

Setting up the debugger's loop that will handle debugging events.

Before we move further, please remember:

Debugger is the process/program which debugs the other process (target-process).

Debuggee is the process being debugged, by the debugger.

Only one debugger can be attached to a debuggee. However, a debugger can debug multiple processes (in separate threads).

Only the thread that created/spawned the debuggee can debug the target-process. Thus, CreateProcess and the debugger-loop must be in the same thread.

When the debugger thread terminates, the debuggee terminates as well. The debugger process may keep running, however.

When the debugger's debugging thread is busy processing a debug event, all threads in the debuggee (target-process) stand suspended. More on this later.

A. Starting the process with the debugging flag

Use CreateProcess to start the process, specifying DEBUG_ONLY_THIS_PROCESS as the sixth parameter (dwCreationFlags). With this flag, we are asking the Windows OS to communicate this thread for all debugging events, including process creation/termination, thread creation/termination, runtime exceptions, and so on. A detailed explanation is given below. Please note that we'll be using DEBUG_ONLY_THIS_PROCESS in this article. It essentially means we want only to debug the process we are creating, and not any child process(es) that may be created by the process we create.

STARTUPINFO si; 

PROCESS_INFORMATION pi; 

ZeroMemory( &si, sizeof(si) ); 

si.cb = sizeof(si); 

ZeroMemory( &pi, sizeof(pi) );

CreateProcess ( ProcessNameToDebug, NULL, NULL, NULL, FALSE, 

                DEBUG_ONLY_THIS_PROCESS, NULL,NULL, &si, &pi );

After this statement, you would see the process in the Task Manager, but the process hasn't started yet. The newly created process is suspended. No, we don't have to call ResumeThread, but write a debugger-loop.

B. The debugger loop

The debugger-loop is the central area for debuggers! The loop runs around the WaitForDebugEvent API. This API takes two parameters: a pointer to the DEBUG_EVENT structure and the DWORD timeout parameter. For timeout, we would simply specify INFINITE. This API exists in kernel32.dll, thus we need not link to any library.

BOOL WaitForDebugEvent(DEBUG_EVENT* lpDebugEvent, DWORD dwMilliseconds);

The DEBUG_EVENT structure contains the debugging event information. It has four members: Debug event code, process-ID, thread-ID, and the event information. As soon as WaitForDebugEvent returns, we process the received debugging event, and then eventually call ContinueDebugEvent. Here is a minimal debugger-loop:

DEBUG_EVENT debug_event = {0};

for(;;)

{

    if (!WaitForDebugEvent(&debug_event, INFINITE))

        return;

    ProcessDebugEvent(&debug_event);  // User-defined function, not API

    ContinueDebugEvent(debug_event.dwProcessId,

                      debug_event.dwThreadId,

                      DBG_CONTINUE);

}

Using the ContinueDebugEvent API, we are asking the OS to continue executing the debuggee. The dwProcessId and dwThreadId specify the process and thread. These values are the same that we received form WaitForDebugEvent. The last parameter specifies if the execution should continue or not. This parameter is relevant only if the exception-event is received. We will cover this later. Until then, we'll utilize only DBG_CONTINUE (another possible value is DBG_EXCEPTION_NOT_HANDLED).

2. Handling debugging events

There are nine different major debugging events, and 20 different sub-events under the exception-event category. I will discuss them, starting from the simplest. Here is the DEBUG_EVENT structure:

struct DEBUG_EVENT

{

    DWORD dwDebugEventCode;

    DWORD dwProcessId;

    DWORD dwThreadId;

    union {

        EXCEPTION_DEBUG_INFO Exception;

        CREATE_THREAD_DEBUG_INFO CreateThread;

        CREATE_PROCESS_DEBUG_INFO CreateProcessInfo;

        EXIT_THREAD_DEBUG_INFO ExitThread;

        EXIT_PROCESS_DEBUG_INFO ExitProcess;

        LOAD_DLL_DEBUG_INFO LoadDll;

        UNLOAD_DLL_DEBUG_INFO UnloadDll;

        OUTPUT_DEBUG_STRING_INFO DebugString;

        RIP_INFO RipInfo;

    } u;

};

WaitForDebugEvent, on successful return, fills-in the values in this structure. dwDebugEventCode specifies which debugging-event has occurred. Depending on the event-code received, one of the members of the union u contains the event information, and we should only use the respective union-member. For example, if the debug event code is OUTPUT_DEBUG_STRING_EVENT, the member OUTPUT_DEBUG_STRING_INFO would be valid.

Read more: Codeproject Part 1, Part 2

QR: 

Posted via email from Jasper-net