Throwing garbage on the sidewalk: The sad history of the rundll32 program

During the development of Windows Vista, the application comaptibility team traced a bunch of issues back to people corrupting the stack by using the rundll32 program to call functions that were not designed to be called by rundll32.

The problems were often subtle. For example, a batch file which used rundll32 incorrectly ended up hanging because the rundll32 process never returned. The misaligned stack resulted in registers being restored from the stack incorrectly, and then the cleanup code inside rundll32 ends up getting confused and wedging itself. The programs got away with it on previous versions of Windows by sheer luck. The version of the compiler used by Windows Vista contains different optimizations, and it ended up arranging stack variables and using registers differently, and what in previous versions of Windows was some corruption that went largely unnoticed became corruption that resulted in the program getting stuck in an infinite loop. Lucky no longer.

I was asked to come up with a solution for this problem, to fix the rundll32 program so it was more resilient to people who used it incorrectly. To fix other people's bugs for them.

Read more: The old new thing
QR:

Posted via email from Jasper-net