NXCOMPAT and the C# compiler

The C# compiler in Visual Studio 2008 and the .NET 3.5 Framework (csc.exe) is now generating PE files with the NXCOMPAT bit set. What is that bit and who cares, you ask? You may very well care if your application interops with native binaries or exposes a plugin model to 3rd parties. First, some background...

DEP is short for Data Execution Prevention. It is a technology that exists in Microsoft operating systems which prevents execution of code from memory pages which are not marked as executable. DEP exists to reduce the attack surface available to malicious software that is trying to hijack a process, and it has been acknowledged to be very helpful in that regard. In Windows Vista, the set of processes and applications to which DEP is applied is configurable by administrators, but there is also a role for application developers.

In the header of a PE file there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT. This flag affects whether or not the OS enables DEP for a process. Setting this flag tells the OS that the image is compatible with DEP. For executable images, if this flag is set, the process is run with DEP enabled unless the machine is configured with the DEP policy set to AlwaysOff. If the image is a DLL and the flag is set, the OS skips checking the DLL against a compatibility database which results in a small performance improvement. All of this applies to x86, 32-bit processes only. On a 64-bit OS, DEP is always enabled for 64-bit processes, but 32-bit processes are configured by the PE flag and system policy as described above. So how does one control the flag in the PE header?

Since the C# compiler emits PE files which are MSIL only and therefore compatible with DEP, the output binaries from the VS 2008 and .NET 3.5 C# compilers have this flag set. Our expectation is that the vast majority of C# executables produced by these compilers will be part of a DEP-compatible application. For that reason we did not surface a compiler switch to configure the NXCOMPAT setting. Of course you can write a C# application that uses a native or mixed binary which is not compatible with DEP. Some ATL types in 7.1 and earlier used to do simple code generation into data pages which is a DEP no-no. If your application is generating IP_ON_HEAP exceptions, then you may need to clear the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you can use EDITBIN.EXE from the VC toolset like so:

editbin.exe /NXCOMPAT:NO <your binary>

Read more: I'm just sayin'

Posted via email from Jasper-net