Even though you set the target platform on the Setup project to x64 the InstallUtil.lib that get’s run is still x86. In order to have it work property, you need to follow the steps identified here:
http://msdn.microsoft.com/en-us/library/kz0ke5xt.aspx
The section “64-bit managed custom actions throw a System.BadImageFormatException exception” covers the steps you need to follow, using the Orca MSI editor to replace the InstallUtilLib.dll from the one that the Setup Project embeds (x86) to a x64 version.
Now, works like a charm…
Read more: Shawn Cicoria