Embedding custom actions written in .net may be very tricky. Some setup developers even consider it evil. However I found that with some research and reverse engineering it could be quite simple.
First of all you need InstalUtilLib.dll it’s a library which contains ManagedInstall procedure. This procedure allows you to call an assembly consisting of Installer inherited class. In .net 2.0 it is found somewhere near %WINDIR%\Microsoft.NET\Framework\v2.0.50727 . You have to embed it in Binary table of MSI file.
You will also need MSVBDPCADLL. This library setups runtime environment for .net assembly. I’ve seen some installer using .net custom actions which run without this library, hovewer for me it turned out that this library is needed. The easiest way to obtain this library is either decompile a vs made setup using dark from wix, or orca tool from microsoft.
Finally an extensive config file is also needed. This file is quite big and it’s best to obtain it like MSVBDPCADLL. In vs genereated setup it is called VSDNETCFG.ibd and is also located in Binary table.
<Binary Id=“InstallUtil“ SourceFile=“C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\InstallUtilLib.dll“ />
<Binary Id=“MSVBDPCADLL“ SourceFile=“Binary\MSVBDPCADLL.ibd“ />
<Binary Id=“VSDNETCFG“ SourceFile=“Binary\VSDNETCFG.ibd“ />
Now you can embed custom action like a normal file inside component
<File Id="MyCustomActionDll" Name="MyCustomAction.dll" Source="MyCustomAction.dll" />
Now it is time to create custom actions. First of all you have to create an action to call CheckFX from MSVBDCALL. It does some magic, which is unclear for me at the moment and sets VSDFxConfigFile property which will be used as a config file. You also need a custom action to call your assembly. You will need one customaction to set a property with commandline to call and another one which calls ManagedInstall from installutil. The property has to be named exactly the same as custom action calling ManagedInstall.
<CustomAction Id=“SetPrereqs“ BinaryKey=“MSVBDPCADLL“ DllEntry=“CheckFX“ />
<CustomAction Id=“MyCASetProp“ Property=“MyCA“ Value=“/installtype=notransaction /action=install /LogFile= "[#MyCustomActionDll]" "[VSDFxConfigFile]"“/>
<CustomAction Id=“MyCA“ BinaryKey=“InstallUtil“ DllEntry=“ManagedInstall“ Execute=“deferred“ />
Finally you have to instruct msi to execute your actions
<Custom Action=“SetPrereqs“ After=“InstallFiles“>NOT Installed</Custom>
<Custom Action=“MyCASetProp“ After=“SetPrereqs“>NOT Installed</Custom>
<Custom Action=“MyCA“ After=“MyCASetProp“>NOT Installed</Custom>