WCF-SQL adapter, using MSDTC cross domain

Today I had to set up the WCF-SQL adapter, to write data to a database used for BI purposes. We have our BizTalk machine located at a hoster, the BI database is located on a server on-premise at the client, and a VPN connection connects the servers. The servers are in different domains.

Generating the schemas went without problems, however once we wanted to do some actual work on the database we got the following 2 errors in the event log:

MSDTC encountered an error (HR=0x80000171) while attempting to establish a secure connection with system BIBIZ.

And

The adapter failed to transmit message going to send port “Communicate with BI” with URL “mssql://bibiz.acme.nl//bibizDatabase?”. It will be retransmitted after the retry interval specified for this Send Port. Details:”System.Transactions.TransactionManagerCommunicationException: Communication with the underlying transaction manager has failed. —> System.Runtime.InteropServices.COMException: The MSDTC transaction manager was unable to push the transaction to the destination transaction manager due to communication problems. Possible causes are: a firewall is present and it doesn’t have an exception for the MSDTC process, the two machines cannot find each other by their NetBIOS names, or the support for network transactions is not enabled for one of the two transaction managers. (Exception from HRESULT: 0x8004D02A)

It turned out, that to use MSDTC across domains you have to set some options. I will list the actions to be taken here, both for my own reference, and to help anyone else running into this problem. Continue reading