While trying to connect to a trading partner AS2 URL, we get the below error in BC 6.4 :
<<<
...
2019 Dec 26 14:56:10:535 GMT -0500 BW.BusinessConnect-Interior_Server Debug [bw.logger] BW-EXT-LOG-300002 Job-327019.327019.zK1TDoJcEwAFOE7v-d0rHHdEEW6.esbint01-Interior-Server HTTPSTransport: Connecting to xx.yy.com:443
2019 Dec 26 14:56:10:535 GMT -0500 BW.BusinessConnect-Interior_Server Error [bw.logger] BW-EXT-LOG-100000 Job-327019.327019.zK1TDoJcEwAFOE7v-d0rHHdEEW6.esbint01-Interior-Server HTTPSTransport: Error connecting to xx.yy.com:443: Error communicating with host xx.yy.com at port 443 . Connection closed by remote host.
2019 Dec 26 14:56:10:535 GMT -0500 BW.BusinessConnect-Interior_Server Debug [bw.logger] BW-EXT-LOG-300002 Job-327019.327019.zK1TDoJcEwAFOE7v-d0rHHdEEW6.esbint01-Interior-Server HTTP sync reply: statusCode = [699] statusMsg = [Failed to connect to xx.yy.com:443: Error communicating with host xx.yy.com at port 443 . Connection closed by remote host.] confirmationID = [dmzmsh-reply-conf-114B042C-5B26-4EA9-B2E4-8FAB29C4E0BE]
...
>>>

The same connection worked fine in BC 6.1. We have used third party tools and even used openssl to force use TLSv1 for the connection (since in TP configuration they are using TLSv1) and it connects successfully. The weird part is, the connection fails right away, there is no handshake been logged before it fails. We are using Entrust provider.
 

After upgrading the version of BusinessConnect to a later version, connections that previously worked are now failing with "Connection closed by remote host" errors.

Windows Server 2012 R2

The reason it was working in 6.1 environment and not in 6.4 environment is because of an upgrade to the security library in the underlying TRA.  The newer version of the security library forces a larger value to the DH key size, which older servers may not be able to handle.

The issue is resolved by adding the below property to the BusinessConnect-Interior_Server.tra file:
java.property.iaik.security.ssl.SSLContext.ephemeralClientDHKeySize=1024
The above property controls the minimum allowed size of the Domestic DH key accepted by the SSL client during a DH Server Key Exchange. 

NOTE : The issue will be resolved by changing the security vendor to SUN.

ephemeralClientDHKeySize, ENTRUST, remote host, connection