TIBCO BusinessWorks 5.12 uses JRE 1.7.x. IPv6 is used by default on Windows starting with JDK 7. This is the reason for the java.net.SocketException. Setting java.property.java.net.preferIPv4Stack=true enables support for IPv4 on Java 7 and resolves the issue.
Resolution
Add java.property.java.net.preferIPv4Stack=true to the designer.tra for Design time and in the deployed application tra file for runtime.
Issue/Introduction
[SME-Approved] BW 5.12 send mail activity fails with "java.net.SocketException: Permission denied: connect".