Resolution:

1). The "filename" value must use an existing folder where the user has privileges. In other words, the property would report "file/directory not found" if there is no such folder. The property has only the privilege to create the file but not the folder.

2). The file path pattern must resemble "c:/temp/soaptest.xml", if you use "c:\temp\soaptest.xml". The application will report "illegal file/directory" in Windows.

3). Check the BW log, TSM log or nohup.out if the SOAP log is not created. You will find error messages in those log files warning that you are using a non-existing folder or wrong pattern.

----------------------------------------------------------------------------------------------
java.property.com.tibco.plugin.soap.trace.inbound=true
java.property.com.tibco.plugin.soap.trace.outbound=true
java.property.com.tibco.plugin.soap.trace.filename=c:/temp/soapNM20160126test.xml
java.property.com.tibco.plugin.soap.trace.pretty=true
java.property.com.tibco.plugin.soap.trace.stdout=true

Failure to obtain a SOAP trace log file.