You can enable remote JMX monitoring with SSL for an AppNode / BWAgent in BW 6 or Application in BW 5.

How to configure JMX with SSL in BW

All

Please ensure the properties below are applied in the respective bwappnode.tra / bwagent.tra / application.tra file
Jmx.Enabled=true
java.property.com.sun.management.jmxremote=true
java.property.com.sun.management.jmxremote.port=<available_port>
java.property.com.sun.management.jmxremote.ssl=true
java.property.javax.net.ssl.keyStore=<file_path_keystore.jks>
java.property.javax.net.ssl.keyStorePassword=<keystore.jks_password>
java.property.com.sun.management.jmxremote.authenticate=false
java.property.com.sun.management.jmxremote.ssl.need.client.auth=false
java.property.com.sun.management.jmxremote.registry.ssl=true

With the 2 properties below required you can then start, for example, JConsole from the command line:
jconsole -J-Djavax.net.ssl.trustStore=<file_path_truststore.jks> -J-Djavax.net.ssl.trustStorePassword=<trustword.jks_password>
When starting JConsole you can have the additional property if required -J-Djavax.net.debug=SSL,handshake to enable debug logging in the JConsole extra console window.
On JConsole you can log in via Remote Process <hostname>:<port>

If using JMC then please add the following properties to the jmc.ini file before running JMC:
-Djavax.net.ssl.trustStore=<file_path_truststore.jks>
-Djavax.net.ssl.trustStorePassword=<trustword.jks_password>
-Djavax.net.debug=SSL,handshake 
If required the property javax.net.debug can be used and if it's set to value "all" then it will produce a heavy load of logs which may use too many resources unless appropriately tuned.

To create your own keystore.jks and truststore.jks please see the following guide which explains how you can use keytool to do this:
https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#CreateKeystore
The 3 sample commands are:
Create a new keystore and self-signed certificate with corresponding public and private keys:
keytool -genkeypair -alias duke -keyalg RSA -validity 7 -keystore keystore
Export and examine the self-signed certificate:
keytool -export -alias duke -keystore keystore -rfc -file duke.cer
Import the certificate into a new truststore:
keytool -import -alias dukecert -file duke.cer -keystore truststore

If you want to incorporate file based authentication then please refer to the following respective KBs as well:
https://support.tibco.com/s/article/How-to-enable-authentication-for-JMX-monitoring-on-AppNode
https://support.tibco.com/s/article/JMX-Authentication-for-BW-5-x

For more details please refer to the documentation for JMX:
https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html