In BW 5.x you can copy a 3rd party JMS client JAR to the BW classpath and then connect to the corresponding JMS server.

The class loading architecture of BW 6.x is completely different from BW 5.x. As BW 6.x works with the OSGI architecture, simply putting the external JMS client JARs in the classpath will not work in BW 6.x. You need to create an OSGI bundle from the external JARs. There is a specific way to put this for the Design time and runtime environment. A BW 5.x project (connection resources and process definitions, etc.) using the Weblogic/MQ/IBM WAS as JNDI/JMS provider will be successfully migrated to a BW 6.x project using the migration tool. However, when you try to do a "test Connection" for the JMS Connection resource, BW 6.x will not be able to find the class required for the third party JNDI Context URL.

As of now (BW 6.2), BW 6 only support TIBCO EMS as the JNDI/JMS provider out of the box. You do not need to install any client JAR in Design time and runtime. It is built in. If you want to use IBM MQ as a JMS Provider. you need to use "TIBCO ActiveMatrix BusinessWorks Plug-in for WebSphere MQ" which can be downloaded from https://edelivery.tibco.com. There is also an enhancement request to  support Oracle AQ/IBM MQ as a JMS Provider.