You can calculate the connections your MQ plug-in application needs as follows.

For example, your project has a process that has MQ activities (listener, put, get). First you will need to know how many MQ activities in this process. For example, if you have a listener, a put, and a get, this process will need 3 connections for one job instance. If the MQ listener is a starter activity and has 5 instances configured, then you could have 5 instances running in parallel and you will need 3 * 5 =  15 connections. The connections needed for one BW application of this project is 15. If you deploy your application to 2 instances then the total connections will be 15* 2 = 30. This is the way to figure out how many connections your application needs. Note that if you have more than one process you will need to sum up the connections of all the processes. 
 

Once you know the connections your application needs, then you can set the Max connections in the connection resource and Max instances per client in the server connection channel larger than the connections your application needs. The Max instance on the connection channel is a sum of all the clients as you may have other applications other than TIBCO using the channel.