Resolution:
In case of an SAP adapter, the encryption is possible only between the SAP adapter and the TIBCO network on which the data is being sent. For example suppose SAP is publishing some data on the TIBCO network via the SAP adapter then the data that goes out from the adapter to the TIBCO network can be encrypted. Similarly for the subscription service, the data going from the TIBCO network to the SAP adapter can be encrypted.

For enabling encryption between the adapter and the TIBCO network there two possible scenarios:-
1. When the transport type is JMS: in this case you have to enable the "Use SSL" in the R3 Adapter design configuration.
2. When the transport type is RV: in this case RV version 7.2 and above support the SSL.

From adapter side there is no specific security configuration that can be used for outbound adapter, as communication with SAP is based on program id. From SAP side you can check out with your SAP admin if any configuration can be made for the program id so that only selected programs can connect etc.  For inbound adapter, security can be achieved using userid/pwd.

Is there any data security possible between SAP system and SAP adapter?