As the error suggests "unknown key type passed to RSA", the certificate uses an encryption method that is unknown/unsupported by TIBCO Data Virtualization.
To check the encryption algorithm used by the certificate, open the CA-signed certificate, under the 'Details' tab, here the parameter 'Public key' will mention the parent algorithm used, and the parameter 'Public key parameters' will state the exact sub-type of the algorithm.
For example, refer to below image. Here the parent type is ECC, and the sub-type is ECDSA_P256.
Now, this sub-type can be cross-checked whether it is a supported type, in the "Java Supported Cipher Suites" section of the Security Guide of that particular TIBCO Data Virtualization version being used.
If it is not a supported algorithm, then a request can be made to the Certificate Authority to provide a new certificate that uses a supported algorithm. The TIBCO Data Virtualization's default certificate uses the RSA algorithm.