You may receive this encryption error because of the encryption type entered in the ktpass command or the krb5.conf file is not supported by the KDC (Active directory domain controller if you are using Microsoft AD). The encryption types supported by an Active Directory domain controller are listed in the msDS-SupportedEncryptionTypes attribute of the domain controller's computer object. In a default installation, they are typically something like the folowing:
RC4_HMAC_MD5
AES128_CTS_HMAC_SHA1_96
AES256_CTS_HMAC_SHA1_96
You can refer to the link below to know the encryption type settings for computer object('Computer Account Encryption Type Setting' section) :
https://blogs.msdn.microsoft.com/openspecification/2011/05/30/windows-configurations-for-kerberos-supported-encryption-type/
Depending on the supported encryption types by an Active Directory domain controller, you need to set the encryption type in the ktpass command and krb5.conf file.
Note: This error is very generic and there could be multiple possible reasons for this error.
You may come across this issue if you have very old version of JCE jars (US_export_policy.jar and local_policy.jar) in Spotfire server installation directory (For example: C:\tibco\tss\10.6.0\jdk\jre\lib\security\policy)
You can download latest versions of jars from this location:
https://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html