Kafka channel fatal error caused by "GSSException: No valid credentials provided"

Kafka channel fatal error caused by "GSSException: No valid credentials provided"

book

Article ID: KB0072900

calendar_today

Updated On:

Products Versions
TIBCO BusinessEvents Enterprise Edition 6.1 and later

Description

When a BE application, using the Kafka Channel, is started, a fatal error occurs:
 
FATAL [main] - [driver.kafka] Failed to connect Kafka server at [ec2-174-129-102-157.compute-1.amazonaws.com:9094]
org.apache.kafka.common.errors.SaslAuthenticationException: 
An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed 
[Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]) 
occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.
Caused by: javax.security.sasl.SaslException: GSS initiate failed 
[Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
...
[WARN][org.apache.kafka.common.security.kerberos.KerberosLogin] [Principal=user1/@KAFKA.SECURE]: 
TGT renewal thread has been interrupted and will exit.

Our Kafka broker requires that clients authenticate with Kerberos. What does this error mean and how can it be avoided?
 

Issue/Introduction

Provides some details and background on the fatal error caused by "GSSException: No valid credentials provided".

Environment

All Supported Platforms

Resolution

This error indicates that the client krb5 configuration file (i.e. krb5.conf or krb5.ini) is not mapping the Kerberos Kafka realm name to the domain name of the Kafka broker machine. To correct this, add the appropriate mapping under the [domain_realm] section. For example:
 
[realms]
KAFKA.SECURE = {
  kdc = ec2-174-129-102-157.compute-1.amazonaws.com
  admin_server = ec2-174-129-102-157.compute-1.amazonaws.com 
}
...
[domain_realm]
  .amazonaws.com = KAFKA.SECURE
  amazonaws.com = KAFKA.SECURE

Here, we have configured the client (i.e. the BE application) to connect to a Kerberos KDC running on an Amazon EC2 instance. Hence, the domain mapping for the 'KAFKA.SECURE' Kerberos realm has been added (since the realm name does not match the Kafka broker machine's domain). Check with your Kerberos administrator to determine the appropriate domain-realm mapping.

To assist in troubleshooting, set the sun.security.krb5.debug system property to 'true'. You may set this in your project's CDD file under Cluster > Properties:

 
sun.security.krb5.debug = true

For detailed guidance on configuring the Kafka Channel for Kerberos authentication, refer to the Knowledge article entitled "How to enable Kerberos SASL_SSL authentication for the TIBCO BusinessEvents Kafka Channel".