Error: "SSL handshake failed" occurs when using the Streaming Kafka adapters
Article ID: KB0072594
Updated On:
| Products | Versions |
|---|---|
| TIBCO Streaming | 10.6 |
Description
When I start my Streaming application, I see the following error when the Kafka adapter attempts to connect to the Kafka broker:
How can I determine the root cause for this SSL handshake failure and resolve it? Setting the adapter's log level to DEBUG or ALL does not provide any additional details.
2022-01-31 11:58:21.661000-0500 [32012:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR org.apache.kafka.clients.NetworkClient: [Consumer clientId=SBClient_kafka-security-topic_9e56f813-b5ee-44f0-a984-255f442257c8, groupId=SBClient_kafka-security-topic_8bc2183d-f116-45f9-af00-f78ca56b51af] Connection to node -1 (ec2-54-210-20-255.compute-1.amazonaws.com/54.210.20.255:9093) failed authentication due to: SSL handshake failed
How can I determine the root cause for this SSL handshake failure and resolve it? Setting the adapter's log level to DEBUG or ALL does not provide any additional details.
Issue/Introduction
Provides general troubleshooting guidance for SSL handshake failures relating to the Streaming Kafka adapters.
Resolution
First, enable Java SSL debug logging by setting the Java system property:
This may be set in your SB Engine HOCON configuration:
Alternatively, you can add this to your jvmArgs section in the SB Engine HOCON configuration:
Or, add it to your Studio Run Configuration, which may be more convenient when testing fragments in Studio:
After starting your application and attempting to make the SSL connection (by sending a ' subscribe' command to the Kafka Consumer adapter, for example), look for the following log entry:
Here, the Kafka broker (i.e. the server) is presenting its public certificate to the client (i.e. the Kafka adapter). Inspect these details, and consider them when inspecting any SSL-related errors that may come shortly after this log entry. Some possible reasons for SSL handshake failures are:
1. The Common Name (CN) value in the Kafka broker's public certificate does not match the actual DNS hostname of the Kafka broker machine.
In this case, the following error will be reported shortly after the " Consuming server Certificate handshake" log entry:
This error indicates that the Kafka broker machine's DNS hostname ( ec2-54-210-20-255.compute-1.amazonaws.com) does not match the Common Name (CN) value shown in the Kafka broker's public certficate (" Not my actual hostname"). To resolve this, set the Kafka client property ' ssl.endpoint.identification.algorithm' in the Kafka adapter Properties view under Advanced Options > Advanced Config:
Here, we are leaving the value for this property empty, as noted in the Kafka documentation:
"The endpoint identification algorithm used by clients to validate server host name. The default value is https. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the broker's certificate. Disable server host name verification by setting ssl.endpoint.identification.algorithm to an empty string."
2. The client is not providing its certificate to the server, or the issuer of the client's certificate is not trusted by the Kafka broker.
In this case, you can expect to see an <empty list> for the ' Certificates' log entry. This log entry comes immediately after the " Produced client Certificate handshake" message:
This will be followed by the " Unexpected handshake message" log entry:
If you see these messages, inspect the Kafka broker's trust store to ensure the issuer certificate of the Streaming client has been added and it has not expired. Trust stores can be inspected using the Java 'keytool' command, supplied with your Streaming installation. For example..
3. The issuer of the Kafka broker certificate isn't trusted by the client.
In this case, the Java SSL debug output will display the following error:
This comes shortly after the " Consuming server Certificate" message shown above.
If you see this message in your log, inspect the Streaming application's (i.e. the client's) trust store to ensure the issuer certificate of the Kafka broker has been added and it has not expired. Again, use the Java keytool command to inspect the Streaming application's trust store.
javax.net.debug=all
This may be set in your SB Engine HOCON configuration:
systemProperties = { "javax.net.debug" = "all" } Alternatively, you can add this to your jvmArgs section in the SB Engine HOCON configuration:
jvmArgs = [ "-Djavax.net.debug=all" ]
Or, add it to your Studio Run Configuration, which may be more convenient when testing fragments in Studio:
After starting your application and attempting to make the SSL connection (by sending a ' subscribe' command to the Kafka Consumer adapter, for example), look for the following log entry:
2022-01-31 12:43:49.625000-0500 [31040:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: javax.net.ssl|DEBUG|4C|OperatorThread(default.InputAdapter.KafkaConsumer) |2022-01-31 12:43:49.625 EST|CertificateMessage.java:366|Consuming server Certificate handshake message ( 2022-01-31 12:43:49.625000-0500 ... "Certificates": [ 2022-01-31 12:43:49.625000-0500 ... "certificate" : { 2022-01-31 12:43:49.625000-0500 ... "version" : "v3", 2022-01-31 12:43:49.625000-0500 ... "serial number" : "3E 7E A2 9D", 2022-01-31 12:43:49.625000-0500 ... "signature algorithm": "SHA256withRSA", 2022-01-31 12:43:49.625000-0500 ... "issuer" : "CN=Not my actual hostname", 2022-01-31 12:43:49.625000-0500 ... "not before" : "2022-01-31 11:46:41.000 EST", 2022-01-31 12:43:49.625000-0500 ... "not after" : "2023-01-31 11:46:41.000 EST", 2022-01-31 12:43:49.625000-0500 ... "subject" : "CN=Not my actual hostname", 2022-01-31 12:43:49.625000-0500 ... "subject public key" : "RSA", Here, the Kafka broker (i.e. the server) is presenting its public certificate to the client (i.e. the Kafka adapter). Inspect these details, and consider them when inspecting any SSL-related errors that may come shortly after this log entry. Some possible reasons for SSL handshake failures are:
1. The Common Name (CN) value in the Kafka broker's public certificate does not match the actual DNS hostname of the Kafka broker machine.
In this case, the following error will be reported shortly after the " Consuming server Certificate handshake" log entry:
2022-01-31 12:43:49.676000-0500 [31040:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: javax.net.ssl|ERROR|4C|OperatorThread(default.InputAdapter.KafkaConsumer) |2022-01-31 12:43:49.676 EST|TransportContext.java:361| Fatal (CERTIFICATE_UNKNOWN): No name matching ec2-54-210-20-255.compute-1.amazonaws.com found (
This error indicates that the Kafka broker machine's DNS hostname ( ec2-54-210-20-255.compute-1.amazonaws.com) does not match the Common Name (CN) value shown in the Kafka broker's public certficate (" Not my actual hostname"). To resolve this, set the Kafka client property ' ssl.endpoint.identification.algorithm' in the Kafka adapter Properties view under Advanced Options > Advanced Config:
Here, we are leaving the value for this property empty, as noted in the Kafka documentation:
"The endpoint identification algorithm used by clients to validate server host name. The default value is https. Clients including client connections created by the broker for inter-broker communication verify that the broker host name matches the host name in the broker's certificate. Disable server host name verification by setting ssl.endpoint.identification.algorithm to an empty string."
2. The client is not providing its certificate to the server, or the issuer of the client's certificate is not trusted by the Kafka broker.
In this case, you can expect to see an <empty list> for the ' Certificates' log entry. This log entry comes immediately after the " Produced client Certificate handshake" message:
2022-01-31 13:59:17.561000-0500 [23184:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: javax.net.ssl|DEBUG|49|OperatorThread(default.InputAdapter.KafkaConsumer) |2022-01-31 13:59:17.561 EST|CertificateMessage.java:330|Produced client Certificate handshake message ( 2022-01-31 13:59:17.561000-0500 [23184:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: "Certificates": <empty list>
This will be followed by the " Unexpected handshake message" log entry:
2022-01-31 13:42:18.457000-0500 [13892:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: javax.net.ssl|ERROR|49|OperatorThread(default.InputAdapter.KafkaConsumer) |2022-01-31 13:42:18.457 EST|TransportContext.java:361| Fatal (UNEXPECTED_MESSAGE): Unexpected handshake message: server_hello (
If you see these messages, inspect the Kafka broker's trust store to ensure the issuer certificate of the Streaming client has been added and it has not expired. Trust stores can be inspected using the Java 'keytool' command, supplied with your Streaming installation. For example..
keytool -list -V -keystore kafka-broker-trust-store.jks
3. The issuer of the Kafka broker certificate isn't trusted by the client.
In this case, the Java SSL debug output will display the following error:
2022-01-31 14:08:00.600000-0500 [30280:OperatorThread(default.InputAdapter.KafkaConsumer)] ERROR com.tibco.ep.dtm.stderr: javax.net.ssl|ERROR|49|OperatorThread(default.InputAdapter.KafkaConsumer) |2022-01-31 14:08:00.600 EST|TransportContext.java:361| Fatal (CERTIFICATE_UNKNOWN): PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
This comes shortly after the " Consuming server Certificate" message shown above.
If you see this message in your log, inspect the Streaming application's (i.e. the client's) trust store to ensure the issuer certificate of the Kafka broker has been added and it has not expired. Again, use the Java keytool command to inspect the Streaming application's trust store.
Feedback
Yes
No
Powered by
