Login is not available at this time, the Primary Director may be offline

Login is not available at this time, the Primary Director may be offline

book

Article ID: KB0138564

calendar_today

Updated On:

Products Versions
DataSynapse GridServer 7.1.0

Description

When attempting to add a new SSL-enabled standalone broker to an existing SSL GridServer architecture, the standalone broker fails to initialize properly after installation.

Users attempting to access the GUI will be met with the following error message:

    "Login is not available at this time, the Primary Director may be offline."

If you inspect the standalone broker's catalina.log file, you will observe a severe SocketException leading to a keystore password verification failure. Depending on whether the keystore is in PKCS12 or JKS format, the Caused by chain will differ slightly:

PKCS12 keystore (most common in GS 7.1.0+):

    WARNING [DefaultPool-0: ] com.livecluster.connection.socket.ServerSocketReceiver$ICRunner.run while getting socket
    java.net.SocketException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
        at javax.net.ssl.DefaultSSLServerSocketFactory.throwException(SSLServerSocketFactory.java:160)
        at javax.net.ssl.DefaultSSLServerSocketFactory.createServerSocket(SSLServerSocketFactory.java:180)
        at com.datasynapse.commons.support.SSLSupport.createServerSocket(SSLSupport.java:124)
        at com.livecluster.connection.socket.ServerSocketReceiver.getServerSocket(ServerSocketReceiver.java:140)
        at com.livecluster.connection.socket.ServerSocketReceiver.access$400(ServerSocketReceiver.java:26)
        at com.livecluster.connection.socket.ServerSocketReceiver$ICRunner.run(ServerSocketReceiver.java:210)
        at com.datasynapse.commons.sync.threading.ReusableThreadPool$InternalThread.run(ReusableThreadPool.java:395)
    Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
        ...
    Caused by: java.io.IOException: keystore password was incorrect
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2090)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getKeyManagers(SSLContextImpl.java:1124)
        ...

JKS keystore (older installations):

    Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:791)
        ...
    Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:789)

In both cases, the DefaultSSLContext fails to initialize, so no SSL server socket is opened. As a result, the gridserver.log on the broker will show a repeating loop of:

    [BrokerLoginPlugin] Sending login message to <director-host>:5635
    [DirectorSocketSender] CloseConnection

approximately every 10 seconds, with no SSL-related detail in gridserver.log itself. The keystore error is only visible in catalina.log.

Note: This is a chicken-and-egg situation. The broker cannot reach the Director because its SSL context failed to initialize, so the GUI is also unavailable until the broker is fixed and
restarted.

Environment

DataSynapse GridServer 7.1.0 (SSL-Enabled Architectures)

Resolution

To resolve this issue, update the broker's configuration file to match the custom password applied to your server.keystore.

Follow these steps on the affected Standalone Broker:

1. Stop the standalone broker service.

2. Navigate to the following directory in your GridServer Manager installation:

       <install_root>\manager\webapps\livecluster\WEB-INF\config\

3. Take a backup copy of broker.xml before making any changes, so it can be restored if needed.

4. Open broker.xml in a standard text editor.

4. Locate the keyStorePassword property. By default it is configured with the encrypted value for
   "changeit":

       <property name="keyStorePassword" value="kimiSgz8t7cXagoyPdSmSA=="/>

   Because this is an existing SSL grid, the recommended approach is to copy the encrypted keyStorePassword value directly from the broker.xml of any other working Manager in the grid. GridServer uses the same DES key across all installations, so the encrypted ciphertext is identical on every host that uses the same password — no need to know the plaintext.

   On any working Manager host, open:

       <install_root>\manager\webapps\livecluster\WEB-INF\config\broker.xml

   Copy the value attribute of the keyStorePassword property and paste it into the same property on the affected broker's broker.xml:

       <property name="keyStorePassword" value="<value copied from working Manager>"/>

   If the plaintext password is known, it can also be entered directly:

       <property name="keyStorePassword" value="Your_Custom_Password"/>

Note: GridServer's CryptString implementation attempts DES decryption first and falls back to treating the value as plain text if decryption fails. Plain text values are therefore safe to use and will be re-encrypted by the UI on the next save.

6. Check whether the ManagerKeyStorePassword property also requires updating. This property controls access to the manager.keystore (used to encrypt RunAs and database passwords). If the manager keystore was also re-keyed to a non-default password, apply the same correction by copying the ManagerKeyStorePassword value from the working Manager's broker.xml in the same way:

       <property name="ManagerKeyStorePassword" value="<value copied from working Manager>"/>

7. Save broker.xml.

8. Restart the standalone broker service.

9. Verify recovery by tailing catalina.log for the absence of NoSuchAlgorithmException on startup, and tailing gridserver.log for a successful "Logged in" message from BrokerLoginPlugin.

Once the broker restarts successfully it will open the keystore, initialize the SSL context, and the GridServer GUI will become accessible.

 

Issue/Introduction

Failed to add a new SSL-enabled standalone broker to the existing SSL grid architecture.

This issue occurs when a non-default password (anything other than the default "changeit") is used for the broker's server.keystore. By default, the GridServer broker configuration expects the standard Java keystore password "changeit". When the standalone broker attempts to open the custom keystore using the default password, the Java Secure Socket Extension (JSSE) rejects the request.

This failure prevents the SSL context from initializing, crashing the broker's internal receivers and severing communication with the Primary Director.