Description:
Cluster is not coming up properly. Error, "Failed connecting to [Metaspace:xxxx] with error SYS_ERROR (connect_timeout - connect timeout). Retrying in [30] seconds".


Symptoms:
Error reported:
 
2014 Sep 11 17:32:52:282 GMT -7 -CS-P01 Warning [main] - [as.runtime.cluster.om.daoprovider] Failed connecting to [Metaspace:xxxx] with error SYS_ERROR (connect_timeout - connect timeout). Retrying in [30] seconds
2014 Sep 11 17:32:52:284 GMT -7 -CS-P01 Debug [main] - [runtime.session] Error performing initAll()
java.lang.RuntimeException: Unable to connect to metaspace [aaaaa] even after 5 tries
        at com.tibco.cep.runtime.service.dao.impl.tibas.ASDaoProvider.initHook(SourceFile:344)
        at com.tibco.cep.runtime.service.om.impl.AbstractDaoProvider.init(AbstractDaoProvider.java:65)
        at com.tibco.cep.runtime.service.cluster.MultiAgentCluster.init(MultiAgentCluster.java:112)
        at com.tibco.cep.runtime.session.impl.RuleServiceProviderImpl.initCluster(RuleServiceProviderImpl.java:707)
        at com.tibco.cep.runtime.session.impl.RuleServiceProviderImpl.configure(RuleServiceProviderImpl.java:271)
        at com.tibco.cep.container.standalone.BEMain.jumpStart(SourceFile:149)
        at com.tibco.cep.container.standalone.BEMain.main(SourceFile:69)
2014 Sep 11 17:32:52:284 GMT -7 xxxx-CS-P01 Fatal [main] - [container.standalone] Unable to connect to metaspace [aaaaa] even after 5 tries
java.lang.RuntimeException: Unable to connect to metaspace [aaaaa] even after 5 tries
        at com.tibco.cep.runtime.service.dao.impl.tibas.ASDaoProvider.initHook(SourceFile:344)
        at com.tibco.cep.runtime.service.om.impl.AbstractDaoProvider.init(AbstractDaoProvider.java:65)
        at com.tibco.cep.runtime.service.cluster.MultiAgentCluster.init(MultiAgentCluster.java:112)
        at com.tibco.cep.runtime.session.impl.RuleServiceProviderImpl.initCluster(RuleServiceProviderImpl.java:707)
        at com.tibco.cep.runtime.session.impl.RuleServiceProviderImpl.configure(RuleServiceProviderImpl.java:271)
        at com.tibco.cep.container.standalone.BEMain.jumpStart(SourceFile:149)
        at com.tibco.cep.container.standalone.BEMain.main(SourceFile:69
Cause:
Question checklist.

1). What is the Cluster architecture? How many seeders, physical machines?

2). Are you able to use 'as-admin' to connect to the metaspace? What's is the space status?

3). Do you have access to the AS logs?

4).  Check the physical machines resources usage such as CPU/Memory usage percentages.

5). Did this Cluster work properly before? Any operations applied on the metaspace before this error was seen?

6). Check the AS log to see if the discovery node has started. If you see a message such as, "waiting for discovery nodes, retry count [6], discovery [tcp://], it shows that the discovery node has not started.

7). Check the CDD configuration.

8). Check the listening URL settings. If you use, for example, tcp://22519, confirm that the engine you want to start uses tcp://<hostname>:22519 and not any other value.

9). To check the hostname, ping it.

The hostname must be the hostname on the local machine. You can type the command "hostname" in the terminal window and use this name.