UDP-based Service Discovery fails in one of our deployment environments, so we have configured in the TIBCO® Streaming NodeDeploy section for each node the section (example):

configuration = {
  NodeDeploy = {
    nodes = {
      "A.bizapp"  = {
        communication = {
          administration = {
            address = ${HOSTNAME}
            transportPort = ${A_ADMINPORT}
          }
          proxyDiscovery = {
            remoteNodes = [ 
              "B.bizapp","C.bizapp","D.bizapp"  
            ]
          }
        }
      }
      "B.bizapp" = { 
        ...
      }
      ...
    }
  }
}

When run in the environment where UDP is blocked, the administrative 'epadmin install node' and related commands work, but 'epadmin display services' does not.

When run in any environment where UDP is allowed, the first node installs correctly, but the second node fails with error:

Installing node
install of node B.bizapp using discovery port 54321 failed: the service name is already in use by service address 10.11.12.13:54321
[B.bizapp] not able to get the node's name from the node: SWSocket::initClient: Call to 'connect' failed: Connection refused [errno:111]

The command 'epadmin display services' reports all defined nodes A, B, C, and D as installed, but only "A" should be installed.

Why is installation blocked?




 

The proxyDiscovery setting is only to be used in environments that block UDP

Proxy Discovery is an optional NodeDeploy configuration that identifies all the expected nodes in a cluster and their explicit hostname and ports when these values cannot be obtained using UDP-based Service Discovery.

In the working environment UDP-based Service Discovery returns no information, so node installation of the second and following nodes succeeds based only on the supplied NodeDeploy configuration.

In the failing environments UDP-based Service Discovery succeeds. When node A is installed Service Discovery reports no other nodes in the cluster, and installation continues. When node B is installed Service Discovery finds node A, which reports its full NodeDeploy configuration which also identifies the presence of B, C, and D. Since the second node is named "B" and a "B" has already been reported through UDP-based Service Discovery node B is not allowed to be installed. This would also be true for any attempt to install the other nodes.

For Proxy Discovery to work, UDP communication must be prevented by the active network configuration for the system on which the node is being installed. When there are several deployment environments, some with UDP available and some without, you must use different NodeDeploy configurations in each environment. It is allowed for each node in a cluster to have a different NodeDeploy configuration with different settings (the settings for Transactional Memory and distribution must agree for the node to manage cluster state correctly).

At the time of this writing this is a known limitation in TIBCO Streaming version 10.6 and the associated defect is "SB-50784 Proxy configuration makes service discovery report missing nodes as installed".


There are four known workarounds when using Proxy Discovery configuration in an environment where UDP-based Service Discovery will also work:

A. Disable UDP at the firewall for all systems running nodes.

B. Install each node on a different Discovery Port so that UDP-based discovery doesn't work.

This uses option "discoveryport" in the 'epadmin install node' command, for example:

epadmin --hostname devsystemA1 install node --nodename=A.bizapp --discoveryport=2201 --adminport=2152  –application=bizapp-ep-application.zip --nodedirectory=/tmp 

epadmin --hostname devsystemA2 install node --nodename=B.bizapp --discoveryport=2202 --adminport=2152  –application=bizapp-ep-application.zip --nodedirectory=/tmp

C. Stop and terminate all nodes in the cluster before installing a new node, then start each node from the terminated state using administrative command (example):

Terminate Nodes:
  epadmin servicename=A.X install node ...
  epadmin servicename=A.X terminate node   

  epadmin servicename=B.X install node ...
  epadmin servicename=B.X terminate node   

Start Nodes from Termination:
  epadmin start node --installpath=./A.bizapp 
  epadmin start node --installpath=./B.bizapp

D. Install and start all nodes at the same time (This is equivalent to installing each node before any node is fully started). All nodes installation commands must be run within approximately 20 seconds of each other to avoid UDP-based service discovery.