Resolution:
Description:
============
In BusinessEvents(BE) 3.x, exception "Failed to start Service "DistributedCache" (ServiceState=SERVICE_STOPPED)"
is thrown after Inference Agent is running for some time



Environment:
==========

   o TIBCO BusinessEvents 3.x
  


Symptoms:
=========
BE Engine throws the following exception -

java.lang.RuntimeException: Failed to start Service "DistributedCache" (ServiceState=SERVICE_STOPPED)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.waitAcceptingClients(Service.CDB:12)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache.waitAcceptingClients(DistributedCache.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.poll(Service.CDB:8)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.poll(Service.CDB:1)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.DistributedCache$BinaryMap.get(DistributedCache.CDB:27)


Cause:
=====
The root for this issue can be that BE agents and Cache Server are not able to
communicate properly. We have seen this issue in past, where it was observed that issue can occur
because of any of the following reasons -          


1. This can be due to network issues like under heavy load BE engines are losing the connection with
   each other.              

2. Engines may be running out of heap space.


Resolution:
==========
1. For Cause 1, check the coherence logs, if there is any network outage.

2. For Cause 2, start the engine after increasing the heap size to larger value.


References:
==========
To dig further into the problem you will require debug logs for Coherence.    
For Enabling Coherence logging add below properties in engine tra file -    
  
be.oracle.debug=true    
java.property.be.cache.dump=true  
java.property.tangosol.coherence.log.level=9  


# Provide unique name for logfile in each Agent/CS tra file
java.property.tangosol.coherence.log= /Coherence-CS.log

For more explanation on above properties, refer to the below link
http://coherence.oracle.com/display/COH32UG/Command+Line+Setting+Override+Feature