Resolution:
Description:
============
RuntimeException: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member.

Environment:
==========
TIBCO BusinessEvents (BE) 4.x
All Operating Systems

Symptoms:
=========
BE Inference agents throws the following errors:

2012 Nov 06 09:25:01:448 GMT +8 G3TTEngine-1-G3TTEngine-1 Info [/Channels/emsChannel/g3Notification.Thread.6] - [user] [inference-class] Got Exception inside g3NotificationTxnPP-20121106DBSSSGS0BRT0991000.
Exception: com.tibco.cep.runtime.model.exception.impl.BEExceptionImpl: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member(Id=1, Timestamp=2012-10-25 14:03:07.26, Address=169.179.81.189:8088, MachineId=31310, Location=site:,machine:hlixtr855,process:19408, Role=TibcoCepContainerBEMain)
        at com.tibco.be.functions.coherence.CoherenceFunctions.loadConceptUsingExtId(CoherenceFunctions.java:509)
        at com.tibco.be.functions.coherence.CoherenceFunctions.C_CacheLoadConceptByExtId(CoherenceFunctions.java:558)
        at be.gen.RuleFunctions.PP.nullg3NotificationTxnPP$9y002fEvents$9y002fg3NotificationEvt$.g3NotificationTxnPP(nullg3NotificationTxnPP$9y002fEvents$9y002fg3NotificationEvt$.java:78)
        at be.gen.RuleFunctions.PP.g3NotificationTxnPP.invoke(g3NotificationTxnPP.java:13)
        at com.tibco.cep.runtime.session.impl.RuleSessionImpl.preprocessPassthru(RuleSessionImpl.java:947)
        at com.tibco.cep.runtime.service.om.coherence.cluster.agents.ClusterTaskController.executeTask(ClusterTaskController.java:63)
        at com.tibco.cep.runtime.scheduler.impl.WorkerBasedControllerV2$WorkerTask.run(WorkerBasedControllerV2.java:386)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at com.tibco.cep.runtime.util.CustomBEManagedThread.run(CustomBEManagedThread.java:23)
Caused by: com.tibco.cep.runtime.model.exception.impl.BEExceptionImpl: java.lang.RuntimeException: java.lang.RuntimeException: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member(Id=1, Timestamp=2012-10-25 14:03:07.26, Address=169.179.81.189:8088, MachineId=31310, Location=site:,machine:hlixtr855,process:19408, Role=TibcoCepContainerBEMain)
        at com.tibco.cep.runtime.service.om.coherence.CoherenceStore.getElement(CoherenceStore.java:1291)
        at com.tibco.cep.runtime.service.om.coherence.CoherenceStore.getElement(CoherenceStore.java:812)
        at com.tibco.be.functions.coherence.CoherenceFunctions.loadConceptUsingExtId(CoherenceFunctions.java:472)
        ... 9 more
Caused by: com.tibco.cep.runtime.model.exception.impl.BEExceptionImpl: java.lang.RuntimeException: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member(Id=1, Timestamp=2012-10-25 14:03:07.26, Address=169.179.81.189:8088, MachineId=31310, Location=site:,machine:hlixtr855,process:19408, Role=TibcoCepContainerBEMain)
        at com.tibco.cep.runtime.service.om.coherence.CoherenceStore.getElementFromStore(CoherenceStore.java:1007)
        at com.tibco.cep.runtime.service.om.coherence.CoherenceStore.getElement(CoherenceStore.java:1285)
        ... 11 more
Caused by: com.tibco.cep.runtime.model.exception.impl.BEExceptionImpl: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member(Id=1, Timestamp=2012-10-25 14:03:07.26, Address=169.179.81.189:8088, MachineId=31310, Location=site:,machine:hlixtr855,process:19408, Role=TibcoCepContainerBEMain)
+++++++++++++++++++++++++++++++

Cause:
=========
This issue is caused by a feature called "Service Guardian" introduced in Coherence 3.5.2. The issue is seen when the Coherence nodes gave delayed responses due to external reasons such as slow network speed and database latency. For more information on service guardian see the link below:
http://wiki.tangosol.com/display/COH35UG/service-guardian  


Resolution:
=========
To fix this issue, you can safely disable the feature as follows:

Add the following property in the tra file:
java.property.tangosol.coherence.guard.timeout=0

RuntimeException: com.tangosol.net.RequestTimeoutException: Abandoned execution of "GetRequest" for service "DistributedCache" on Member.