Description:
IllegalMonitorStateException encountered while using Coherence cache.
Symptoms:
2013 Jul 18 11:55:06:260 GMT -5 InferenceAgent2 Fatal [TIBCO_EMS_Session_Dispatcher_(30879)] - [runtime.service] [InferenceAgent] (Wrapped: Failed request execution for DistributedCache service on Member(Id=13, Timestamp=2013-07-17 14:55:33.185, Address=IP:port, MachineId=ID, Location=site:,machine:MACHINE_NAME,process:19736, Role=TibcoCepContainerBEMain)) null
(Wrapped: Failed request execution for DistributedCache service on Member(Id=13, Timestamp=2013-07-17 14:55:33.185, Address=IP:port, MachineId=ID, Location=site:,MACHINE_NAME,process:19736, Role=TibcoCepContainerBEMain)) java.lang.IllegalMonitorStateException
        at com.tangosol.util.Base.ensureRuntimeException(Base.java:288)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.tagException(Grid.CDB:36)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onPutRequest(PartitionedCache.CDB:50)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$PutRequest.run(PartitionedCache.CDB:1)
        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:1)
        at com.tangosol.coherence.component.util.DaemonPool$WrapperTask.run(DaemonPool.CDB:32)
        at com.tangosol.coherence.component.util.DaemonPool$Daemon.onNotify(DaemonPool.CDB:63)
        at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalMonitorStateException
at java.lang.Object.notifyAll(Native Method)
        at com.tangosol.net.internal.StorageVersion.commit(StorageVersion.java:176)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ResourceCoordinator.processEvent(Part
itionedCache.CDB:94)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ResourceCoordinator.finalizeInvokeSin
gleThreaded(PartitionedCache.CDB:56)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache$ResourceCoordinator.finalizeInvoke(Pa
rtitionedCache.CDB:9)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.processChanges(PartitionedCache.CDB:4)
        at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onPutRequest(PartitionedCache.CDB:34)
        ... 6 more


Cause:
The defect refers to a concurrency bug where a distributed query gets interrupted at some nodes if it is performed in the middle of frequent updates.

IllegalMonitorStateException encountered while using Coherence cache.

This issue has been discussed with Oracle support and it was found that it is a known defect(COH-6434) in the Coherence version 3.7.1.2 and was  fixed in the next minor release. The details of the defect is given below.

COH-6434: Fixed Issue with IllegalMonitorState being thrown while performing a query during rapid updates. The defect  refers to a concurrency bug where a distributed query gets interrupted at some nodes if it's performed in the middle of frequent updates. As the defect is on the concurrency ,oracle strongly recommended us to upgrade to as it might lead  to inconsistent cache data , deadlocks and ultimately engine crash .

Based on this discovery, we(TIBCO) have created  a CR BE-19095 in TIBCO BusinessEvents to include the upgraded version of coherence jar and have the fix in BE 4.0.2-HF6. Hence, we  recommend applying HF6 prior to moving this application into production or move any other upgraded BE application to production with BE 4.0.2 HF4.