Resolution:

Description:
============
Import Records event is throwing a java.util.ConcurrentModificationException on a cluster environment while having no duplicate data in the data source loaded.

Environment:
===========
TIBCO Collaborative Information Manager(CIM) 8.2.1

Symptoms:
========
2012-08-24 10:26:50,956 [TIBCO EMS Session Dispatcher (51543)] DEBUG com.tibco.mdm.workflow.engine.activities.AsyncCatalogImportRecords -
<MqException: BEGIN>
Code: ERROR
ID: 0AFEE26B_8A7E62EB39554988013959073F630F36
DATETIME: Fri Aug 24 10:26:50 EDT 2012
EXCEPTIONMESSAGE: java.util.ConcurrentModificationException
STACKTRACE: java.util.ConcurrentModificationException
    at java.util.HashMap$AbstractMapIterator.checkConcurrentMod(HashMap.java:122)
    at java.util.HashMap$AbstractMapIterator.makeNext(HashMap.java:127)
    at java.util.HashMap$EntryIterator.next(HashMap.java:188)
    at java.util.HashMap$EntryIterator.next(HashMap.java:181)
    at java.util.HashMap.writeObject(HashMap.java:841)
    at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:600)
    at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1004)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1470)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1401)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1401)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:332)
    at com.tibco.mdm.infrastructure.cache.CacheUtil.writeObject(CacheUtil.java:186)
    at com.tibco.mdm.repository.maps.CatalogInputMap.writeExternal(CatalogInputMap.java:1245)
    at com.tibco.mdm.infrastructure.cache.as.ExternalizableTupleCreateCommand.createTuple(ExternalizableTupleCreateCommand.java:25)
    at com.tibco.mdm.infrastructure.cache.MqCacheManagerActiveSpaces.putAll(MqCacheManagerActiveSpaces.java:1392)
    at com.tibco.mdm.session.repository.maps.CatalogInputMapCacheHelper.addLatestCatalogInputMapToCache(CatalogInputMapCacheHelper.java:126)
    at com.tibco.mdm.session.repository.maps.CatalogInputMapSsnBean.getCatalogInputMap(CatalogInputMapSsnBean.java:574)
    at com.tibco.mdm.session.repository.maps.CatalogInputMapBusinessDelegate.getCatalogInputMap(CatalogInputMapBusinessDelegate.java:86)
    at com.tibco.mdm.workflow.engine.activities.MqActivityInstImportCatalogRecordsHelper.importAllRecords(MqActivityInstImportCatalogRecordsHelper.java:1460)
    at com.tibco.mdm.session.repository.core.CatalogProcessorSsnBean.importRecords(CatalogProcessorSsnBean.java:3510)
    at com.tibco.mdm.session.repository.core.EJSRemoteStatelessCatalogProcessorSsn_a56c0ff9.importRecords(Unknown Source)
    at com.tibco.mdm.session.repository.core._CatalogProcessorSsnRemote_Stub.importRecords(_CatalogProcessorSsnRemote_Stub.java:1254)
    at com.tibco.mdm.session.repository.core.CatalogProcessorBusinessDelegate.importRecords(CatalogProcessorBusinessDelegate.java:294)
    at com.tibco.mdm.workflow.engine.activities.MqActivityInstImportCatalogRecordsHelper.importAllDataAsync(MqActivityInstImportCatalogRecordsHelper.java:1364)
    at com.tibco.mdm.workflow.engine.activities.AsyncCatalogImportRecords.process(AsyncCatalogImportRecords.java:199)
    at com.tibco.mdm.workflow.engine.activities.AsyncCatalogImportRecords.onAsyncCall(AsyncCatalogImportRecords.java:165)
    at com.tibco.mdm.integration.messaging.AsyncCallMessageListener.onMessage(AsyncCallMessageListener.java:56)
    at com.tibco.mdm.integration.messaging.message.MqNativeMessageListener.onMessage(MqNativeMessageListener.java:93)
    at com.tibco.tibjms.TibjmsxSessionImp._submit(TibjmsxSessionImp.java:4165)
    at com.tibco.tibjms.TibjmsxSessionImp._dispatchAsyncMessage(TibjmsxSessionImp.java:2267)
    at com.tibco.tibjms.TibjmsxSessionImp$Dispatcher.run(TibjmsxSessionImp.java:3689)
ERRORMESSAGE: Import Catalog Product failed:ERROR: java.util.ConcurrentModificationException

Cause:
=====
Multiple nodes trying to modify the same tuple.

Resolution:
==========
Change the following in CacheConfig.xml in $MQ_HOME/config directory for CATALOGINPUTMAP

From the following:

                                <Cache>
                                                <Name>CATALOGINPUTMAP</Name>
                                                <Type>near</Type>
                                                <ReplicationCount></ReplicationCount>
                                                <SingleByteObjectSize>864</SingleByteObjectSize>
                                                <MultiByteObjectSize>1522</MultiByteObjectSize>
                                </Cache>

To:

                                <Cache>
                                                <Name>CATALOGINPUTMAP</Name>
                                                <Type>distributed</Type>
                                                <ReplicationCount></ReplicationCount>
                                                <SingleByteObjectSize>864</SingleByteObjectSize>
                                                <MultiByteObjectSize>1522</MultiByteObjectSize>
                                </Cache>


Make the following change to the configuration file:

1).    Shutdown CIM servers.
2).    Shutdown AS servers.
3).    Start AS servers
4).    Restart the CIM cluster.