One of the common problems for TIBCO Scribe® Insight Queue based integrations is the creation of duplicate records when multiple messages for the same entity/record are generated into the ScribeIn Message Queue at the same time by the Dynamics Publishers. This could occur due to some type of workflow process that saves a record after the initial record is created or through an automated save in the UI that saves a record before it is complete (e.g. a document number is assigned to the record and saved)

For example, assume a new Account, XYZ , was created and multiple messages were created – 1 Insert message and 1 Update message. The issue is both may get processed at the same time by separate message processors.

Normally a DTS uses some sort of Seek or Lookup operation to determine if the Account exists and then does either an Insert operation or an Update operation depending on whether or not the account already exists. For a new Account, the Seek/Lookup for each message processor could return the same result, Account doesn’t exist, and perform two Inserts rather than an Insert and then an Update. It is a matter of timing. Below are three options that can be used to prevent duplicate record creation:

Option 1

Set the number of message processors to 2 limiting all queue based integrations to a single thread. This solution works well if the volume of messages is small enough that they can be processed in a timely manner.
To set the number of message processors:

1. Open the TIBCO Scribe® Insight Console.
2. Expand the tree in the left pane.
3. Right-click Integration Server and select Properties.
4. On the Integration Server Properties dialog in the Processor Configuration section, set the Active field to 2.
5. Click OK to save your changes.

 

Option 2
 

NOTE: This option requires TIBCO Scribe® Insight version 7.9.1 or higher, and a Professional or Enterprise license. To verify the License type, open TIBCO Scribe® Insight Workbench and select Help > About Scribe Insight.

Create an additional Processor Group with a single message processor assigned and add the integration to that group. This limits the integration to a single thread, but other Processor Groups can allow multi-threading integrations where duplicate record creation is not an issue.

To create an additional group:

1. Open the TIBCO Scribe® Insight Console.
2. Expand the tree in the left pane.
3. Right-click Integration Server and select Properties.
4. On the Integration Server Properties dialog in the Processor Groups section, click Add.
5. Enter a Name for the group and click OK to save it. The new group displays in the Integration Server Properties dialog with a single processor.
6. Click OK to save your changes.

To assign an Integration to that group:

1. Open the TIBCO Scribe® Insight Console.
2. Expand the tree in the left pane.
3. Expand Integration Server and select Integration Processes.
4. Select the process that contains the Integration that is generating duplicates.
5. Click Properties.
6. Select Step 1.
7. In the Processor Group field, select the name of your new Processor Group.
8. Click Apply to save.

 

Option 3
 

NOTE: This option requires TIBCO Scribe® Insight version 7.9.1 or higher, and a Professional or Enterprise license. To verify the License type, open TIBCO Scribe® Insight Workbench and select Help > About Scribe Insight.

For scenarios where neither of the options above are feasible, a third option using DTS design may serve as a workaround to the problem. This option uses the Scribe Operation field on the source XML to control processing based on whether the value is an Insert or an Update.

The diagram below illustrates this process.
 
For the DTS design, use a Seek step with a Pre-Operation Step Flow formula to control this behavior. The Pre-Operation Step Flow formula determines whether the message is an Insert or Update.

• If it is an Insert, process the message.
• If it is an Update, verify if the account exists.
• If the Account does not exist, fail the job and send the message to the ScribeRetry queue.

The message keeps retrying until the Account gets created or it has reached the maximum number of retries. If it reaches the maximum number of retries, it goes to the ScribeDeadMessage queue.
 

One of the common problems for TIBCO Scribe® Insight Queue based integrations is the creation of duplicate records when multiple messages for the same entity/record are generated into the ScribeIn Message Queue at the same time by the Dynamics Publishers.