There are a number controls you can use to optimize the performance of messages going to Dynamics 365 / CRM.  We have made this convenient by having these controls right inside TIBCO Scribe® Online, both at the Map level and in the Connection dialog for Dynamics 365.  Some of these settings require reaching out to Microsoft directly to increase the ability to process more data.

Before You Start

It is important to understand how certain TIBCO Scribe® Online settings are used and how they affect throughput to a target datastore.

Target Operation Blocks, such as Create, used in TIBCO Scribe® Online Maps have a configuration setting labeled Process this operation in batches of. The value of this setting instructs the TIBCO Scribe® Online Agent to group and cache a set of source records into a batch before sending that data to the CRM Connector.  For more information see Batch Processing in the TIBCO Scribe® Online Help.
 
When the CRM Connector gets the group of records, the settings in the Connection Dialog for Dynamics CRM dictate how the records are sent. 
 

Bulk Batch Size

This Connection dialog setting configures the number of records to send in a single message. Based on our testing, Dynamics CRM accepts a maximum of 1000 per call.  However, trial Organizations and some basic deployments of a Dynamics CRM organization timeout before processing 1000 records.  Therefore, the size must be reduced until records process consistently. 

"Maximum batch size – there is a limit to how many requests can be added to a request collection. If that limit is exceeded, a fault is thrown before the first request is ever executed. A limit of 1000 requests is typical though this maximum amount can be set for the Microsoft Dynamics 365 deployment. The deployment setting for this limit is BatchSize." 


For additional information, see the Run-time limitations section of  Use ExecuteMultiple to improve performance for bulk data load in the MSDN Help.

 

Bulk Concurrent Calls

This TIBCO Scribe® Online Connection dialog setting configures the number of threads to use when sending multiple messages at the same time.  Threads are not used unless dictated by the size.  The maximum number of threads that can be used is 32.  Even if the thread count is at 15, it doesn’t mean that all 15 threads are used.  It all depends on the batch size mentioned in the Bulk Batch Size section of this article.   

In our testing, more concurrent calls with lower batch sizes appears to run faster. If the system can process it, we have found that 32 simultaneous single record calls is the fastest.  If CRM Online does not tolerate these settings, you see Generic SQL errors, which are an indication that the concurrency is set too high.

Trial and basic online organization installations are historically capped at 2 concurrent requests.  The following is the constraint that sets this max size for the ExecuteMultipleRequest message.

"Throttling of concurrent calls – for Microsoft Dynamics 365 (online) there is a limit of 2 concurrent ExecuteMultipleRequest executions per organization. If that limit is exceeded, a “Server Busy” fault is thrown before the first request is ever executed. For an on-premises deployment, throttling is not enabled by default. The deployment setting for this limit is ExecuteAsyncPerOrgMaxConnectionsPerServer."


For additional information, see the Run-time limitations section of  Use ExecuteMultiple to improve performance for bulk data load in the MSDN Help.
 

Example Use Cases 

1. Source query returns 3000 records, Target Operation Block batches parameter is set to 2000.  The Connection dialog Batch Size setting is set to 500 with Bulk Concurrent Calls or threads set to a maximum of 10 threads.  In this case, since the Agent is processing 2000 records a pass, only 4 threads would be used – each sending 500 for a single pass. The second pass would use 2 threads each sending 500.
2. Source query returns 3000 records, Target Operation Block batches parameter is set to 5000. The Connection dialog Batch Size setting is set to 500 with Bulk Concurrent Calls or threads set to a maximum of 10 threads.  In this case, 6 threads would be used.
3. Source query returns 3000 records, Target Operation Block batches parameter is set to 5000.  The Connection dialog Batch Size setting is set to 100 with Bulk Concurrent Calls or threads set to a maximum of 10 threads.  In this case, only 10 threads would be used.

Note that these are only examples. It may take a few tries to find the right combination of batch size and concurrent calls that works for your Organization.
 

Map level and Connection dialog settings that allow your TIBCO Scribe® Online Organization to take advantage of features designed to optimize your Solutions that use the Dynamics 365 Connector.