The Batch Processing feature, also known as Bulk Processing, allows an Agent to group and cache a set of source records into a batch before sending that data to the Target Connector. When the Target Connector receives the batch of records it uses the Target Connector's API methods for bulk, batch, or asynchronous processing to complete the appropriate operation for that set of records. Not all Connector APIs support batch processing. See the diagram below for an overview of how the Agent executes a Map with Batch Processing enabled.
 

For Connectors that support batch processing, such as Salesforce, performance is improved significantly because their APIs accept batches of data and can process the data set much faster. However, if you use target Lookup functions or Lookup blocks in your Maps, performance improvements derived from using batch processing are reduced due to the time it takes to look up data in the target for each record. See Lookups And Batch Processing, Lookup Block, and Lookup Functions.

If you configure a batch size limit to be larger than your target application permits, most Connectors correct the batch to the largest possible size accepted by the API during processing. For example: if you have 2,000 source records and the Upsert Block is set to a batch size of 2,000 for the Salesforce SOAP API, TIBCO Scribe® Online sends 10 batches of 200 records.

 

Benefits Of Batch Processing
 

• Improve performance when there are a large number of source records to process.
• Reduce the number of round-trips between TIBCO Scribe® Online and the target datastore.
• Reduce the number of API calls to the target application. Some applications, such as Salesforce, limit the number of API calls per customer.

Batch Processing Issues
 

• Batch processing does not reduce the number of records processed. See Counting Processed Records.
• Data fields and Result Fields from Blocks with batch processing enabled cannot be used in subsequent Blocks, because the data may not have been stored when those Blocks execute.
• The Lookup Block does not support batch processing. Batch processing provides the greatest benefit when all Blocks in the Map are using it. If your Map includes a Block that doesn't, such as a Lookup, the benefit is reduced. This is also true if any of your field mappings use a target lookup function.

Enabling Batch Processing
 

Batch processing can be enabled only if the target Connector supports batch processing. This option can be enabled for the following TIBCO Scribe® Online operations:

Integration Maps

Enable batch processing on the General tab of the following operation blocks.

• Create Block
• Delete Block
• Update Block
• Upsert Block

This article provides an overview of the TIBCO Scribe® Online Batch Processing feature, the pros and cons of using Batch Processing, and where to enable it.