Salesforce has strict API limits. If they are exceeded, any TIBCO Scribe® Online Solutions using Salesforce Connections may not be able to run. This document provides excerpts from the TIBCO Scribe® Online Connector For Salesforce help that specifically address API limits and batch processing.
 

API Usage Limits

Salesforce limits the number of API calls in any 24-hour period for every organization.

Note: Refreshing metadata for the TIBCO Scribe® Online Connector consumes API calls.

If you are an administrative user, you can view the number of API requests your Salesforce organization has used in the last 24 hours, as follows:

1. From the Salesforce User menu, select Setup. The Setup menu displays on the left side of the page.
2. Select Administer > Company Profile > Company Information.
3. The API Requests, Last 24 Hours field displays the current number of API requests along with the maximum available to your organization.

 

Bulk API vs Web Services API

There are two methods for integrating data into or replicating data from Salesforce, Bulk API or Web Services API. In general, using the Bulk API is slower but uses fewer API calls than the Web Services API.

• Salesforce Bulk API is optimized to load large amounts of data with a minimal number of API calls. The Bulk API can process up to 10,000 records in a batch and up to 2000 batches per 24-hour period.
• The Web Services API processes sets of 200 records at a time.

Example:

If you select the default batch size of 2000 records, for each batch:

• The Bulk API uses at least three API calls, one each to start, run, and end the batch. For this example, the Bulk API would use only three API calls.
• The Web Services API creates 10 sets of 200 records each, which uses at least 12 API calls. One call to start, one for each 200 record set, and one to end the process.

Note: If you constantly integrate or replicate large amounts of data, you may want to test both methods to determine which method works best for your site.

Note: Due to unpredictable Salesforce limits, you may receive errors from Salesforce when trying to use either the Bulk API or batch processing. In these cases, you must use the non-batch method to integrate your data.
 

Enabling Salesforce Bulk API For Replication Services (RS) Solutions

When you select the RS - Use Salesforce Bulk API for initial replication check box from the TIBCO Scribe® Online Connection dialog for Salesforce, the Salesforce Connection uses the Salesforce Bulk API, rather than the Salesforce Web Services API. The Bulk API can be especially useful for initial replication. It may be slower than the Web Services API, but the Bulk API requires far fewer Salesforce API calls.

 

Replicating Data Within Salesforce API Usage Limits

Depending on your edition of Salesforce, your TIBCO Scribe® Online Replication Solution may exceed the Salesforce API call usage limit for your Salesforce account.
Managing the Salesforce API call usage limit:

• If your replication reaches the API call limit threshold, it will most likely be during your initial replication run. After the initial replication, your Solution only updates records that have changed, which requires fewer API calls.
• If your replication is stopped by the API call limit, the replication restarts where it left off once Salesforce resets your API usage meter. Typically, the usage meter is reset every 24 hours. No data is lost if the replication stops and restarts.

Preventing your site from reaching the API call limit:

• Consider using the Salesforce Bulk API feature by selecting the Use Bulk API for initial replication option from the TIBCO Scribe® Online Connection dialog for Salesforce. This feature optimizes Salesforce for loading large sets of data by allowing TIBCO Scribe® Online to insert a large number of records asynchronously. Using the Bulk API feature is generally slower, but uses fewer API calls. Using the Web Services API is faster, but uses more API calls. See Bulk API vs Web Services API.
• Break your initial replication up into smaller chunks of data. This may be especially helpful for the initial replication. Use the Selected Entities feature of the Select Source Entities page of the RS Solution. Each day, replicate a limited number of entities, for example, 10 entities on day 1, 10 more entities on day 2, and so on.
• Balance the frequency of your replications with your available number of API calls. Each time a Solution runs, it generates a certain number of API calls. For example, let's say each time a replication runs we generate a minimum of 100 API calls. In any 24-hour period, running the replication once an hour generates 24 x 100=2400 API calls. Running the same replication every 15 minutes generates 24 x 4 x 100=9600 API calls.
• Only enable the RS - Automatically refresh metadata on every run option on the TIBCO Scribe® Online Connections dialog when you have modified the schema, for example, by adding custom fields. Disable it after the replication is complete to prevent it from refreshing the metadata the next time the replication runs.

For more information, see API usage metering under Implementation Considerations in the Salesforce.com Web Services Development Guide.
 

Batch Processing For Salesforce IS/MS Solutions

• When integrating data into a Salesforce target or using Salesforce as a source, TIBCO Scribe® Online provides two methods to process records in batch mode:
  • Salesforce Web Services API with batch processing
  • Salesforce Bulk API
• For Integration Maps, you can set batch processing options on both Source and Target Blocks.
• The Upsert Operation can be used with batch processing to reduce the number of records processed.

 

Using The Web Services API With Batch Processing

To use the Web Services API for batch processing, make sure that the IS - Use Salesforce Bulk API option is not selected on the TIBCO Scribe® Online Connection dialog for your Salesforce Connection.
 

Using The Bulk API With Batch Processing

Make sure that the IS - Use Salesforce Bulk API option is selected on the TIBCO Scribe® Online Connection dialog for your Salesforce Connection and that batch processing is enabled in the Map. See Enabling Batch Processing below for additional information.

Note: If you select the Bulk API, but do not select batch processing in the map, the Bulk API is ignored and records are processed one at a time using the Salesforce Web Services API.
 

Enabling Batch Processing

After you have specified whether to use the Bulk API, you need to enable batch processing, and specify the batch size for specific Blocks within an Integration Map:

1. From the TIBCO Scribe® Online Connection dialog for Salesforce select IS/MS - Use Salesforce Bulk API for batch operations.
2. Begin designing your Integration Map.
3. For any Blocks where Salesforce is your Source Connection, select the Process this operation in batches check box. Typically this is a Target block.
4. From the Edit Block Properties dialog page, specify the batch size.

 

Salesforce Batch Size

The size of a batch can be between 1 and 10,000 records. The default size is 2000. Larger batches use fewer Salesforce API calls, but, especially if the individual records are very large, Salesforce processing may time out.

See the Bulk API and Web Services documentation on the Salesforce.com website for details about how Salesforce processes large amounts of data.
 

Matching Criteria And Bulk API

In the images above, Bulk API was used with an Update block. To keep Salesforce API hits to a minimum the match field on the update must be to the primary key field of the Salesforce target entity.

That assumes the source data contains the primary key information for the target. In Salesforce this is often not the case. Especially for one-way integrations the source unique field is stored in an External ID field in Salesforce.

If you are using Bulk API and you match on the External ID field as shown below, there will be a least 1 API hit for each lookup.  This scenario eliminates the benefits of using the Bulk API.

 

 

Use Upsert

In the scenario outlined above, Upsert should be used instead of Update in conjunction with Bulk API. An Upsert performs either an Insert or Update depending on whether or not the source field mapped to the External ID field exists in Salesforce.

The main difference from a configuration point of view is that there is no matching field involved in Upsert as there is in the Update block. It is implied to be the External ID field.

If there is more than one External ID field that is mapped - Only the longest living external field based on the Salesforce creation date of the field, is used to determine if an update or insert should occur.

 

For additional information, see the following:

TIBCO Scribe® Online Help: TIBCO Scribe Online Connector For Salesforce, Batch Processing 
 

Salesforce has strict API limits. If they are exceeded, any TIBCO Scribe® Online Solutions using Salesforce connections may not be able to run. This article provides excerpts from the TIBCO Scribe® Online Connector For Salesforce help that specifically address API limits and batch processing.