How to avoid out of memory issues when we have to fetch large amount of data using Salesforce QueryAll Activity using batch size and Subset size.

How to avoid out of memory issues when we have to fetch large amount of data using Salesforce QueryAll Activity using batch size and Subset size.

book

Article ID: KB0078240

calendar_today

Updated On:

Products Versions
TIBCO ActiveMatrix BusinessWorks Plug-in for Salesforce.com ALL

Description

When we fetch large amount of data using Salesforce QueryAll Activity, we may encounter OutOfMemory issue.

Issue/Introduction

How Batch Size and Subset Size works in Salesforce QueryAll Activity.

Environment

ALL

Resolution

Out of Memory issue when retrieving large amount data can be avoided using appropriate Batch and Subset size in Salesforce QueryAll Activity. Explanation on how Batch and Subset size works for better memory management is explained below:
*************************************************
In a Salesforce QueryAll activity, if we set batch size to 200, and the returned data will contain 500 records. Then internally, the first 200 pieces of data will be returned by Salesforce, and stored in 
local memory, and then another SOAP call will be triggered to get the second 200 pieces, and finally the last 100 pieces. After all the 500 pieces of sObject are all returned by Salesforce, these data will be passed out to other activities, and then the memory for storing the data will be released.

But in some cases, if query request return's a very large amount of data, and if all the data is stored in memory until being passed to other activities, then we may encounter OutOfMemory exception. To avoid this, we need to use subset setting. This configuration will divide the large amount of data into smaller subsets which will be passed out to other activities. So the memory could be released periodically. Say for example, if you set batch size to 200, and set subset size to 2, and you are expecting that 500 pieces of data will be returned at run time. Then when doing the query, the activity will first get 200 pieces, then another SOAP call is made to get the second 200 pieces; at this time, it received 2 batches (which is the defined by the subset size) of data which has been stored in local memory, then the data will be passed outside the activity, and the memory is released. You need to configure a loop surround the query activity. After other activities in the same loop finished the processing for those 400 records, the loop will redirect the flow to the query activity, who will continue to query for the last 100 records.
*************************************************
NOTE
--------
**The value of Batch Size must be set between 200 and 2000. If the input value equals -1, it is set to the default value which is 500.
If the input value is greater than 2000, it is set to 2000. If the input value is less than 200 (except -1), it is set to 200. 

**Coming to the subset size, the value in subset size will only be taken in consideration only if the ProcessInSubsets is set as "True" in the Input tab. Also Subset Size value ranges from 1-10.