Managing large XML files with TIBCO BusinessWorks.
Article ID: KB0091941
Updated On:
| Products | Versions |
|---|---|
| TIBCO ActiveMatrix BusinessWorks | - |
| Not Applicable | - |
Description
Resolution:
Background
1). Use the TIBCO BusinessWorks Large XML plugin.
2). Otherwise, consider the following solution within BusinessWorks.
Like the scenario (Read File and then a Parse XML) i.e., reading a large file and then parsing the file; Read File cannot read part of a file.
Read the full file.> parse it > using mapper to map it to a smaller schema.
TIBCO BW uses the SAX parser. The max size of the XML message can only be determined by trial and error. The values of the Java heap size, Max jobs, Flow limit and Activation limit affect the performance and capability of BW.
===============================================================================================================================
Resolution:
1). Allocate more heap memory.
2). Turn on MemorySavingMode:
****** MemorySavingMode: Release reference to unused process data *******
The BW engine currently holds on to each piece of process data through the life of a job even if it is used once at the beginning and then never reused. Setting the specified property to 'true' will force the BW engine to release the process data as soon as it is no longer needed. This mode is off by default. To turn this mode on, set the following property:
EnableMemorySavingMode= true
This mode is recommended for processes that have large amounts of data. It is turned off by default because it does add some computational overhead. In order to make sure this mode is available for deployed projects, add the following lines to the bwengine.xml file under [TIBCO_HOME]\bw\5.x\bin:
<property>
<name>Memory Saving Mode</name>
<option>EnableMemorySavingMode</option>
<default>true</default>
<description>controls process variable memory saving mode</description>
</property>
It is not advisable to parse very large XML files into a single in-memory tree. The parse XML activity will parse the whole document at one time, and this may not be reasonable if you have very large XML files. However, large XML file can be handled in smaller pieces using some custom parsing code. Depending on your experience and the resources you have available, we would recommend that you use the SAX API to Xerces (which is the Java parser used by BW, documented at http://www.saxproject.org) to parse the file. Implement the interface ContentHandler during parsing and write out a series of smaller XML documents with an appropriate number of records. Determining the proper number of records to process in a batch would depend on what you want to do with the data after parsing and would likely take some testing to determine an optimal amount.
Using the SAX parser has the advantage of treating the file as XML, ensuring the well-formness and validity of the input and output data. There are future product plans to better accommodate very large XML files that will address concerns about handling large XML files, without the need for custom code.
Background
1). Use the TIBCO BusinessWorks Large XML plugin.
2). Otherwise, consider the following solution within BusinessWorks.
Like the scenario (Read File and then a Parse XML) i.e., reading a large file and then parsing the file; Read File cannot read part of a file.
Read the full file.> parse it > using mapper to map it to a smaller schema.
TIBCO BW uses the SAX parser. The max size of the XML message can only be determined by trial and error. The values of the Java heap size, Max jobs, Flow limit and Activation limit affect the performance and capability of BW.
===============================================================================================================================
Resolution:
1). Allocate more heap memory.
2). Turn on MemorySavingMode:
****** MemorySavingMode: Release reference to unused process data *******
The BW engine currently holds on to each piece of process data through the life of a job even if it is used once at the beginning and then never reused. Setting the specified property to 'true' will force the BW engine to release the process data as soon as it is no longer needed. This mode is off by default. To turn this mode on, set the following property:
EnableMemorySavingMode= true
This mode is recommended for processes that have large amounts of data. It is turned off by default because it does add some computational overhead. In order to make sure this mode is available for deployed projects, add the following lines to the bwengine.xml file under [TIBCO_HOME]\bw\5.x\bin:
<property>
<name>Memory Saving Mode</name>
<option>EnableMemorySavingMode</option>
<default>true</default>
<description>controls process variable memory saving mode</description>
</property>
If testing from Designer, you can put this property into a properties.cfg file and load it when running the tester. To load the cfg file, when starting a process a dialog box appears. Go to "Advanced" and specify "-p <the path and cfg file>" in the Test Engine User Args).
3). Control your process starter to make sure only one job starts at a time for the BW process which contains the Parse XML activity.
4). After applying a), b) and c), if you still see memory issues, you may need to consider the following:
3). Control your process starter to make sure only one job starts at a time for the BW process which contains the Parse XML activity.
4). After applying a), b) and c), if you still see memory issues, you may need to consider the following:
Internally, we use the SAX API to parse XML, but we do not give the user control over the output. We build our own internal structure representing the entire XML tree (similar to DOM), so very large XML documents will require significant memory.
It is not advisable to parse very large XML files into a single in-memory tree. The parse XML activity will parse the whole document at one time, and this may not be reasonable if you have very large XML files. However, large XML file can be handled in smaller pieces using some custom parsing code. Depending on your experience and the resources you have available, we would recommend that you use the SAX API to Xerces (which is the Java parser used by BW, documented at http://www.saxproject.org) to parse the file. Implement the interface ContentHandler during parsing and write out a series of smaller XML documents with an appropriate number of records. Determining the proper number of records to process in a batch would depend on what you want to do with the data after parsing and would likely take some testing to determine an optimal amount.
Using the SAX parser has the advantage of treating the file as XML, ensuring the well-formness and validity of the input and output data. There are future product plans to better accommodate very large XML files that will address concerns about handling large XML files, without the need for custom code.
Issue/Introduction
Managing large XML files with TIBCO BusinessWorks.
Additional Information
Feedback
Yes
No
Powered by
