What is the reason for getting a "java.lang.OutOfMemoryError" when the data is fetched in a variable of type LONGVARCHAR using a script/procedure in TIBCO Data Virtualization?

What is the reason for getting a "java.lang.OutOfMemoryError" when the data is fetched in a variable of type LONGVARCHAR using a script/procedure in TIBCO Data Virtualization?

book

Article ID: KB0070969

calendar_today

Updated On:

Products Versions
TIBCO Data Virtualization All supported versions.

Description

If a TDV script or procedure is fetching data from a web API or a published web service into a LONGVARCHAR type variable, then a "java.lang.OutOfMemoryError" can be observed, and the stack trace is as below:
--------

ERROR [jetty thread pool-2736] 2023-05-31 21:04:43.455 +0530 Util - User Exception Occurred
com.compositesw.common.UserException: An internal error has occurred.
Cause: Java heap space [Log ID: 9b1286b6-1ab5-4990-95eb-4ebc022d79e0] 
    at com.compositesw.common.UserException$Builder.build(UserException.java:165) [cscommon-080502002.jar:?]
    at com.compositesw.cdms.webapi.service.Util.createWebapiException(Util.java:87) [cswebapi-server.jar:?]
    at com.compositesw.cdms.webapi.service.Util.createWebapiException(Util.java:73) [cswebapi-server.jar:?]
    at com.compositesw.cdms.webapi.service.WProcResultImpl.getOutputValues(WProcResultImpl.java:312) [cswebapi-server.jar:?]
    at com.compositesw.cdms.webapi.service.WMetaDataImpl.procresult_getOutputValues(WMetaDataImpl.java:1408) [cswebapi-server.jar:?]
    at com.compositesw.cdms.webapi.service.WMetaDataSvc_Tie.invoke_procresult_getOutputValues(WMetaDataSvc_Tie.java:3201) [cswebapi.jar:?]
    at com.compositesw.cdms.webapi.service.WMetaDataSvc_Tie.processingHook(WMetaDataSvc_Tie.java:21570) [cswebapi.jar:?]
    at com.sun.xml.rpc.server.StreamingHandler.handle(StreamingHandler.java:350) [webservices-rt-2.1.jar:2.1]
    at com.sun.xml.rpc.server.http.JAXRPCServletDelegate.doPost(JAXRPCServletDelegate.java:465) [webservices-rt-2.1.jar:?]
    at com.sun.xml.rpc.server.http.JAXRPCServlet.doPost(JAXRPCServlet.java:119) [webservices-rt-2.1.jar:?]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [servlet-api-3.1-cis-trunk.jar:3.1.0]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [servlet-api-3.1-cis-trunk.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:811) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at com.compositesw.server.services.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:101) [csserver-080502002.jar:?]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:142) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:576) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:222) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1126) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:514) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:184) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1060) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:140) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:212) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:96) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.Server.handle(Server.java:498) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) [jetty-all-9.2.30.v20200428.jar:9.2.30.v20200428]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3745) ~[?:?]
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120) ~[?:?]
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) ~[?:?]
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156) ~[?:?]
    at com.compositesw.common.IO.transfer(IO.java:44) ~[cscommon-080502002.jar:?]
    at com.compositesw.common.IO.readContentsToString(IO.java:188) ~[cscommon-080502002.jar:?]
    at com.compositesw.data.stream.AbstractManagedDataStreamFactory.materialize(AbstractManagedDataStreamFactory.java:60) ~[cscommon-080502002.jar:?]
    at com.compositesw.cdms.ds.rest.RestfulStreamFactory.materialize(RestfulStreamFactory.java:45) ~[?:?]
    at com.compositesw.data.internal.StringValue.materializeValue(StringValue.java:525) ~[cscommon-080502002.jar:?]
    at com.compositesw.data.internal.StringValue.makeDurable(StringValue.java:112) ~[cscommon-080502002.jar:?]
    at com.compositesw.cdms.webapi.service.WProcResultImpl$ProcRef.materializeValues(WProcResultImpl.java:897) ~[cswebapi-server.jar:?]
    at com.compositesw.cdms.webapi.service.WProcResultImpl$InvokerThread.run(WProcResultImpl.java:819) ~[cswebapi-server.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    ... 1 more 

--------

The reason for this is explained below.

Issue/Introduction

This article will explain the reason for getting a "java.lang.OutOfMemoryError" when data from an API or published web service is fetched in a LONG VARCHAR variable in TIBCO Data Virtualization.

Environment

All Supported Environments.

Resolution

The error is "Caused by: java.lang.OutOfMemoryError", which means TIBCO Data Virtualization (TDV) has enough memory, but it's trying to store a text more than its limit.

When using a TDV script to fetch/download a dataset into a variable of type LONGVARCHAR, the problem is that the max length of LONGVARCHAR (the "output" parameter) is 2GB.
So, using this method, the maximum data that can be loaded in TDV at a time is 2GB.

Optionally, a CJP can be designed for this.
If there is a need to work with data higher than 2GB, using a CJP, the data coming in through the input stream can directly be routed to the file output stream, thus eliminating the 2GB limit.
Powered by Wolken Software