Resolution:
To pass in java options to RTView, use RTV_JAVAOPTS.
For example, for java heap size you can use the following:

RTV_JAVAOPTS=-Xmx256m

Here’s what's mentioned in the documentation:

“Java options specified in RTV_JAVAOPTS will be used by all of the Enterprise RTView run scripts. These will be passed into the java process from the Enterprise RTView run scripts. These will not be used when Enterprise RTView applications are run from the Windows Start menu. This environment variable is optional.”