Resolution:
Review the BE project and check if all the RV destinations are you using has MaxEvents configured as 0? That means unlimited.

We have seen memory improvements if you set the MaxEvents to some value e.g. 5000 for your RV destination in BE.

The MaxEvents parameter defines:  the number of events that a queue can hold—either to curb queue growth, or implement specialized dispatch semantics.
Zero specifies an unlimited number of events; in this case, the strategy must be LimitPolicyStrategy.DiscardNone.

You can find more details about these parameters at TIBCO Rendezvous Concepts / Chapter 7 Events

Note if you set it to smaller value than expected load you would see RV advisory messages in BE logs / nohup output on the subject _RV.WARN.SYSTEM.QUEUE.LIMIT_EXCEEDED.&ltqueue_name>
If the event queue exceeded its event limit,  it will start discarding the incoming events further.

Please try setting it to the appropriate value for the main RV listener as per your load test.

*** Note: we recommend reproducing this memory growth issue and the above suggestion in your TEST setup first before trying it in production. ***