On the TDV server instance, a "saw-tooth" appearance of memory usage is observed on the memory graph. Over the period of time, TDV slowly consumes several GB's of memory even when there is not much user activity. Eventually, Garbage Collection kicks in returning the memory and then the behavior repeats holding up memory.
Even though Garbage Collection is running but there is an uncertainty about what is holding up the memory.
Issue/Introduction
On all the TDV server instances, a "saw-tooth" appearance of memory usage on the memory graph. Something slowly consumes several gigs of memory, then GC kicks returning the memory and then the behavior repeats holding up memory.
Resolution
Generally the sawtooth pattern is preferred as it indicates that the Garbage collection is actively cleaning up the memory utilized by the java objects. In this case though, memory was being held.
Steps that could potentially help are: Review the Datasources involved and set up the datasource connection pooling to a higher number on the Advanced Section of the datasources. Optimize this setting by running few requests. Increasing the request timeout from the default of 30 seconds to a higher value.