Resolution:
BW Engine pages out the Jobs correctly.
However, since the Java does the Garbage collection on it own and the Garbage collection thread has a low priority, you see the memory growth.
You can adjust some of the Garbage Collection parameters.
For e.g you can use the following property in your tra file -
java.extended.properties=-Xincgc  -XX:NewSize=128m  -XX:MaxNewSize=128m  -verbose:gc

Java manages garbage collection using generations, or memory pools holding objects of different ages. Collection is performed for a generation when the memory assigned to that generation is filled up. Therefore, collection frequency is determined by the amount of memory assigned to generations. After setting total Java heap size, you should also estimate the amount of memory required for the young generation.

The young generation is the first generation, where new objects are created. Collections of objects in the young generation are called minor collections. By avoiding frequent minor collections, engine throughput can be increased.

To set the young generation heap size, use the following parameters:
     NewRatio  The ratio between the amount of memory for the young generation and all other generations. This is the default parameter used to size the young generation.
     NewSize  The lower bound number of megabytes of memory
     MaxNewSize  The upper bound number of megabytes of memory. This value should not exceed one-half the total JVM heap size.

The following values are recommended as a starting point:
java.extended.properties=-XX: NewSize=128m -XX: MaxNewSize=256m

Increasing the new generation size increases engine throughput for larger numbers of short-lived objects. A general guideline for this value is 25% of -Xmx, the maximum heap size. Setting NewSize and MaxNewSize to the same value fixes the young generation memory size to that value.

Do we have any memory issues when BW Engine pages out Jobs?