Resolution:
the error of “java.lang.OutOfMemoryError: Java heap space” means the java.heap.size.max is not large enough.  

The places to setup this parameter for starting up your Admin Server/Hawkagent are different.
When starting Hawk/Administrator from the command line, they will get it from your /tra/domain/&ltdomainName>/hawkagent_<
domainName>.tra file and \tibco\administrator\domain\&ltdomain name>\bin\tibcoadmin_&ltdomainName>.tra on your deployment machine.
While starting Hawk/Administrator from the Windows Servces, they will get it from your Windows Registry like:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TIBHawkAgent-&ltdomain name >\Parameters\ java.heap.size.max
and
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TIBCOAdmin-&ltdomain name>\Parameters\java.heap.size.max

1. When starting Hawk/Administrator from the command line:

If you are seeing out of Memory Errors in Administrator.log  then increase
the current java.heap.size.max settings value in your tibcoadmin_&ltdomainName>.tra
file to a higher value and restart the Admin Server. The possibility of such errors
occurring in Admin log is more in a File Based Domain, increasing it to a value of
1024M is recommended for  larger domains.

If you are seeing the out of memory errors in your Hawk or tsm.logs then increase
the java.heap.size.max settings in your /tra/domain/&ltdomainName>/hawkagent_<
domainName>.tra file which is on your deployment machine and restart the
Hawkagent.

2. When starting Hawk/Administrator from the NT service and deploying BWC-only applications:

If you are seeing out of Memory Errors in Administrator.log then increase
the current java.heap.size.max settings value in your Windows Registry parameter of
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TIBCOAdmin-&ltdomain name>\Parameters\ java.heap.size.max to a higher value and restart the Admin Server.

If you are seeing the out of memory errors in your Hawk or tsm.logs then increase
the java.heap.size.max settings in your Windows Registry parameter of
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\TIBHawkAgent-&ltdomain name >\Parameters\ java.heap.size.max which is on your deployment machine and restart the
Hawkagent.

Note: Choosing an optimum heap size to your Admin Server/Hawkagent also depends upon
your machine  resources. It is very important to set the maximum heap size to a value that
allows the heap to  be contained within physical memory. If the heap exceeds the
physical memory, then heap can begins swapping to disk which could cause
performance to decrease.