Resolution:
The startup scripts for TIBCO Administrator and TRA in Solaris sets the file descriptor to 256 (ulimit  -n  256).
This is done only for Solaris environment. For other environments (HP-UX, AIX, Linux) it takes the ulimit value that has been set for the user starting the init scripts.

So, if you see the "Too many open file" error in BW Engine on Solaris, please make sure that you have increased the "ulimit" parameter in the "initTIBCOAdmin" and "initTRA" scripts. These scripts are found in $TIBCO_HOME/tra/1.0/tools/rcscripts  directory.

You will see this issue only when starting BW Engine from the Administrator.

In BW5.1.2,  you will have to edit the "inithawk" and "initTIBCOAdmin" scripts to change the ulimit value.