Resolution:
This error implies that the specified heap size cannot be allocated to the JVM.  Here is a test you could do to determine the max heap size  that can be allocated to the JVM:

1. At the command prompt change directory to /opt/tibco/jre/1.5.0/bin
  
  2. Type the below command to determine the jre version,  
    
     java -version
    
      Example:
      //
       [support@vm-rhel4-64 bin]$ ./java -version
       java version "1.5.0_15"
       Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
       Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
     //
    
  3. Then type the below command to allocate the maximum heap size say 1024 mb
    
     java -Xmx1024M -version java version "1.5.0_15"
    
      //
       Example:
         [support@vm-rhel4-64 bin]$ ./java -Xmx1024M -version java version "1.5.0_15"
         java version "1.5.0_15"
     Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_15-b04)
     Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_15-b04, mixed mode)
      ///
    
     If the specified memory cannot be allocated then following error is thrown:
    
     //
     [support@vm-rhel4-64 bin]$ ./java -Xmx1600M -version      Error occurred during initialization of VM Could not reserve enough space for object heap
                                                               Could not create the Java virtual machine.
     ///
    
    In the test at my end  on  1gb RAM PC, I was not able to allocate more 1570, see below:
    
    //
    [support@vm-rhel4-64 bin]$ ./java -Xmx1580M -version java version "1.5.0_15" Error occurred during initialization of VM
    ///

Following Error is thrown on startup of BE applicaton on UNIX -JVMJ9VM015W Initialization error for library j9gc23(2): Failed to instantiate heap.1G requested Failed to create the Java VM