Resolution:
Description:
============
TIBCO application crashes with IOException or FileNotFoundException with the exception message "Too many open files in system".


Environment:
===========
TIBCO Administrator 5.x.x
TIBCO Runtime Agent 5.x.x
Linux all version


Symptoms:
=========
The exception stack may look like the following:
<----------
2012 Apr 28 19:09:34:214 GMT -4 Error [ApplicationConfiguration] AESDKJ-0000 [main]  com.tibco.pof.entity.EntityRuntimeException: java.io.IOException: Too many open files in system
    at com.tibco.pof.administrator.model.impl.AdministratorPlugInFileImpl.getClassLoader(AdministratorPlugInFileImpl.java:98)
    at com.tibco.pof.admindomain.impl.DeployableSoftwareImpl.getClassLoader(DeployableSoftwareImpl.java:103)...
Caused by: java.io.IOException: Too many open files in system
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.checkAndCreate(File.java:1704)
    at java.io.File.createTempFile(File.java:1792)
    at com.tibco.pof.util.TempFileManager.createTempFile(TempFileManager.java:126)
    at com.tibco.pof.util.TempFileManager.createTempFile(TempFileManager.java:121)
    at com.tibco.pof.administrator.model.impl.AdministratorPlugInFileClassLoader.createTempFile(AdministratorPlugInFileClassLoader.java:125)...
---->
or
<-----
2012 Apr 28 19:16:46:854 GMT -4 Error [ApplicationConfiguration] AESDKJ-0000 [http-8080-Processor4]  com.tibco.objectrepo.OperationFailedException: Error retrieving stream for VFile .folder URI=
    at com.tibco.archive.impl.filevff.FileVFileStream.getInputStream(FileVFileStream.java:32)...
Caused by: java.io.FileNotFoundException: &ltFileName>  (Too many open files in system)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.&ltinit>(FileInputStream.java:106)
    at com.tibco.archive.impl.filevff.FileVFileStream.getInputStream(FileVFileStream.java:29)
    ... 47 more
Underlying cause:
java.io.FileNotFoundException: &ltFileName>  (Too many open files in system)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.&ltinit>(FileInputStream.java:106)
    at com.tibco.archive.impl.filevff.FileVFileStream.getInputStream(FileVFileStream.java:29)...
----->


Cause:
======
The Maximum Number Of Open Files / File Descriptors (FD) was reached.


Resolution:
========

1). Check the "open files" number with thecommand "ulimit -a" and increase it.

2). Increase system-wide limit on the number of open files via the command:
        echo 943718 > /proc/sys/fs/file-max
  
    Or run the following command to make the change permanent:
        sysctl -w fs.file-max=943718  

More information:
/proc/sys/fs/file-max
   This file defines a system-wide limit on the number of open files for all processes.
/proc/sys/fs/file-nr  

The three values in "file-nr" denotes the number of allocated file handles, the number of allocated but unused file handles, and the maximum number of file handles. You can monitor this to keep a check on the file handles.


References:
==========
http://www.kernel.org/doc/Documentation/sysctl/fs.txt
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/