Under what conditions does a file poller miss a file in the pointed directory?
Article ID: KB0086317
Updated On:
| Products | Versions |
|---|---|
| TIBCO ActiveMatrix BusinessWorks | - |
| Not Applicable | - |
Description
Resolution:
We have seen fe cases where file poller could miss new files.
1). When old and new files have the same properties.
2). When external process which creates/copies the file still has the lock.
3). Larger size files.
First case:
File poller keeps track of processed files in an internal list. At every polling interval it checks for files and creates jobs based on the configured criteria. The internal processed files list will be updated on every run. File poller will not pick up new files if it matches the following properties with previously processed files:
1). File Name
2). File Size
3). Last Modified Date
In normal cases, the processed files are being removed by another activity. The previously processed files internal list will be cleared periodically and the new files will be processed. The problem only arises when the same files are copied before the internal list is cleared.
Second case:
A transient state between writing/closing the file writer stream and the file poller reading the directory list. Implement a procedure which first writes the files to the directory but with a name which will not match the filter mask, and then rename/move the file to a name which will match the filter mask.
In both cases, to overcome this issue, the file name or last modified date can be changed so that it will not match the previously processed files using the touch command and/or mv (move) command or equivalent for Windows. This should be integrated into the external process which is creating the files.
Third case:
Copying a very big file into the target folder. In this case, it is possible that the copying is not finished in more than one poller interval, so the system may think it is an old file and will not trigger a new process. For this case, the possible solution is to use a longer poller interval to make sure that the file can be copied in one interval.
If polling large size files from the folder, it is suggested that you use file adapter instead of file poller, which will work in this scenario.
We have seen fe cases where file poller could miss new files.
1). When old and new files have the same properties.
2). When external process which creates/copies the file still has the lock.
3). Larger size files.
First case:
File poller keeps track of processed files in an internal list. At every polling interval it checks for files and creates jobs based on the configured criteria. The internal processed files list will be updated on every run. File poller will not pick up new files if it matches the following properties with previously processed files:
1). File Name
2). File Size
3). Last Modified Date
In normal cases, the processed files are being removed by another activity. The previously processed files internal list will be cleared periodically and the new files will be processed. The problem only arises when the same files are copied before the internal list is cleared.
Second case:
A transient state between writing/closing the file writer stream and the file poller reading the directory list. Implement a procedure which first writes the files to the directory but with a name which will not match the filter mask, and then rename/move the file to a name which will match the filter mask.
In both cases, to overcome this issue, the file name or last modified date can be changed so that it will not match the previously processed files using the touch command and/or mv (move) command or equivalent for Windows. This should be integrated into the external process which is creating the files.
Third case:
Copying a very big file into the target folder. In this case, it is possible that the copying is not finished in more than one poller interval, so the system may think it is an old file and will not trigger a new process. For this case, the possible solution is to use a longer poller interval to make sure that the file can be copied in one interval.
If polling large size files from the folder, it is suggested that you use file adapter instead of file poller, which will work in this scenario.
Issue/Introduction
Under what conditions does a file poller miss a file in the pointed directory?
Feedback
Yes
No
Powered by
