Error opening CSV file after FileMonitor reports file exists
book
Article ID: KB0074740
calendar_today
Updated On:
Description
Seen on the Windows 10 operating system, a copy operation of a large file does not complete before the operating system alerts the FileMonitor Adapter that the file exists. The file is not ready to be read and the File Reader Adapter reports an error like:
2017-01-13 16:04:19.555-0500 [OperatorThread(default.InputAdapter2:1)] WARN InputAdapter2 - Error opening CSV file 'data.csv': com.streambase.sb.operator.ResourceNotFoundException: Resource "data.csv" cannot be found
The structure of the application is:
Resolution
Solutions:
A. Prevent the FileMonitor from seeing the file until copy is complete.
Steps:
1. Copy the large file to the same filesystem that the FileMonitor is watching, but to a different directory.
2. Rename (move) the file to the monitored directory.
The FileMonitor will send an event when the file appears in the monitored directory, but this file is already complete and closed in the filesystem and ready to be read.
B. Get an error from the CSV File Reader and loop back after a short delay to try again.
To do this, enable a status output port on the CSV File Reader. When the "ResourceNotFoundException" occurs, and event will come out this port and can trigger a second try after a delay. The easiest way to insert a delay without impacting the rest of the running application is to use a concurrent (running in own thread) Map operator with the expression: add, delay_field, sleep(seconds(10)).
C. Between the FileMonitor and the File Reader adapters, insert a fixed delay that will accommodate the time it takes to finish the copy operation.
This is the simpler version of option "B." which is to place a concurrent Map operator with a delay between the FileMonitor and the CSV File Reader. The delay should be about 20% longer than the longest it might take the operating system to complete the copy operation. You will need to determine this through testing. The delay expression is (example):
sleep(seconds(10))
Issue/Introduction
This is a race condition with several solutions.
Feedback
thumb_up
Yes
thumb_down
No