LogLogic LMI may re-collect and retain data that has already been purged off the appliance and will subsequently be visible in searches/reports after the re-collection. This article explains why that happens and how to prevent it.

When using file transfer rules there are two things that you must do:
1. You must ensure the file transfer history retention in LMI is longer than the raw data retention period as defined in LMI for the file-based sources. See article 000031531 for changing this for ST appliances. For LX/MX appliances go to Administration->System Settings->Data Retention tab.

2. You must have a clean-up process that removes collected files from the source's log directory, or from the intermediate file server's directory, if that is being used instead of having LMI connect directly to the sources.

If the transfer history retention is shorter than the raw data retention specified for these sources in LogLogic LMI then LMI will retain the data when it collects it again the first time after the transfer history has been removed. However, even if the file transfer history is shorter than the raw data retention specified for the source, the data cannot be retained by LMI if it cannot re-collect it in the first place. The easiest way to ensure LogLogic LMI cannot re-collect the data is to remove it from the source. The purge from the source should occur after some safe interval beyond when LMI should have successfully collected it to ensure data that has not been collected is not yet purged from the source.

Another advantage of timely purging of data on sources is to reduce wasted time and resources downloading unchanging log files. For file transfer rules that have been configured with a wildcard, for example "*.log", LMI will collect every file that matches the wildcard criteria as long as the files exist in the specified directory. LMI will discard them if they have not changed but if they are removed from the source then LMI will not waste time and resources transferring the data.

File transfer history information is simply made up of metadata rather than actual event data so the amount of storage the history consumes is minimal. Therefore having a large retention value set for that does not impact the system in most cases. Unlike the data retention policies, there is a single global setting for the file transfer history information so if you have multiple different retention policies defined for various file-based sources then you will need to be aware of what the longest retention policy is when configuring the file transfer history retention. If the longest raw data retention is 1 year then ideally your file transfer history should be at least 1 year as well.

The file transfer history is metadata that is stored for each file that is collected by file transfer rules by each LogLogic LMI appliance, even if the files are not specifically named, such as when a wildcard is specified for the path in a file transfer rule. For each file LMI will track a few pieces of information, the most important of which is the checksum and last modified time because these are used to determine if the file has changed.