NetApp devices have a snapshot feature (enabled by default) that captures snapshots of changed files stored on the NAS. This feature is not supported in LogLogic LMI. Note that this is not the same thing as SnapLock feature which is supported by LogLogic LMI. NetApp exposes the snapshot data to the NFS client in a directory at the root of the file share called .snapshot. Because of the leading period character that directory is hidden unless you execute an ls -la on the /loglogic/data/archive/mount_1/vol1 directory like shown below:

Logapp root:/loglogic/data/archive/mount_1/vol1 0$ ls -la
total 5
drwxrwxrwx 3 logapp logapp    3 Sep 25 15:38 .
drwxr-xr-x 3 root   root   4096 Sep 25 15:31 ..
drwxr-xr-x 4 root   logapp    4 Sep 25 14:15 .snapshot
drwxr-xr-x 4 root   logapp    4 Sep 25 14:15 2018
Logapp root:/loglogic/data/archive/mount_1/vol1 0$

The /loglogic/bin/index_remove program is called by engine_archive to purge index data that is out of retention. If there is any index data that has been moved to the archive storage then the index_remove program must traverse the directory structure. However if the .snapshot directory exists this can cause a problem for the index_remove program because it's not expecting to traverse that directory structure. It will generate errors like shown below:

[snip]
2017-08-17 17:00:53,310 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/11
2017-08-17 17:00:53,312 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/11
2017-08-17 17:00:53,314 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/05
2017-08-17 17:00:53,315 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/05
2017-08-17 17:00:53,317 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/06
2017-08-17 17:00:53,319 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/06
2017-08-17 17:00:53,320 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/10
2017-08-17 17:00:53,322 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/10
2017-08-17 17:00:53,324 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/07
2017-08-17 17:00:53,326 INFO [main] [com.loglogic.logapp.idxcore.util.IndexerUtils] - Cannot parse calendar string: .snapshot/hourly.5/2017/07/21/07
[snip]

These snapshots can cause the index_relocate and index_remove programs to spend hours trying to traverse these directories; the longer the snapshot feature is enabled the more time it has to make snapshots of data and therefore the more directories for the index programs to traverse. While the index_remove/index_relocate program is doing this the engine_archive process will not be archiving or purging anything else. This can cause the local storage of the appliance to become full because, although engine_archive can still archive/purge some data it will be a fraction of the data it would otherwise be able to process because each iteration through the archive loop will require waiting on the aforementioned index programs to finish completing their tasks of traversing these snapshot directories. The longer the snapshot feature is enabled in NetApp the more impact it will cause.

You may also see errors like this because the .snapshot directories are read-only (set by NetApp):
2018-09-27 16:50:14,426 ERROR [main] [com.loglogic.indexer.common.IndexPurger] - Cannot purge index directory: /loglogic/data/archive/mount_2/vol1/.snapshot/log_snap_1520979786_1538053203_490/2018/08/15/0800index/0
java.io.IOException: Unable to delete file: /loglogic/data/archive/mount_2/vol1/.snapshot/log_snap_1520979786_1538053203_490/2018/08/15/0800index/0/_2d.cfs
    at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1390)
    at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1044)
    at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:977)
    at com.loglogic.indexer.common.IndexFileUtils.deleteDirectory(IndexFileUtils.java:36)
    at com.loglogic.indexer.common.IndexPurger.purgeIndexDirectory(IndexPurger.java:65)
    at com.loglogic.indexer.index.IndexMaintenanceTask.purgeIndexDir(IndexMaintenanceTask.java:541)
    at com.loglogic.indexer.index.IndexMaintenanceTask.rebuildOrphanIndexInfos(IndexMaintenanceTask.java:747)
    at com.loglogic.indexer.index.IndexMaintenanceIndexer.runDeleteExpiredDocuments(IndexMaintenanceIndexer.java:216)
    at com.loglogic.indexer.index.IndexMaintenanceIndexer.run(IndexMaintenanceIndexer.java:138)
    at com.loglogic.indexer.IndexLauncher.launch(IndexLauncher.java:89)
    at com.loglogic.indexer.IndexRemoveCommand.main(IndexRemoveCommand.java:37)
 

Only LMI appliances with archiving enabled using NetApp storage devices as the NFS server. This applies whether indexing is enabled or not because the affected program areas are executed in either scenario.

This incompatibility between LMI and the snapshot directory is resolved in LMI 6.3.0 and higher. For users still using older versions listed in the version field above you can follow TIBCO's recommendation which is to disable the snapshot feature on the NetApp volume. You can also hide it by being completely invisible to clients but that has disadvantages as described below. Disabling can be accomplished using this command:

netapp::> vol modify -vserver mynfssrv1 -volume vol_logs_1234 -snapdir-access false
Volume modify successful on volume vol_logs_1234 of Vserver mynfssrv1.

Hiding the snapshot directory can be accomplished using the procedure documented on NetApp's site and provided below. The disadvantages to this method are:
1. It's completely hidden from the client (i.e. LMI appliance) which can make troubleshooting more complicated when item #2 occurs
2. the hidden snapshot directories steal storage away from the file share for storing the snapshot data. This can cause a 5TB share to only have, for example, approx. 3 TB as usable storage because the other 2TB are consumed by snapshot data. This causes the du command on LMI to differ in results from the df command. The du command will only show what's actually used by what LMI can see but the df command will reflect the real usage by including the snapshot data. This can be confusing when troubleshooting issues when LMI starts performing emergency purging even though the du command will not reflect usage being high enough to trigger the purging. The only way to see the snapshot data usage in this configuration is to view the NetApp interface. If you still want to proceed with hiding the snapshots then follow the procedure below.

A feature enabled on NetApp systems can cause "cannot parse calendar string" errors in the index-remove log because the index_remove program is not expecting the directory structure to exist. The index_relocate program is also affected.