When the index retention time is increased does the TIBCO LogLogic LMI appliance go back to index old data?

When the index retention time is increased does the TIBCO LogLogic LMI appliance go back to index old data?

book

Article ID: KB0077892

calendar_today

Updated On:

Products Versions
TIBCO LogLogic Log Management Intelligence all versions

Description

When the index retention time is increased, does the LogLogic appliance go back and index old data?

The answer is yes but only when using default indexing settings. After the index retention time is increased, the appliance will go back to index the old data as it continues indexing recent data.  The mechanism that allows this is the default index algorithm of latestFirst. This is defined in /loglogic/conf/indexer-config.properties. That file contains other settings as well but here are the relevant settings:

# only index oldest N hours
#indexer.loglogicIndexer.indexOption=oldestOnly

# only index latest N hours
#indexer.loglogicIndexer.indexOption=latestOnly

# index oldest, latest, second oldest, second latest... up to N hours
#indexer.loglogicIndexer.indexOption=oldestFirst

# index latest, oldest, second latest, second oldest... up to N hours
indexer.loglogicIndexer.indexOption=latestFirst

As you can see, the default algorithm will place equal priority on both old and new index data. Each time the indexer queries the stDataFiles database table it will do so based on a time criteria involving the length of the index retention. Files that were not already indexed within the new defined retention period will be included in the query results. The indexer will use those results for its next round of data files to index. Technically, the oldestOnly and oldestFirst algorithm settings would also allow older data to be indexed but at the expense of newer data, which may or may not be desired. The only setting that would not include older data would be latestOnly.

If you find that your indexer is not indexing the older data after extending the index retention then, assuming the older data still exists (index retention can't be larger than raw data retention), you can verify your indexer's algorithm is set to the correct value to allow for that. If it is anything other than latestOnly it should be indexing the older data that was not previously indexed. But if you must change the algorithm setting just comment out the setting you don't want to use and uncomment the one you do. You'll have to restart the engine_indexer to make the change take effect by using this command:

$ mtask -s engine_indexer restart

We advise also checking the /loglogic/logs/indexer.log to see what the indexer is currently working on and continue monitoring that log to see if it ever starts to index the older data.

The "N hours" referenced in the indexer-config.properties file is defined by the hourly threads setting in the /loglogic/conf/capability.xml file. Typically the default value of this setting is 2 but it can differ from that depending on the appliance model. The setting looks like this inside that file:
<propertyName>indexer_hourly_threads</propertyName>
                <propertyValue>2</propertyValue>


You'll need to view the capability.xml file and search for your platform type to verify the exact value being used. If your platform section doesn't contain this setting then  the setting's value in the "All configurations" section at the bottom of the file will be used.

 

Issue/Introduction

This article discusses whether LogLogic LMI will index data that was previously outside the retention period when the index retention value is increased or when the indexer is re-enabled after being disabled for a very long time.