TIBCO LogLogic LMI - Command to drop tables that exceeds retention policy

TIBCO LogLogic LMI - Command to drop tables that exceeds retention policy

book

Article ID: KB0073284

calendar_today

Updated On:

Products Versions
TIBCO LogLogic Enterprise Virtual Appliance 6.1.0 and higher

Description

As Administrators may know - the parsed data is stored in data-folder for each Real Time Database under '/loglogic/data':
 

ll_auth_logs
ll_cached_results
ll_files
ll_files_in
ll_firewall_logs
ll_ids_logs
ll_misc_logs
ll_postupgrade
ll_stats
ll_temp_results
ll_vpn_logs
ll_www_logs
 
You can find the retention policies for each Real Time data in the Web GUI at "Administration > System Settings>Database Table Retention" for LX/MX platforms.
At times it may occur that the system is unable to age off the tables to keep up the retention policy. In that case Administrators need to manually drop the old views/tables to allow reports to work properly.

Resolution

To begin with - we need to keep in mind that the base view/table are not to be purged. We can isolate the base views/tables with below command:
 
$ cd /loglogic/data/ll_auth_logs

$ ls -l *_0*
-rw-r----- 1 mysql users     0 Jul 15 16:21 mrgAuthentication1day_0.MYD
-rw-r----- 1 mysql users  1024 Jul 15 16:21 mrgAuthentication1day_0.MYI
-rw-r----- 1 mysql users 13546 Jul 15 16:21 mrgAuthentication1day_0.frm
-rw-r----- 1 mysql users     0 Jul 15 16:22 mrgAuthentication1hrConn_0.MYD
-rw-r----- 1 mysql users  1024 Jul 15 16:22 mrgAuthentication1hrConn_0.MYI
-rw-r----- 1 mysql users 13398 Jul 15 16:22 mrgAuthentication1hrConn_0.frm
-rw-r----- 1 mysql users     0 Jul 15 16:21 mrgAuthentication1hr_0.MYD
-rw-r----- 1 mysql users  1024 Jul 15 16:21 mrgAuthentication1hr_0.MYI
-rw-r----- 1 mysql users 13546 Jul 15 16:21 mrgAuthentication1hr_0.frm

You will find 3 datafiles for each view/table: .frm (table structure), .MYD (table data) and the .MYI (indexes). The views/tables that we need to isolate and eventually delete can be shown with below command:
 
$ cd /loglogic/data/ll_auth_logs

$ ls -l *_1*.MYD

Above command will list a bunch of datafiles containing a unix timestamp in the name:
...
-rw-r----- 1 mysql users   16420 Jul 26 18:02 mrgAuthentication1day_1627257600.MYD
-rw-r----- 1 mysql users   16724 Jul 27 18:02 mrgAuthentication1day_1627344000.MYD
-rw-r----- 1 mysql users    6012 Jul 28 06:02 mrgAuthentication1day_1627430400.MYD
-rw-r----- 1 mysql users    2544 Jul 16 00:02 mrgAuthentication1hrConn_1626307200.MYD
-rw-r----- 1 mysql users   10688 Jul 17 00:02 mrgAuthentication1hrConn_1626393600.MYD
-rw-r----- 1 mysql users    9792 Jul 18 00:03 mrgAuthentication1hrConn_1626480000.MYD
-rw-r----- 1 mysql users    9792 Jul 19 00:02 mrgAuthentication1hrConn_1626566400.MYD
...

That unix timestamp will tell us the time that the view/table was created. However - there may be hundreds of view/tables so we cannot check one by one. Also the time showed by the command ls -lrt may not coincide with the creation time file as the datafiles are often updated by the system. With that in mind, a command like:
 
$ find ./*_1*.MYD -type f -mtime 10

May not show the correct list of datafiles to be dropped. We need to grab the access time instead which is always the same as the creation time. For datafiles in MySQL the access time ('atime') keep aligned with the creation time so we can use that to list the datafiles that are older out of retention time. For instance - in the ll_auth_logs datafolder - given a retention policy of 10 days we can list the views/tables to be dropped with below command:

$ cd /loglogic/data/ll_auth_logs

$ find ./*_1*.MYD -type f -atime +10
./mrgAuthentication1day_1626307200.MYD
./mrgAuthentication1day_1626393600.MYD
./mrgAuthentication1day_1626480000.MYD
./mrgAuthentication1hrConn_1626307200.MYD
./mrgAuthentication1hrConn_1626393600.MYD
./mrgAuthentication1hrConn_1626480000.MYD
./mrgAuthentication1hr_1626307200.MYD
./mrgAuthentication1hr_1626393600.MYD
./mrgAuthentication1hr_1626480000.MYD
./mrgAuthentication_1626307200.MYD
./mrgAuthentication_1626393600.MYD
./mrgAuthentication_1626480000.MYD

You can confirm that the unix timestamp in the name is actually older that 10 days with below command:
 
$ date -ud@1626307200      # for first file - current date: 28/07/2021
Thu Jul 15 00:00:00 UTC 2021
$ date -ud@1626480000      # for last file - current date: 28/07/2021
Sat Jul 17 00:00:00 UTC 2021

At this stage we need to extract the actual view/table name from the file name:

$ cd /loglogic/data/ll_auth_logs

$ find ./*_1*.MYD -type f -atime +10 | awk -F "./" '{print $2}' | awk -F "." '{print $1}'
mrgAuthentication1day_1626307200
mrgAuthentication1day_1626393600
mrgAuthentication1day_1626480000
mrgAuthentication1hrConn_1626307
mrgAuthentication1hrConn_1626393
mrgAuthentication1hrConn_1626480
mrgAuthentication1hr_1626307200
mrgAuthentication1hr_1626393600
mrgAuthentication1hr_1626480000
mrgAuthentication_1626307200
mrgAuthentication_1626393600
mrgAuthentication_1626480000

Having the actual table name as listed above we are now able to cleanly drop the views/tables with below command:

$ cd /loglogic/data/ll_auth_logs

$ for TABLE in `find ./*_1*.MYD -type f -atime +10 | awk -F "./" '{print $2}' | awk -F "." '{print $1}'`; do mysql ll_auth_logs -e "drop table $TABLE"; done

That should be it - the system should automatically rebuild the merge file under 'logapprtrpt' datafolder so no engine_mysqld restart is required.

Issue/Introduction

This article suggests a command to manually drop tables from Real Time data-folders that are out of retention time.