What is the compression ratio for storing raw logs using a TIBCO LogLogic LMI appliance?
Article ID: KB0079054
Updated On:
| Products | Versions |
|---|---|
| TIBCO LogLogic Log Management Intelligence | all versions |
Description
LogLogic LMI users typically can expect anywhere from 10:1 to 12:1 compression ratio of raw data on average of BFQ files. The remainder of this article explains this in more detail and how this affects file storage containing your event data.
LogLogic LMI appliances use a proprietary compression called llzip. This provides better compression than gzip and also provides for the use of sparse files. However, the minimum size for an llzip-based file is 256KB. This is not of any consequence unless your LMI appliance is collecting very few logs in which case the compressed size may be larger than the uncompressed logs.
By default, a single BFQ file containing syslog data will contain data for 1 minute of time. This means all syslog data flowing into the system within a given minute will be stored in the same file, with one exception to be discussed later. This dictates 60 files of syslog data per hour will exist in each hourly directory within the BFQ. The BFQ is located at /loglogic/data/vol1 (for local data) or /loglogic/data/archive/mount_x/vol1 (for archived data, where x is the mount number). The exception referenced above occurs when using custom data retention rules in LMI. In that case you may have files spanning multiple minutes, as well as multiple files per minute, as a result of LMI storing device data in specific files based on the retention rules. Regardless of whether granular retention rules are used the BFQ files will be compressed individually.
The compression rate specified in the GUI is based on the percentage of the file size after compression compared to its original size. You can see the compression percentage in the LMI web UI by clicking on Administration->Data Files. This value represents the percentage of the original logs' size that the compressed file consumes on disk. If all the logs for a given file represent 1 MB (i.e. 1024 KB) of data when uncompressed and the compressed file listed in the Data Files page now shows a size of 102 KB and a compression of 10% then that means the compressed size is 10% of the original uncompressed size. This of course matches the size values given because 102 KB is 10% of 1 MB. This should not be interpreted as only reflecting a 10% compression.
At this point we can obtain the compression ratio using the following formula:
<UNcompressed file size> / <compressed file size> = compression ratio
Example: 1 MB / 102 KB = 10, or as a ratio it would be 10:1.
To determine the compression rate we use the following:
(<UNcompressed file size> - <compressed file size>) / <UNcompressed file size> * 100 = compression rate
Example: (1 MB - 102 KB) / 1 MB * 100 = 90%
LogLogic LMI appliances use a proprietary compression called llzip. This provides better compression than gzip and also provides for the use of sparse files. However, the minimum size for an llzip-based file is 256KB. This is not of any consequence unless your LMI appliance is collecting very few logs in which case the compressed size may be larger than the uncompressed logs.
By default, a single BFQ file containing syslog data will contain data for 1 minute of time. This means all syslog data flowing into the system within a given minute will be stored in the same file, with one exception to be discussed later. This dictates 60 files of syslog data per hour will exist in each hourly directory within the BFQ. The BFQ is located at /loglogic/data/vol1 (for local data) or /loglogic/data/archive/mount_x/vol1 (for archived data, where x is the mount number). The exception referenced above occurs when using custom data retention rules in LMI. In that case you may have files spanning multiple minutes, as well as multiple files per minute, as a result of LMI storing device data in specific files based on the retention rules. Regardless of whether granular retention rules are used the BFQ files will be compressed individually.
The compression rate specified in the GUI is based on the percentage of the file size after compression compared to its original size. You can see the compression percentage in the LMI web UI by clicking on Administration->Data Files. This value represents the percentage of the original logs' size that the compressed file consumes on disk. If all the logs for a given file represent 1 MB (i.e. 1024 KB) of data when uncompressed and the compressed file listed in the Data Files page now shows a size of 102 KB and a compression of 10% then that means the compressed size is 10% of the original uncompressed size. This of course matches the size values given because 102 KB is 10% of 1 MB. This should not be interpreted as only reflecting a 10% compression.
At this point we can obtain the compression ratio using the following formula:
<UNcompressed file size> / <compressed file size> = compression ratio
Example: 1 MB / 102 KB = 10, or as a ratio it would be 10:1.
To determine the compression rate we use the following:
(<UNcompressed file size> - <compressed file size>) / <UNcompressed file size> * 100 = compression rate
Example: (1 MB - 102 KB) / 1 MB * 100 = 90%
Note the compression ratio and compression rate will differ from file to file because it's based on the data being compressed. Some data compresses better than other data. For example, longer log messages will compress better than shorter messages.
Note that the compression algorithm is applied to both file-based and syslog-based data.
Issue/Introduction
This article describes how to determine the compression ratio for raw log storage when using LogLogic LMI.
Feedback
Yes
No
Powered by
