The Advanced Features of LogLogic Log Management intelligence (LMI) introduced in 6.1.0, and with subsequent enhancements in later versions, introduces many new terms, especially for queries. When just considering queries themselves there are many different types of queries, which can make the learning curve a bit steep. The introduction of additional functionality in 6.2.0 with correlation, tail/real-time and aggregation queries has further complicated this. This page is intended to provide information to illustrate the various different advanced query types.
Infrastructure query - An infrastructure query is used to obtain metadata about the LMI application such as a list of data models, configuration bloks, log source types, etc. Infrastructure queries do not use a time frame because the data is not time-related.
Example: use LogLogic_Config_BloksĀ
Non-infrastructure query - A generic name for any cached, forward-only, or tail query that is used to retrieve log source event data other than infrastructure-related data.
Example: use system | sys_eventTime IN -50m
Cached query - a query whose results are cached within the Query or Correlation node's H2 database. Query results for the query node can be deleted by closing the query tab on the Advanced Search page, deleting it from the Search Queries page or calling deleteQuery() using the REST API. The Search Queries page is accessible at Management->Advanced Features->Queries, Search tab (this page is available starting with LMI 6.2.0).
Example: use LogLogic_Config_Bloks
Forward-only query - This type of query is not cached, so the query results are simply being forwarded from Data node to Query node then to the webapp, otherwise the query functions like a cached query. This is used by the REST API, advanced dashboards and internally by the Correlation node to query historical data from Query node. Caching allows sub-select queries to be performed efficiently.
Example: Any dashboard widget query is executed as forward-only even though the query would otherwise be a cached query when executed interactively on the advanced search page.
Aggregation query - A non-infrastructure query that uses specific functions that perform aggregation on the requested data. Aggregation functions are functions like sum(), avg(), etc. If the query matches an aggregation rule then the data returned is already aggregated data via the Aggregation node (available starting with LMI 6.2.0) which eliminates the aggregation function at query time. The pre-aggregated data is stored in MySQL.
Example: use LogLogic_Monitor_Cpu | GROUP BY ll_nodeId, weeks(sys_eventTime), days(sys_eventTime), hours(sys_eventTime) COLUMNS max(ll_systemCPU), max(ll_processCPU), avg(ll_systemCPU), avg(ll_processCPU)
Correlation query - A correlation query uses event correlation language (ECL) for defining a correlation blok that can be used to trigger advanced alerts. It is not executed by the Query node but rather only the Correlation node. It can be used with the REST API but this is currently undocumented. The correlation blok can be re-used to create triggered alerts. Results are cached in Correlation node's own H2 database. Cached query results can be deleted by closing the tab on the Advanced Search page. This query is available starting with LMI 6.2.0.
Example: USE LogLogic_Appliance WITHIN 30m EVENT GROUP [My EVENTS] WHERE [ll_deviceTypeID] ="17" HAVING AT LEAST 2 DISTINCT [ll_sourceIP]
Tail query - A tail query is one that uses the EQL keyword "tail" to retrieve only newly collected data in near real-time for display on the search tab in the Advanced Search web page. Results are cached in Query node's H2 database before the UI can begin fetching them from that cache. The default limit for H2 database results is 10,000 rows. When the limit is reached the old events can be removed from the H2 cache to make room for new events.
Example: use system | tail
Real-time query - A query that uses the default "real-time" filter time blok. Functionally this is equivalent to a tail query.
Example: use system and contingent upon specifying a time block value of 'real-time' which translates into the following query in the query node log: use system | TAIL at UTC time 2018-12-12 13:50:57
Scheduled query - This is a cached query that is scheduled to execute at specific frequency and whose results are sent to a pre-defined recipient list. These are listed on the Management->Advanced Features->Queries, Scheduled tab (this page is available starting with LMI 6.2.0).
Example: Any cached query that is scheduled
Note: H2 is a small, very fast Java-based SQL database.