The appliance will maintain the timestamp that is in the message for messages received via Syslog, OPSEC, and SNMP, but it will also add its own timestamp to the metadata associated with the event. The timestamp that is stored in the metadata is the collection time. Device type and the source IP are also stored in the metadata for each event. The time of collection is stored because some devices do not include a timestamp, which is one of the required record keys. When you search by time, the time that is being searched is the time when the LogLogic appliance receives the message for syslog, OPSEC and SNMP. Because those collection methods occur in essentially real-time, the time difference between the original event time stored in the event header and the collection time stored by LMI is typically less than a second or two (assuming all system times are synchronized) so collection time is effectively the same as the original event time except in rare occasions.

When Windows events are collected via an agent and forwarded to a syslog product like LMI the agent will add a syslog header to the Windows event. This occurs whether you are using TIBCO LogLogic Universal Collector, Snare or some other product for collecting the Windows events. This syslog header contains a timestamp which results in the Windows event containing 2 timestamps: one from Windows and one from the collection agent. When the event is received by LogLogic LMI the event will have the collection time associated with it in the event's metadata just like any other real-time event. This makes Windows event data the only source type that has 3 different timestamps associated with the event. And ideally all 3 timestamps should be within a second or two of each other if no delays in transmission and processing have occurred and if all systems are in time sync.

As seen in the screenshot below, the Windows event displayed is an example of a discrepancy that can occur if the collection agent is not working correctly. To create the discrepancy below the UC service had been disabled for weeks then started up. The sys_eventTime in the advanced search shows the current date/time (July 25 2019 11:54:35 AM) but the syslog header added by UC to the event shows June 25 2019 10:33:15 AM and this was simply taken from the original event time that Windows inserted into the event. The timestamp from the syslog header added by UC will always match the original event time but what can differ is the sys_eventTime (i.e. collection time) compared to the timestamps in the event.



The next screenshot uses advanced search again with a Windows event showing a much smaller time gap between the collection time and the original event time. In this case the gap is just over 1 second with the collection time being in the past when compared to the original event time. How is that possible? This is just an artifact of the 2 hosts' system time(LMI and the Windows source) not being completely in sync.



Here is another screenshot showing how the classic index search shows a Windows event's original event time compared to the collection time. The collection time for the classic search is displayed in the Time column. In this example the collection time is 12:31:19 compared to the original event time of 12:31:18.823 so these are very close together.




For messages that LogLogic appliances received that are stored as file-based data the appliance will attempt to parse the timestamp of every event in the file. LMI will create a .time file associated with the original event file if the parsing is successful. In this manner the appliance maintains an accurate record of when the message was created. However, file-based data is stored in the BFQ directory structure according to the date and time when it was collected. When a user searches by time for file data the time being searched is technically the time when the message was written to the file rather than the collection time like it is for non-file-based data. Indexing is also performed based on the original event timestamps in the file. There is one exception to this behavior for file-based data and that is if LMI fails to recognize the timestamp pattern in file. In that situation the time associated with the event is the collection time. LMI recognizes approximately 20 different timestamp structures/patterns so it is unlikely for LMI to encounter a timestamp it cannot process. One scenario in which this can occur though if for custom applications where the developers didn't use standardized timestamp format.

Currently there aren't any supported file types that LMI parses that do not contain a timestamp at the beginning of the message. However if there is no timestamp, LogLogic LMI will not add a timestamp and the collection time will be used in the metadata for search purposes.  

TIBCO recommends that all file transfer devices are synchronized to an NTP server to ensure that all searches are correlated correctly and that all timestamps are synchronized.

This article discusses how LMI processes and uses timestamps for purposes of event collection and searching.