Log Viewer Proposal

Status

• Created: 2006-02-10 by JohnMoser

• Priority: LowPriority

• People:
• Contributors:
• Interested:

• Status: DraftSpecification

• Branch: Packages:
• Depends:
• Dependents:

Introduction

The aims of this specification include creating a modular log file aggregating framework to allow all types of logs to be aggregated in all types of ways.

Rationale

The current gnome-system-log application looks very nice; but there are directions it's just not feasible to go without major mutually exclusive or cumbersom changes. These include types of data to log, methods for logging, and methods of display.

Scope and Use Cases

The use of a system log viewer or real-time system activity tracker should be to easily review logs and track system activity in real-time. This may not simply apply to syslog and kernlog; but, for example, may be desirable for monitoring the snort IDS or all network activity through libpcap in the vein of Ethereal.

Furthery, it may be desired to take the display of this data in new and interesting directions. Typically, output is in the form of a list of logging data which administrators and auditors have to sift through to find anomolies; however, we may want to add more ways to analyze the data, either on screen or in other ways. Interesting examples include:

• Music. The concept behind the iSIC project is to utilize logging data to build music, which will contain detectable anomolies when the input data is anomolous.

• Visual. Other output may involve using the logging data to generate visualization data, much like a visualization in xmms or WinAmp. The advantage here is that the logs can be 'replayed' at a high speed and the visual patterns can be examined for anything that 'looks' strange.

• Strategic. Perhaps a most interesting visual-auditory interpretation would be to design a complex representation in which the log viewer showed what appeared to be a game of The Battle for Wesnoth or FreeCraft with AI behavior controlled by the logs. The AI would work on a subroutine crossed with log data and lagging pick-up; basically, the AI would "instruct" the game through a filter, and would "see" the results, and thus (having complete understanding of the filter) adjust its "instructions" to compensate for the filter after several odd moves. Sudden strange behavior, such as opposing teams suddenly fighting along side eachother to slaughter allies or attack groups suddenly scattering, would be indicative of strange log behavior.

• Homeostasis. Anil Buntwal Somayaji describes a security feature of an operating system, called Process Homeostasis. Using the same concept of homeostasis, log output filters can tag arbitrary flags and weights to "strange looking activity" in the context of normal system behavior. These weights may become integral to the output of generic log components, as well as the more far-fetched musical and visual logging concepts. Note that the Strategic visualizer described above would behave "normally" as long as the system remained in homeostasis anyway.

These examples are all out of scope for core GNOME developers; but they are very interesting. Additionally, GNOME developers cannot predict every possible system event that can be logged; consider Oracle 9i database logs or other proprietary information logging. Because of these factors, our scope would involve a modular logging system with the following components:

• Core. The core logging component. This component defines all plug-in and data structure interfaces.

• Data. Logging data plug-in end. This is the interface between the logs and the Core; plug-ins here grep logs and monitor subsystems via any means necessary to deliver information about anything. The resultant information is delivered with two data structures that explain its "generic" and "specific" representation.

• Analysis. The analysis-end logging filters tag interesting-looking log events and weight them. Display plug-ins can decide which analysis plug-ins to pay attention to.

• Display. The display end sends the data to the user. This is where the output happens, whether through audio-visual effects or simple log aggregation. Display plug-ins request logging data as needed; this can happen rapidly for audio-visual outputs.

INTERESTING NOTES THAT NEED TO BE MOVED ELSEWHERE ON THIS PAGE

It is important to note here that the Analysis plug-ins deal with data from Data plug-ins; but we do not specify this data for the program. Analysis occurs based on data patters. For example:

• A homeostasis plug-in may recognize that the first field in a data is spread over a wide and unpredictable range, and thus exhibits no pattern; it may then decided that repeated values in mass would mean something weird is going on. This would for example lead it to believe that sudden mass connections from the same IP address are anomolous, without understanding the concept of IP addresses.

• By the same token, breaking pattern would become anomolous as well. If a certain field is typically within a small range of numbers, say 80, 21, 23, and 125; then suddenly having that field exhibit different values could be considered anomolous. This would happen on a server servicing HTTP, FTP, SSH, and SMTP; sudden NNTP or mass ICMP-ECHO requests would show up as odd.
• Snort IDS logs would of course typically not show anything. Suddenly finding a logged attack would show an anomoly in the data stream.

Also note that in the case of audio-visual representation, small anomolies which may be missed by a homeostasis plug-in or other analysis plug-in may be detectable by the user.

We probably want to use DBUS for plug-ins; recent trends have indicated that this may be better than integrating dynamically linked plug-ins. This was brought to my attention in an unrelated conversation about security and mandatory access policy with Arjan van de Ven (speaking on his own behalf, not on the behalf of his current or former employers or associates).

Implementation Plan

Implementation Process

Pre-planning on this page.

Data Preservation and Migration

none

Packages Affected

User Interface Requirements

• Do we want to hard-rely on GTK+, or consider that this stuff may be globally useful?
  • It may be possible to daemonize the log collection back-ends later; we may avoid GUI code here without much trouble, and if it becomes a good log sourcer we may just make it a daemon and separate out the front-end.

Outstanding Issues