The best way to monitor the health of your IT system is perhaps to dive deeper into the system logs. Organizations generate tons of data everyday just in the form of logs, carrying information on alerts and notifications generated at different points of time. In fact, a large deployment of mail server, such as Microsoft Exchange generates 1 GB worth of logs every day! It is imperative that organizations store and analyze log data to gain business insights, along with adherence to local laws and regulations.
The Log Life Cycle
The cycle highlights how log data is assimilated, stored, transformed, consumed and then deleted. Also, this process helps understand how log events from different log streams can be correlated to gain meaningful insights.
Application and System Log Generated
Organizations collect log data from the various log generation hosts using shipping tools like Rsyslog, syslog . Rsyslog is an open source tool and can be used for assimilation of data. The tool can be configured with rules to capture data on the basis of specific occurrences. This helps in recording specific alert messages generated by applications. Also, the tool captures the System log data, web server log data and security log information.
Storage of Log Data in Private Cloud
The log data is then stored in the central repository system, in this case the Private cloud environment. Since the initial levels of data are low, they can be stored in such environments.
Transfer of Log Data into Public Cloud
As the data size increases, it is moved on to the public cloud, such as Azure or AWS. This helps in managing the large amount of data. Upon transfer, the data needs to be pre-processed so that it can be later used for indexing and searching. Since the data format differs, it is important to complete this parsing step using a pre-processing and forwarding tool such as Logstash.
An open source, server-side data processing tool, Logstash allows for different rule configurations, helping data in different formats to be reformatted accordingly, thus enabling further indexing and analysis.
Index and Analysis of Log Data
Now that the data has been parsed and stored in the public cloud, organizations index the data using tools such as Elasticsearch. A powerful search and analysis tool, Elasticsearch accepts JSON data as input and indexes it into binary format, that’s optimized for faster distributed services. The tool scales horizontally to handle massive number of events per second, automatically managing how indices are distributed across clusters for smooth operations. The Elasticsearch indexed data is stored in cloud and is retrieved whenever necessary for analysis to local.
Kibana is the visualizer tool that’s used for a graphic representation of data. This enables to view the events from different perspectives and uncover uncommon relationships among different sets of data.
Archive to Object Storage
As the log data increases in size, it becomes cumbersome and difficult to retain it in cloud. Hence, after it is consumed, the log data is archived and pushed into Object Storage.
Retrieval for Analysis
In case there is a need for further analysis, organizations can retrieve the log data from the object storage and apply analytical tools to gain further insights from the data.
After a certain period, say 3-6 months, and when the log data is no longer in use, it can be deleted from the Object Storage archive.
There are four governance aspects to log management in a hybrid cloud setup:
- Policy Enforcement – Admin can configure policies to enable pushing of log data from private to public clouds. For example, a policy can be configured, as per which system and application log data can be stored in private cloud as long as it meets predefined size and duration requirements, say 10GB and one month. Once those requirements are exceeded, the data is pushed into public cloud as per policy.
- Specific log storage – Admin can set rules that will permit only specific logs, such as parsed and filtered logs meant for further analysis, to be stored. This rule will ensure that storage space is economically used and only vital information is retained.
- Auto remediation – Error alerts are automatically noted and remediation measures implemented. For example, if the log contains an error alert stating Hard Disk is full, then Admin can configure an auto remediation measure of resizing Hard Disk. This feature is also beneficial in case of security issues. For instance, if any deviations are found in the system log pattern, such as unusual spike in log size then this is immediately notified to the admin so that appropriate action can be taken.
- Private/Public Storage Strategy – The organization’s cloud storage strategy for log data can be easily implemented. Admins can set rules on how long the log data needed to be stored on private cloud, after which the data could be pushed to public clouds.
Benefits of Log Operations Governance
There are multifaceted benefits of governing log operations:
- Organizations are required by certain local regulations to maintain logs. In such scenario, such an operations management process helps them to adhere to the compliance measures.
- Developer time which would have otherwise been spent on manually capturing and transforming data has been eliminated. The human intervention is required only for initial rule configuration and then analysis-based decision making.
- Top officials gain visibility on the status of IT heath and can take strategic steps towards fixing vital issues such as system downtime in real time.
- Easy identification of anomalies and malicious activity, against which measures can be deployed.