Logging and Log Management

Organizations that adopt multiple systems, servers and applications may find it difficult to track security logs that they generate. And with the evolution of microservice architectures, logging has become increasingly important. Security logs can help developers analyze errors, identify attacks, and gather insights. Logging allows organizations to improve their servers and systems and are essential to troubleshoot application/ infrastructure performance. Actively reviewing the security log keeps cybercriminal activities at bay. A comprehensive log management system can be tailored to alert users regarding malware detection, unauthorized login attempts, DoS attacks, data export, and other such events. 

Choosing the Right Logging Tool

Centralized logging assists organizations to gather, analyze, and display their event logs at a single location. Different types of logging tools are available on the internet such as Loggly, Sumo Logic, Splunk, etc. While these are some of the popular options, the cheapest alternative is maintaining an ELK (Elasticsearch, Logstash, and Kibana) Stack. They all have more or less the same features to offer. 

What is ELK Stack?

ELK Stack is a combination of Elasticsearch, Logstash, and Kibana, and is the most popular open-source log analysis platform. Logstash aggregates the logs, transforms/ parses data -> Elasticsearch stores and indexes incoming logstash data -> Kibana analyses and visualizes the data from Elasticsearch. In addition to that, Beats ships log data to Elasticsearch and Logstash, using various types of shippers for different types of files – Filebeats, Metricbeat, etc.

Well-known companies like Netflix, Stack Overflow, LinkedIn, etc. opted for ELK Stack. This shouldn’t come as a surprise considering all of the critical capabilities and services that this stack provides:

1. A central logging system for all microservices, with real-time logging analytics and alerting system.
2. Simplified, scaled deployment, vertically and horizontally.
3. Data visualization that captures and displays the analytics.

Configuring ELK Stack

In the following demo, we’ll analyse NginX and Docker logs using Filebeats and visualize them in Kibana.

We first set up ELK Stack 7.8.1 on docker. You can find the file here. If you want to install the system directly please see this.

To set up the docker, run:

$ sudo docker-compose up -d

You are all set to proceed if the local host http://localhost:80 returns a positive response.

*username – admin; password – admin

Elastic – http://localhost:80/elastic

Kibana – http://localhost:80/kibana

Now that you’re all set up, let’s have a look at the logs in Kibana.

Go to Kibana ->Stack Management -> Index Pattern -> Add Index.

Add logstash-server-* and logstash-logs-*. 

Choose @timestamp field as time filer

Now go to the Discover panel to see your logs.

Once you are able to see the logs, you can create visualizations to represent critical business metrics.

To create a new visualization, find the option ‘Visualize’ on the side panel and follow the instructions mentioned here. You can add more fields in the Logstash pipeline config based on your requirements and visualize daily/ monthly/ yearly/ custom time range. Here is an example:

Other features of ELK Stack

• Define the structure of your logs and create visualizations accordingly.
• Subscribe to Slack/ email alerts to be notified about ERROR logs.
• Monitor your services such as MySQL, Kafka, Mongo, EC2 system, etc., using Metricbeat.
• Set alerts for a daily summary of your infrastructure, based on the log data. Eg. distinct new users login.
• Add ML pipelines in between to analyse the logs and take decisions accordingly. For instance, take a look at the load on service and predict the future load. Based on which you can scale your services in advance.

ELK Stack allows users to analyze and visualize data from any source, in any format. The stack is owned by the company Elastic that combines their three open source products Elasticsearch, Logstash, and Kibana. Which means that the stack’s centralized logging capabilities and its supplemental features are available to anyone, free of cost. This makes ELK Stack a popular choice among developers, for log analysis.