ELK Stack Powered Analytics
Just what is the ELK Stack?
The (ELK) stack consists of Elasticsearch, Logstash, and Kibana opensource applications.
Each of these software packages is open source, yet managed by the open-source vendor Elastic. Even though they are separate applications, they’ve been designed and tested to work together magnificently.
To help understand what the technology is all about, I’ll start with a run down of each application.
The Document Storage Engine is Elasticsearch. It comes with good default settings to get you up and running quickly, but it’s also possible to scale to hundreds of millions in records while maintaining sub-second search speed latency. Here’s a great tutorial on how to setup your first index and start querying.
This pipeline tool takes data in, processes and provides output. For example, I grab only attack data from my server logs for later investigation.
To Visualize Your Log Data, Kibana is an amazing way to do it. A beautifully designed web-based front-end to your data.
By rolling your own ELK stack, you’ll gain the power and control of having your own solution, and save significantly over remotely hosted SAAS log analyzers as well.