Skip to main content

Posts

Showing posts with the label distributed event logs

Running data analytics on application events and logs using Elasticsearch, Logstash and Kibana

In this post, we will learn how to use Elasticsearch, Logstash and Kibana for running analytics on application events and logs. Firstly, I will install all these applications on my local machine. Installations You can read my previous posts on how to install Elasticsearch , Logstash , Kibana and Filebeat on your local machine. Basic configuration I hope by now you are have installed Elasticsearch, Logstash, Kibana and Filebeat on your system. Now, Let's do few basic configurations required to be able to run analytics on application events and logs. Elasticsearch Open elasticsearch.yml file in [ELASTICSEARCH_INSTLLATION_DIR]/config folder and add properties to it. cluster.name: gauravbytes-event-analyzer node.name: node-1 Cluster name is used by Elasticsearch node to form a cluster. Node name within cluster need to be unique. We are running only single instance of Elasticsearch on our local machine. But, in production grade setup there will be master nodes, data nodes a

Elasticsearch setup and configuration

What is Elasticsearch? Elasticsearch is highly scalable, broadly distributed open-source full text search and analytics engine. You can in very near real-time search, store and index big volume of data. It internally use Apache Lucene for indexing and storing data. Below are few use cases for it. Product search for e-commerce website Collecting application logs and transaction data for analyzing it for trends and anomalies. Indexing instance metrics(health, stats) and doing analytics, creating alerts for instance health on regular interval. For analytics/ business-intelligence applications Elasticsearch basic concepts We will be using few terminologies while talking about Elasticsearch. Let's see basic building blocks of Elasticsearch. Near real-time Elasticsearch is near real-time. What it means is that the time (latency) between the indexing of document and its availability for searching. Cluster It is a collection of one or multiple nodes (servers) that together h