Skip to main content

Posts

Showing posts with the label Logstash

Data Analytics: Watching and Alerting on real-time changing data in Elasticsearch using Kibana and SentiNL

In the previous post , we have setup ELK stack and ran data analytics on application events and logs. In this post, we will discuss how you can watch real-time application events that are being persisted in the Elasticsearch index and raise alerts if condition for watcher is breached using SentiNL (Kibana plugin). Few examples of alerting for application events ( see previous posts ) are: Same user logged in from different IP addresses. Different users logged in from same IP address. PermissionFailures in last 15 minutes. Particular kind of exception in last 15 minutes/ hour/ day. Watching and alerting on Elasticsearch index in Kibana There are many plugins available for watching and alerting on Elasticsearch index in Kibana e.g. X-Pack , SentiNL . X-Pack is a paid extension provided by elastic.co which provides security, alerting, monitoring, reporting and graph capabilities. SentiNL is free extension provided by siren.io which provides alerting and reporting function

Running data analytics on application events and logs using Elasticsearch, Logstash and Kibana

In this post, we will learn how to use Elasticsearch, Logstash and Kibana for running analytics on application events and logs. Firstly, I will install all these applications on my local machine. Installations You can read my previous posts on how to install Elasticsearch , Logstash , Kibana and Filebeat on your local machine. Basic configuration I hope by now you are have installed Elasticsearch, Logstash, Kibana and Filebeat on your system. Now, Let's do few basic configurations required to be able to run analytics on application events and logs. Elasticsearch Open elasticsearch.yml file in [ELASTICSEARCH_INSTLLATION_DIR]/config folder and add properties to it. cluster.name: gauravbytes-event-analyzer node.name: node-1 Cluster name is used by Elasticsearch node to form a cluster. Node name within cluster need to be unique. We are running only single instance of Elasticsearch on our local machine. But, in production grade setup there will be master nodes, data nodes a

Installing Logstash

Logstash Logstash is data processing pipeline which ingests the data simultaneously from multiple data sources, transform it and send it to different `stash` i.e. Elasticsearch, Redis, database, rest endpoint etc. For example; Ingesting logs files; cleaning and transforming it to machine and human readable formats. There are three components in Logstash i.e. Inputs, Filters and Outputs Inputs It ingests data of any kind, shape and size. For examples: Logs, AWS metrics, Instance health metrics etc. Filters Logstash filters parse each event, build a structure, enrich the data in event and also transform it to desired form. For example: Enriching geo-location from IP using GEO-IP filter, Anonymize PII information from events, transforming unstructured data to structural data using GROK filters etc. Outputs This is the sink layer. There are many output plugins i.e. Elasticsearch, Email, Slack, Datadog, Database persistence etc. Installing Logstash As of writing Logstash(6.2.3) r