Skip to main content

Data Analytics: Watching and Alerting on real-time changing data in Elasticsearch using Kibana and SentiNL

In the previous post, we have setup ELK stack and ran data analytics on application events and logs. In this post, we will discuss how you can watch real-time application events that are being persisted in the Elasticsearch index and raise alerts if condition for watcher is breached using SentiNL (Kibana plugin).

Few examples of alerting for application events (see previous posts) are:

  • Same user logged in from different IP addresses.
  • Different users logged in from same IP address.
  • PermissionFailures in last 15 minutes.
  • Particular kind of exception in last 15 minutes/ hour/ day.

Watching and alerting on Elasticsearch index in Kibana

There are many plugins available for watching and alerting on Elasticsearch index in Kibana e.g. X-Pack, SentiNL.

X-Pack is a paid extension provided by which provides security, alerting, monitoring, reporting and graph capabilities.

SentiNL is free extension provided by which provides alerting and reporting functionality to monitor, notify and report changes in elasticsearch index using standard queries, programmable validators and configurable actions.

We will be using SentiNL for watching and alerting on Elasticsearch index.

Installing SentiNL


For debian, we need libfontconfig and libfreetype6 libraries, if not installed already.

sudo apt-get install libfontconfig libfreetype6

For centos, we need fontconfig and freetype libraries, if not installed already.

sudo yum install fontconfig freetype

// Installing SentiNL plugin
/opt/kibana/bin/kibana-plugin --install sentinl -u

Configuring SentiNL

SentiNL have wide range of actions that you can configure for watchers. You can send an email, integrate with Slack channel or pushapps, send payload to custom webhook. Open kibana.yml file and add below properties for SentiNL. For our example, we will only enable notification through email.

    host: 'localhost'
    port: 9200
      active: true
      host: ""
      user: "[EMAIL_ID]"
      password: "[PASSWORD]"
      port: 465
      domain: ""
      ssl: true
      tls: false
      authentication: ['PLAIN', 'LOGIN', 'CRAM-MD5', 'XOAUTH2']
      timeout: 20000  # mail server connection timeout
      # cert:
      #   key: '/full/sys/path/to/key/file'
      #   cert: '/full/sys/path/to/cert/file'
      #   ca: '/full/sys/path/to/ca/file'
      active: false
      username: 'username'
      hook: ''
      channel: '#channel'
      active: false
      host: 'localhost'
      port: 9200
      # use_https: false
      # path: ':/{{payload.watcher_id}}'
      # body: '{{payload.watcher_id}}{}}'
      # method: POST
      active: false
      executable_path: '/usr/bin/chromium' # path to Chrome v59+ or Chromium v59+
      timeout: 5000
      # authentication:
      #   enabled: true
      #   mode:
      #     searchguard: false
      #     xpack: false
      #     basic: false
      #     custom: true
      #   custom:
      #     username_input_selector: '#username'
      #     password_input_selector: '#password'
      #     login_btn_selector: '#login-btn'
      # file:
      #   pdf:
      #     format: 'A4'
      #     landscape: true
      #   screenshot:
      #     width: 1280
      #     height: 900
      active: false
      api_key: ''
That's it!!! Let's start Kibana to configure watcher and alerting in SentiNL.

Creating Watchers and alerting in Kibana

We will be configuring watcher for different users logged in from same IP address and will send e-Mail alerts.

  • Open Kibana dashboard on your local machine (Url for Kibana on my local machine is http://localhost:5601).
  • Click on SentiNL option in the left nav-pane. You will see a dashboard as below. Click on the New option to create a new watcher.
  • Click on the Watcher link highlighted as below.
  • Enter watcher name and schedule in the General tab.
  • Click on Input tab and enter below mentioned query json in the body. You can also give a name to the query and save.
      "search": {
        "request": {
          "index": [
          "body": {
            "query": {
              "bool": {
                "filter": [
                    "range": {
                      "@timestamp": {
                        "gte": "now-30m"
                    "query_string": {
                      "default_field": "appEvent.eventType",
                      "query": "LOGIN_SUCCESS OR LOGIN_FAILURE"
            "aggs": {
              "group_by_requestIP": {
                "terms": {
                  "field": "appEvent.requestIP.keyword",
                  "size": 5
                "aggs": {
                  "group_by_identifier": {
                    "terms": {
                      "field": "appEvent.identifier.keyword",
                      "size": 5
                    "aggs": {
                      "get_latest": {
                        "terms": {
                          "field": "@timestamp",
                          "size": 1,
                          "order": {
                            "_key": "desc"
  • Click on Condition tab and enter below mentioned condition json in the body. You can also give a name to this condition and save.
      "script": {
        "script": "var requestIPbuckets = payload.aggregations.group_by_requestIP.buckets; payload.collector = []; requestIPbuckets.filter(function(requestIP) { return requestIP.key; }).forEach(function(requestIP) { var requestIPKey = requestIP.key; var users = requestIP.group_by_identifier.buckets; if (users.length > 1) { users.filter(function(user) { return user.key; }).forEach(function(user) { payload.collector.push({ 'ip': requestIPKey, 'identifier': user.key, 'count': user.doc_count  }); }); }}); payload.collector.length > 0;"
  • Click on Action tab and select email as an action for alerting. Give title, to, from, subject and add below mentioned content in the body of email.
    Found {{payload.collector.length}} Events
    ip : {{ip}}, identifier: {{identifier}}, count: {{count}}
  • Save the watcher.

This watcher will run periodically based on the schedule that you have set and if the condition for breach is met, will send an email alert. The configured email looks like below.

This is how you can watch real-time changing data in Elasticsearch index and raise alerts based on the configured conditions.


Popular posts from this blog

React Ecosystem: Building BlogPost application with React and React Hooks

Building a Blog Post application Let's create a blog post application. It will have below features: Option to search blog posts. Option to list blog posts. Option to show blog post. Create a new project with npx create-react-app react-blog-posts --template typescript . Create BlogPosts.tsx component under src/components folder and IBlogPost model under src/models . import React from 'react'; import IBlogPost from '../models/IBlogPost'; interface IBlogPostsProps { posts: Array<IBlogPost> } function BlogPosts(props: IBlogPostsProps) { return ( <div className="blog-container"> <ul className="blog-posts"> { => <li key={}>{post.title}</li>) } </ul> </div> ); } export default BlogPosts; interface IBlogPost { id: number title: string content: string author: string postedOn: string tags: string[]

Java 8 - Lambda expressions

In this post, we will cover following topics. What are Lambda expressions? Syntax for Lambda expression. How to define no parameter Lambda expression? How to define single/ multi parameter Lambda expression? How to return value from Lambda expression? Accessing local variables in Lambda expression. Target typing in Lambda expression. What are Lambda expressions? Lambda expressions are the first step of Java towards functional programming. Lambda expressions enable us to treat functionality as method arguments, express instances of single-method classes more compactly. Syntax for Lambda expression Lambda has three parts: comma separated list of formal parameters enclosed in parenthesis. arrow token -> . and, body of expression (which may or may not return value). (param) -> { System.out.println(param); } Lambda expression can only be used where the type they are matched are functional interfaces . How to define no parameter Lambda expression? If the la