Showing posts with label ELK stack. Show all posts

Filebeat

Filebeat is a light-weight log shipper. It is installed as a agent and listen to your predefined set of log files and locations and forward them to your choice of sink (Logstash, Elasticsearch, database etc.)

Installation

deb
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-amd64.deb
sudo dpkg -i filebeat-6.3.2-amd64.deb
rpm
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-x86_64.rpm
sudo rpm -vi filebeat-6.3.2-x86_64.rpm
mac
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-darwin-x86_64.tar.gz
tar xzvf filebeat-6.3.2-darwin-x86_64.tar.gz
docker
docker pull docker.elastic.co/beats/filebeat:6.3.2
Windows

Download the filebeat from official website and do the following configurations.

1) Extract the zip file to your choice of location. e.g. C:\Program Files.
2) Rename the filebeat--windows directory to Filebeat.
3) Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select Run As Administrator).
4) From the PowerShell prompt, run the following commands to install Filebeat as a Windows service:

// Command to execute from powershell
cd 'C:\Program Files\Filebeat'
.\install-service-filebeat.ps1

Kibana

Kibana is a visualization dashboard for Elasticsearch and you can choose many available charts like graphs, pie, bar, histogram etc. or real time textual data and can gain meaningful analytics.

Installation

Installating Kibana directly from tar files

For Linux installation

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-linux-x86_64.tar.gz
shasum -a 512 kibana-6.2.3-linux-x86_64.tar.gz 
tar -xzf kibana-6.2.3-linux-x86_64.tar.gz
cd kibana-6.2.3-linux-x86_64/

For Windows installation

//Dowload Kibana
https://artifacts.elastic.co/downloads/kibana/kibana-6.2.3-windows-x86_64.zip

//running kibana
/bin/kibana.bat

Installation from packages

Debian package installation

// Import elatic PGP key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

//install https transport module
sudo apt-get install apt-transport-https

//save repository definition
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

//installation command
sudo apt-get update && sudo apt-get install kibana

rpm package installation

//Download and install public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in a new .repo file in your /etc/yum.repos.d/ directory

[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
//Installation command
sudo yum install kibana 
sudo dnf install kibana 
sudo zypper install kibana 

Logstash

Logstash is data processing pipeline which ingests the data simultaneously from multiple data sources, transform it and send it to different `stash` i.e. Elasticsearch, Redis, database, rest endpoint etc. For example; Ingesting logs files; cleaning and transforming it to machine and human readable formats.

There are three components in Logstash i.e. Inputs, Filters and Outputs

Inputs

It ingests data of any kind, shape and size. For examples: Logs, AWS metrics, Instance health metrics etc.

Filters

Logstash filters parse each event, build a structure, enrich the data in event and also transform it to desired form. For example: Enriching geo-location from IP using GEO-IP filter, Anonymize PII information from events, transforming unstructured data to structural data using GROK filters etc.

Outputs

This is the sink layer. There are many output plugins i.e. Elasticsearch, Email, Slack, Datadog, Database persistence etc.

Installing Logstash

As of writing Logstash(6.2.3) requires Java 8 to run. To check the java version run the following command

java -version

The output on my system is

java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)

If Java 8 is not installed then please download it from Oracle website and follows instruction for installation. Also, set the JAVA_HOME variable.

Installing from binaries

You can directly download the binaries from here.

Installing from package repositories

Installation with APT

//ADD PUBLIC SIGNING KEY
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

//add https-transports
sudo apt-get install apt-transport-https

//save the repository definition
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

//installation command
sudo apt-get update && sudo apt-get install logstash

Installation with YUM

// Download and install the public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in a new .repo file in your /etc/yum.repos.d/ directory

[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
// Installation command
sudo yum install logstash

Docker installation

You can follow the link for docker installation.