How to install ElastAlert with Elasticsearch on Ubuntu

ElastAlert is a framework that lets you set up easy & flexible alerts with Elasticsearch.

If you have to manage huge amounts of data then you will find yourself one day wishing for a tool that would simply point out the anomalies or inconsistencies in the data and alert you in real time.

What is ElastAlert?

ElastAlert is designed to exactly do that. It is a simple framework that alerts when it detects anomalies, spikes, or other patterns of rules from data added in the Elasticsearch.

For example, you could be setting up a ‘frequency’ alert, which will notify you when there is X number of events in Y time.

Or you may want to be immediately warned when there is a ‘spike’ event, that is when the rate at which an event occurs suddenly increases or decreases.

Other rules types that are included are:

  • ‘flatline’ – when there are less than X events in Y time
  • ‘blacklist/whitelist’ – when a certain field matches ‘blacklist’ or ‘whitelist’
  • ‘any’ – when an event that matches a given filter happens
  • ‘change’ – when a field has two different values within a specified period of time

Supported Alert Types

Currently, ElastAlert has built-in support for the following alert types.

  • Command
  • Email
  • JIRA
  • OpsGenie
  • SNS
  • HipChat
  • Slack
  • Telegram
  • GoogleChat
  • Debug
  • Stomp
  • theHive

Install ElastAlert with Elasticsearch on Ubuntu

In this article, we show you how to install ElastAlert on ubuntu 18.04.

Requirements

  • Elasticsearch
  • ISO8601 or Unix timestamped data
  • Python 2.7
  • pip, see requirements.txt – (https://github.com/Yelp/elastalert/blob/master/requirements.txt)
  • Packages for ubuntu –  python-pip python-dev libffi-dev libssl-dev

Installing Prerequisites

Install Python 2.7:

sudo apt-get install python-minimal

Check the Python version:

sudo python --version

Then you will get output for python 2.7.

Python Version
Python Version

Install needed packages:

sudo apt-get install python-pip python-dev libffi-dev libssl-dev

There are few different ways to install ElastAlert and here we are going to do the installation by cloning git repository.

So we need to install “git” before proceeding. Usually, Ubuntu 18.04 has git already installed.

Check for the installed or available version of git:

sudo apt-cache policy git

This will give the details of the installed and candidate git versions.

Git
Git version

If you can’t see installed git version, run following command.

sudo apt-get install git

We are going to clone ElastAlert repository to “/opt” folder, therefore change directory.

sudo cd /opt

Now clone a git repository.

sudo git clone https://github.com/Yelp/elastalert.git

Now install modules.

sudo pip install "setuptools>=11.3"
sudo python setup.py install

You may get an error like this.

Pip Error
Pip Error

Then run below command to install “PyOpenSSL”

sudo pip install PyOpenSSL

Here we are going to integrate with Elastic search 6.x. So Elasticsearch 5.0+ will be installed here.

sudo pip install "elasticsearch>=5.0.0"

Configure ElastAlert

We cloned ElastAlert repo to “/opt”directory, so change directory before continuing.

sudo cd /opt/elastalert/

Now we get a copy of config.yaml.example file as a config.yaml

sudo cp config.yaml.example config.yaml

Modify config.yaml file.

vim config.yaml

Uncomment the following lines and modify.

ElasticSearch Hostname or IP

es_host: elk-server

ElasticServer port

es_port: 9200

Uncomment basic-authentication:

es_username: 
es_password:

Config Yml File
Config Yml File

Save and close the file.

Create ElastAlert index.

sudo elastalert-create-index

Creating a Rule

Now edit the file titled “example_frequency.yaml” inside the “/opt/elastalert/example_rules/” folder

sudo vim example_rules/example_frequency.yaml

Uncomment and modify index as follows:

index: filebeat-*

Now define a filter for an alert. Here we filter keywords with the string “exception”.

filter:
- query_string:
    query: "message:*exception*"

Configure Alter with Slack. Here you need to create a Slack channel and incoming webhook. Then add configuration details as follows.

alert:
 - "slack"
slack:
slack_webhook_url: "https://hooks.slack.com/services/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"
slack_username_override: "Fosslinux-Elastic-Bot"
slack_channel_override: "#fosslinuxalert"
slack_emoji_override: ":robot_face:"
slack_msg_color: "danger"

Rules File

You can follow below steps to create Slack channel.

Configuring Slack channel for ElastAlert

If you don’t have a slack account, you can get one by simply signing up. Go to “slack.com” and enter your email address and click “GET STARTED”.

Sign Up Slack
Sign Up Slack

Then click ‘create new workspace’ and verify your email address. Now you can log in and view the dashboard.

Go to Browse apps -> Custom Integrations -> Incoming Webhooks -> New Configuration

Slack
Slack New Configuration

Then click on  ‘Create new channel’ to create a channel for send Alerts.

Create Channel
Create Channel

Then click the ‘Create Channel’ button and you will be taken to the Webhook integration page.

Incoming Web hooks
Incoming Webhooks

Click ‘Add Incoming WebHooks Integration’ button. This will create integration settings.

Slack Settings
Slack Settings

Test Rule

Change Directory to ElastAlert.

sudo cd /opt/elastalert/

Run below command to test configured rule.

sudo elastalert-test-rule example_rules/example_frequency.yaml

Test Rule
Test Rule

Run ElastAlert

We will start ElastAlert as a background service. This command should be run inside “/opt/elastalert/” folder.

sudo python -m elastalert.elastalert --verbose --rule example_frequency.yaml &

Started ElastAlert
Started ElastAlert

Now ElastAlert will begin to check queries on Elasticsearch (On ELK server). If there is a match it will shoot alert to Slack.

Alert triggered.

Alert Sent
Alert Sent

Alert will go to Slack Channel.

Slack Alert
Slack Alert

That’s it, we successfully installed and configured ElastAlert with the elastic search, and also set up alerts to Slack. We hope this exhaustive tutorial will help you to install ElastAlert and set up some rules to trigger alerts easily. Questions and feedback are welcome in the comments section.

Darshana
Hey! I'm Darshana, a Linux / DevOps Engineer and also a contributor to FOSS Linux. I enjoy working on various kind of Linux distributions and cloud technologies. During my free time, I love to swim and hike across nature trails. Linux is my love and I'm here to share all my learnings with all of you! Hope you enjoyed reading my article.

1 COMMENT

Leave a Reply to rajesh Cancel reply

Please enter your comment!
Please enter your name here

STAY CONNECTED

23,414FansLike
377FollowersFollow
16SubscribersSubscribe

LATEST ARTICLES

Mastering the SQLite Database in Python

SQLite is a relational database management system based on the SQL language; it is a serverless, Zero-configuration database engine. It is one of the...

Basics of Working with the SQLite Database in Python

A database is one of the most useful and popular files for storing data; they can be used to store any kind of data, including text, numbers, images, binary data, files, etc. SQLite is a relational database management system based on the SQL language. It is a C library, and it provides an API to work with other programming languages, including Python. It does not require a separate server process to be run as needed in large database engines like MySQL and Postgresql.

5 Ways to Check the Linux Version

When most people talk of Linux, they are always referring to a Linux distribution. However, this is not the case. Linux itself is a kernel which acts as a bridge between user applications and the hardware. When we talk of a Linux distribution, we refer to an operating system developed from the Linux kernel. A distribution comes with a package manager, pre-installed applications, a Desktop Environment, and several more features.

Getting Started with Linux Operating System

The Linux operating system brings forth a vibrant mix of features and security, making it the best alternative to macOS or Windows operating systems. In this post, we will give you a master guide on Getting started with Linux systems - taking you from a complete beginner to a level where you can begin testing the various Linux distributions available with much ease.

How to Create a Comprehensive Mail Server on Ubuntu

Postal is a free and open-source mail server used to send and receive emails. It comes loaded with tons of excellent features and functionalities, making it extremely popular among large organizations as well as in enterprise settings.

The 10 Best Linux Performance Monitoring Tools

Do you want to monitor the performance of your Linux system? Are you looking for some powerful performance monitoring tools to help you out? If you agree, it's your day as we have put together a detailed list of the ten best Linux performance monitoring tools.

MUST READ

Buyers who wish to go for a machine that is based on Linux often show interest in Chromebooks due to the form factor and extended battery life capabilities. Although ChromeOS power these machines, users can still miss out on a more genuine Linux experience. For those who happen to agree, the new Lemur Pro by System76 might get some heads turning.
Linux is growing faster than ever. As per the latest report, there is a drop in the Windows 10 market share for the first time, and Linux's market share has improved to 2.87% this month. Most of the features in the list were rolled out in the Pop OS 20.04. Let's a detailed look into the new features, how to upgrade, and a ride through video.

Beaker Browser: A P2P web browser you must try

I think we can all agree on the fact that the web browser is an integral part of our Linux systems, or any computer system, for that matter. We have had several fulfilling options for browsing the web, like Firefox, Brave, Vivaldi, Tor, etc. and they have worked pretty well, but also pretty much in the same way, except for Tor, which is much better for anonymity.

6 ways to find out your Linux file system type

Any Operating system in the market whether its Windows, Linux, Unix, macOS, and any other, must be able to access and manage files and data on storage devices.

VIDEO: MX Linux 19 Features and Desktop Tour

MX Linux is a Linux distribution based on Debian stable and using core antiX components. The distributions MX Tools is very popular among users and combined with other several ready-to-use tools, it is great for users who prefer to tweak their distro to their liking. In this video, we will take you through the features tour showing casing MX Linux 19.

6 cool and fun Linux commands you don’t want to miss

The real purpose of this article is to help Linux newbies get comfortable and confident with the Linux command-line.  While knowledge and comfort of the Linux GUI allow great power, it is the mastery of the command line, or CLI, affords the Linux user unlimited power and certifies them as a Linux power user well on their way to becoming an expert.