How to install ElastAlert with Elasticsearch on Ubuntu

ElastAlert is a framework that lets you set up easy & flexible alerts with Elasticsearch.

If you have to manage huge amounts of data then you will find yourself one day wishing for a tool that would simply point out the anomalies or inconsistencies in the data and alert you in real time.

What is ElastAlert?

ElastAlert is designed to exactly do that. It is a simple framework that alerts when it detects anomalies, spikes, or other patterns of rules from data added in the Elasticsearch.

For example, you could be setting up a ‘frequency’ alert, which will notify you when there is X number of events in Y time.

Or you may want to be immediately warned when there is a ‘spike’ event, that is when the rate at which an event occurs suddenly increases or decreases.

Other rules types that are included are:

  • ‘flatline’ – when there are less than X events in Y time
  • ‘blacklist/whitelist’ – when a certain field matches ‘blacklist’ or ‘whitelist’
  • ‘any’ – when an event that matches a given filter happens
  • ‘change’ – when a field has two different values within a specified period of time

Supported Alert Types

Currently, ElastAlert has built-in support for the following alert types.

  • Command
  • Email
  • JIRA
  • OpsGenie
  • SNS
  • HipChat
  • Slack
  • Telegram
  • GoogleChat
  • Debug
  • Stomp
  • theHive

Install ElastAlert with Elasticsearch on Ubuntu

In this article, we show you how to install ElastAlert on ubuntu 18.04.

Requirements

  • Elasticsearch
  • ISO8601 or Unix timestamped data
  • Python 2.7
  • pip, see requirements.txt – (https://github.com/Yelp/elastalert/blob/master/requirements.txt)
  • Packages for ubuntu –  python-pip python-dev libffi-dev libssl-dev

Installing Prerequisites

Install Python 2.7:

sudo apt-get install python-minimal

Check the Python version:

sudo python --version

Then you will get output for python 2.7.

Python Version
Python Version

Install needed packages:

sudo apt-get install python-pip python-dev libffi-dev libssl-dev

There are few different ways to install ElastAlert and here we are going to do the installation by cloning git repository.

So we need to install “git” before proceeding. Usually, Ubuntu 18.04 has git already installed.

Check for the installed or available version of git:

sudo apt-cache policy git

This will give the details of the installed and candidate git versions.

Git
Git version

If you can’t see installed git version, run following command.

sudo apt-get install git

We are going to clone ElastAlert repository to “/opt” folder, therefore change directory.

sudo cd /opt

Now clone a git repository.

sudo git clone https://github.com/Yelp/elastalert.git

Now install modules.

sudo pip install "setuptools>=11.3"
sudo python setup.py install

You may get an error like this.

Pip Error
Pip Error

Then run below command to install “PyOpenSSL”

sudo pip install PyOpenSSL

Here we are going to integrate with Elastic search 6.x. So Elasticsearch 5.0+ will be installed here.

sudo pip install "elasticsearch>=5.0.0"

Configure ElastAlert

We cloned ElastAlert repo to “/opt”directory, so change directory before continuing.

sudo cd /opt/elastalert/

Now we get a copy of config.yaml.example file as a config.yaml

sudo cp config.yaml.example config.yaml

Modify config.yaml file.

vim config.yaml

Uncomment the following lines and modify.

ElasticSearch Hostname or IP

es_host: elk-server

ElasticServer port

es_port: 9200

Uncomment basic-authentication:

es_username: 
es_password:

Config Yml File
Config Yml File

Save and close the file.

Create ElastAlert index.

sudo elastalert-create-index

Creating a Rule

Now edit the file titled “example_frequency.yaml” inside the “/opt/elastalert/example_rules/” folder

sudo vim example_rules/example_frequency.yaml

Uncomment and modify index as follows:

index: filebeat-*

Now define a filter for an alert. Here we filter keywords with the string “exception”.

filter:
- query_string:
    query: "message:*exception*"

Configure Alter with Slack. Here you need to create a Slack channel and incoming webhook. Then add configuration details as follows.

alert:
 - "slack"
slack:
slack_webhook_url: "https://hooks.slack.com/services/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"
slack_username_override: "Fosslinux-Elastic-Bot"
slack_channel_override: "#fosslinuxalert"
slack_emoji_override: ":robot_face:"
slack_msg_color: "danger"

Rules File

You can follow below steps to create Slack channel.

Configuring Slack channel for ElastAlert

If you don’t have a slack account, you can get one by simply signing up. Go to “slack.com” and enter your email address and click “GET STARTED”.

Sign Up Slack
Sign Up Slack

Then click ‘create new workspace’ and verify your email address. Now you can log in and view the dashboard.

Go to Browse apps -> Custom Integrations -> Incoming Webhooks -> New Configuration

Slack
Slack New Configuration

Then click on  ‘Create new channel’ to create a channel for send Alerts.

Create Channel
Create Channel

Then click the ‘Create Channel’ button and you will be taken to the Webhook integration page.

Incoming Web hooks
Incoming Webhooks

Click ‘Add Incoming WebHooks Integration’ button. This will create integration settings.

Slack Settings
Slack Settings

Test Rule

Change Directory to ElastAlert.

sudo cd /opt/elastalert/

Run below command to test configured rule.

sudo elastalert-test-rule example_rules/example_frequency.yaml

Test Rule
Test Rule

Run ElastAlert

We will start ElastAlert as a background service. This command should be run inside “/opt/elastalert/” folder.

sudo python -m elastalert.elastalert --verbose --rule example_frequency.yaml &

Started ElastAlert
Started ElastAlert

Now ElastAlert will begin to check queries on Elasticsearch (On ELK server). If there is a match it will shoot alert to Slack.

Alert triggered.

Alert Sent
Alert Sent

Alert will go to Slack Channel.

Slack Alert
Slack Alert

That’s it, we successfully installed and configured ElastAlert with the elastic search, and also set up alerts to Slack. We hope this exhaustive tutorial will help you to install ElastAlert and set up some rules to trigger alerts easily. Questions and feedback are welcome in the comments section.

Darshana
Hey! I'm Darshana, a Linux / DevOps Engineer and also a contributor to FOSS Linux. I enjoy working on various kind of Linux distributions and cloud technologies. During my free time, I love to swim and hike across nature trails. Linux is my love and I'm here to share all my learnings with all of you! Hope you enjoyed reading my article.

1 COMMENT

LEAVE A REPLY

Please enter your comment!
Please enter your name here

STAY CONNECTED

23,555FansLike
365FollowersFollow
16SubscribersSubscribe

LATEST ARTICLES

The 10 Best Linux Server Distros For Home And Businesses

By the year 2020, it is estimated that there are close to 600 Linux distributions in the market. It includes both servers and Desktop versions; therefore, if you are looking for lightweight Linux distribution for your old PC or a reliable desktop version for employees in your organization, you may be overwhelmed with the number of choices for finding one for your use.

How to list the installed Packages on Ubuntu

It is only natural to start installing more and more software on your Ubuntu PC once you start using it. Similarly, it is also natural to lose track of the different packages that you have installed on your system. Eventually, you may be overwhelmed with the number of packages in your system and with the ones you hardly used.

How to configure the Ubuntu Firewall (UFW)

A properly configured firewall is a crucial part of establishing preliminary system security. Keeping this in mind, here we will go over how to configure the firewall on your Ubuntu PC.

How to install and use Zoom on Ubuntu

If you work from home or hold meetings with other people remotely, then you have probably heard of Zoom. It is one of the popular video conferencing software app available today.

How to Restart Networking on Ubuntu

Sometimes resetting your Ubuntu network is required to apply some network configurations like changing your IP from Automatic DHCP to static one. Restarting the network is not a big deal, but should be done carefully.

How to install Chrome and Chromium Browser on Pop!_OS

By default, Pop!_OS comes with Mozilla Firefox Browser installed to help you browse the internet. It is a reliable web browser covering almost all the features and functionalities you need. However, working with Firefox is noticeably different compared to Google Chrome, especially from the speed point of view. Google Chrome is a tad quicker than Firefox.

MUST READ

Linux is growing faster than ever. As per the latest report, there is a drop in the Windows 10 market share for the first time, and Linux's market share has improved to 2.87% this month. Most of the features in the list were rolled out in the Pop OS 20.04. Let's a detailed look into the new features, how to upgrade, and a ride through video.
Elementary OS 5.1 Hera has received a point release with a handful of new features and bug fixes, and we will be reviewing the significant changes in this article. For those new to elementary OS, this Ubuntu-based Linux distribution uses their inhouse built Pantheon desktop environment and AppCenter.

10+ Best Python IDEs for Linux

Python programming language is applied in so many areas of computer technology, i.e., Scripting, GUI development, Website development, Artificial Intelligence (AI), Machine Learning, Data Science, Computer Networking, and Network Automation, and Cyber Security.

Pop!_OS 20.04 Review: Professional Linux Distribution Ever Made

Linux is growing faster than ever. As per the latest report, there is a drop in the Windows 10 market share for the first time, and Linux's market share has improved to 2.87% this month. Most of the features in the list were rolled out in the Pop OS 20.04. Let's a detailed look into the new features, how to upgrade, and a ride through video.

How to dual-boot Ubuntu and Fedora on your PC

Every Linux distribution has its capabilities and preferences as per the user requirements. That brings forth the need to run different distros on one laptop. For example, a CyberSecurity enthusiast might prefer to use Ubuntu for development and Kali Linux for penetration testing.

Top 5 reasons to switch from Windows to Linux right now

Do you love Windows or Linux? Being on a Linux website, it's not hard to take a guess, but what are the strong reasons to switch to Linux? Linux is a 100% free OS. Is that reason alone enough? Let's find out! Never have there been more compelling reasons to make the switch. Here are just five great reasons for users to create that magical move from the wayward world of Windows to the lauded land of Linux.