Powerful alerting with ElastAlert 2
Deploy in a few minutes one of the most complete alert system.
Deploy in a few minutes one of the most complete alert system.
Last updated July 28th, 2022
ElastAlert 2 is an alerting framework originally designed by Yelp. It is able to detect anomalies, spikes, or other patterns of interest. It is production-ready and is a well known standard of alerting in the Elasticsearch/OpenSearch ecosystem. Their mojo is : "If you can see it in your dashboards, ElastAlert 2 can alert on it." In this document you will learn how to deploy this component on Logs Data Platform thanks to its compability with OpenSearch through aliases and indexes. Logs Data Platform also allows you to host ElastAlert meta-indices on Logs Data Platform..
Note that in order to complete this tutorial, you should have at least:
In order to deploy ElastAlert, it is important that you have data on which you want to alert things on. If you only have Graylog stream, you can use aliases to enable the OpenSearch API on your stream data. Here is how:
Add an alias
button.Save
button.Attach content to the alias
option.If you only have indices, you can use them directly in the ElastAlert configuration.
ElastAlert configuration consists in three steps:
Installing ElastAlert can be done in different ways as described in their documentation.. You can either use the docker image or install the python 3 packages.
You must clone the Jertel ElastAlert repository for the most recent changes:
$ git clone https://github.com/jertel/elastalert2.git
Install the module:
$ pip install "setuptools>=11.3"
$ python setup.py install
If there is any error about any missing packages, install them manually. For example:
$ pip install setuptools_rust
The next step is to configure ElastAlert meta-indices using the provided tool elastalert-create-index. ElastAlert needs 5 indices to operate:
The following command will create the indices on Logs Data Platform directly from OpenSearch API.
$ elastalert-create-index --host <ldp-cluster>.logs.ovh.com --port 9200 --username <username> --password <password> --ssl --index <username>-i-<suffix>
Tou should pay attention to the following points:
<ldp-cluster>
must be the one assigned to you (find on the Home page of the LDP Manager).<username>
is the username used to connect to the API or to the Logs Data Platform interfaces (Graylog or OpenSearch Dashboards).<password>
is the associated password. You can use tokens in place of the username/password couple for your credentials.--index
is the most important here since you must follow the index naming convention of Logs Data Platform. Use the presented form <username>-i-
as a base name for your meta-indices. <suffix>
can be personalized to any alphanumeric characters.This command will prompt you with different questions:
Verify TLS certificates? t/f: t
Enter optional OpenSearch URL prefix (prepends a string to the URL of every request):
Name of existing index to copy? (Default None)
Reading Elastic 7 index mappings:
Reading index mapping 'es_mappings/7/silence.json'
Reading index mapping 'es_mappings/7/elastalert_status.json'
Reading index mapping 'es_mappings/7/elastalert.json'
Reading index mapping 'es_mappings/7/past_elastalert.json'
Reading index mapping 'es_mappings/7/elastalert_error.json'
New index logs-**-*****-i-***** created
Done!
This will then create 5 indices and place the mapping on them. All you need after is to create the ElastAlert configuration file and some rules.
Without further delay here is a sample config.yml file you can use for your configuration:
rules_folder: /opt/elastalert/rules
run_every:
minutes: 5
buffer_time:
hours: 6
es_host: <ldp-cluster>.logs.ovh.com
es_port: 9200
use_ssl: True
verify_certs: True
es_username: <username>
es_password: <password>
writeback_index: <username>-i-<suffix>
alert_time_limit:
days: 2
You can find all the available options here.
elastalert-create-index
.In this example, we will create a frequency.yml rule which will send a email if the field user with the value Oles appears more than 3 times in less than 4 hours and send an email. If your machine cannot send an email, you can still test the rule (it will just fail at the sending step).
name: Example frequency rule
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency
# (Required)
# Index to search, wildcard supported
index: <index-or-alias-to-check>
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 3
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
hours: 4
timestamp_field: timestamp
timestamp_type: custom
timestamp_format: '%Y-%m-%d %H:%M:%S.%f'
timestamp_format_expr: 'ts[:23]'
# (Required)
# A list of OpenSearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: https://opensearch.org/docs/latest/opensearch/query-dsl/index/
filter:
- term:
user: "Oles"
# (Required)
# The alert is used when a match is found
alert:
- "email"
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "elastalert@example.com"
We won't detail all the parameters since most of them are self-explanatory. However, please pay attention to the index parameter. This index or alias is the one containing the logs or documents you want to be alerted from.
It's also important to customize the timestamp parameters according to the timestamp of your logs or documents. Here we customize a custom timestamp on the timestamp_field timestamp
with the format used in the logs pipeline %Y-%m-%d %H:%M:%S.%f
. Because this format can have more than 3 extra numbers, we need to truncate them using the timestamp_format_expr option. Note that Elastalert does not support nanoseconds, so you will have to limit the precision of your timestamp to microseconds.
To launch ElastAlert, use the following command:
$ elastalert --config config.yml
To test your alert you can use the following curl command sending logs to our OpenSearch endpoint:
$ curl -H 'Content-Type: application/json' -u '<username>:<password>' -XPOST https://<ldp-cluster>.logs.ovh.com:9200/ldp-logs/message -d '{ "X-OVH-TOKEN" : "stream-token>" , "test_field" : "OVHcloud" , "user": "Oles", "short_message" : "Hello OpenSearch input", "host" : "OVHcloud_elastalert" }'
If you send this event more than 3 times, the elastalert process will try to send an alert to the configured email address.
ElastAlert has a lot of integrations for alerting including Email, JIRA, OpsGenie, SNS, HipChat, Slack, MS Teams, PagerDuty, Zabbix, custom commands and many more.
Please feel free to give any suggestions in order to improve this documentation.
Whether your feedback is about images, content, or structure, please share it, so that we can improve it together.
Your support requests will not be processed via this form. To do this, please use the "Create a ticket" form.
Thank you. Your feedback has been received.
Access your community space. Ask questions, search for information, post content, and interact with other OVHcloud Community members.
Discuss with the OVHcloud community