Alerts

Alerts are created using Event Definitions that consist of Conditions. When a given condition is met it will be stored as an Event and can be used to trigger a notification. If your system has an enterprise license, then Events may be combined to create Correlations.

Graylog ships with default alert conditions and alert notifications, and both can be extended with Plugins.

Alerts & Events

As of Graylog 3.1.0, the Alerts page has changed to reflect a new method of generating Alerts. An Alert is triggered when a defined Event is detected. An Event is a condition that matches a log to a time period or aggregation. The Event may be used to group similar fields, change field content, or create new field content for use with Alerting and Correlation (an enterprise feature.)

If no Events have been defined, the Alerts & Events page will display the “Get Started!” button as shown below.

../../_images/alerts_starting_page_no_events.png

Defining an Event

Event Details

When you click on the “Get Started!” button you will be presented with a set of dialogues that allow you to set Title, Description, and Priority. You may click back on the selection bar to step backward in the definition process at any time.

../../_images/alerts_event_details.png

Filter & Aggregation

Filter

By combining a Filter and an Aggregation you can specifically describe the criteria of an Event. Define a Filter by using Search Query in the same syntax as the Search page. Select a Stream in which the message can found. Define the window of time that the Filter will search backward to match messages. The search will be executed at the given interval. If the Filter matches an Event can be created. However, it may be useful to augment the filtered data with an aggregation

../../_images/alerts_filter_definition.png

If the defined Filter matches messages currently within the Graylog Server, they will be displayed in the Filter Preview panel on the right.

Aggregation

An Aggregation can run a mathematical operation on either a numeric field value or the raw count of messages generated that match the Filter. Also, Aggregation can now group matches by a selected field before making the comparison. For instance, if the field username is defined, then it is possible to alert on five successive failed logins by a particular username. This use case is shown below.

../../_images/alerts_aggregation_example.png

Fields

Creating Custom Fields allows the Event to populate data from the original log into the Graylog Events index. This prevents the operator from having to run subsequent searches to get vital information. This can also be used to limit the amount of data sent to a Notification target. The Event will be recorded to the “All Events” stream and will contain the Custom Field, as well as the result of the Aggregation that triggered the Event.

../../_images/alerts_customField_display.png

Notifications

After defining the Events that are needed to trigger an Alert it is possible to attach a Notifcation. By attaching a Notification to an Event or group of Events we can determine how and when information will flow out from Graylog. Notifications can be created by selecting the Notifications button under the Alerts tab, or by defining them in the Event workflow.

Alert notifications types explained

In this section we explain what the default alert notifications included in Graylog do, and how to configure them. Alert notifications are meant to be extensible through Plugins, you can find more types in the Graylog Marketplace or even create your own.

Important

In previous versions of Graylog (before 2.2.0), the email alarm notification was used, when alert conditions existed for a stream, but no alarm notification had been created before. This has been changed, so that if there is no alarm notification existing for a stream, alerts will be shown in the interface but no other action is performed. To help users coming from earlier version, there is a migration job which is being run once, creating the email alarm notification explicitly for qualifying streams, so the old behavior is preserved.

Email alert notification

The email alert notification can be used to send an email to the configured alert receivers when the conditions are triggered.

Make sure to check the email-related configuration settings in the Graylog configuration file.

Three configuration options are available for the alert notification to customize the email that will be sent. The email body and email subject are JMTE templates. JMTE is a minimal template engine that supports variables, loops and conditions. See the JMTE documentation for a language reference.

We expose the following objects to the templates.

stream

The stream this alert belongs to.

  • stream.id ID of the stream

  • stream.title title of the stream

  • stream.description stream description

stream_url

A string that contains the HTTP URL to the stream.

check_result

The check result object for this stream.

  • check_result.triggeredCondition string representation of the triggered alert condition

  • check_result.triggeredAt date when this condition was triggered

  • check_result.resultDescription text that describes the check result

backlog

A list of message objects. Can be used to iterate over the messages via foreach.

message (only available via iteration over the backlog object)

The message object has several fields with details about the message. When using the message object without accessing any fields, the toString() method of the underlying Java object is used to display it.

  • message.id autogenerated message id

  • message.message the actual message text

  • message.source the source of the message

  • message.timestamp the message timestamp

  • message.fields map of key value pairs for all the fields defined in the message

The message.fields fields can be useful to get access to arbitrary fields that are defined in the message. For example message.fields.full_message would return the full_message of a GELF message.

../../_images/alerts_email_notification.png

HTTP alert notification

The HTTP alert notification lets you configure an endpoint that will be called when the alert is triggered.

Graylog will send a POST request to the notification URL including information about the alert. Here is an example of the payload included in a notification:

{
    "check_result": {
        "result_description": "Stream had 2 messages in the last 1 minutes with trigger condition more than 1 messages. (Current grace time: 1 minutes)",
        "triggered_condition": {
            "id": "5e7a9c8d-9bb1-47b6-b8db-4a3a83a25e0c",
            "type": "MESSAGE_COUNT",
            "created_at": "2015-09-10T09:44:10.552Z",
            "creator_user_id": "admin",
            "grace": 1,
            "parameters": {
                "grace": 1,
                "threshold": 1,
                "threshold_type": "more",
                "backlog": 5,
                "time": 1
            },
            "description": "time: 1, threshold_type: more, threshold: 1, grace: 1",
            "type_string": "MESSAGE_COUNT",
            "backlog": 5
        },
        "triggered_at": "2015-09-10T09:45:54.749Z",
        "triggered": true,
        "matching_messages": [
            {
                "index": "graylog2_7",
                "message": "WARN: System is failing",
                "fields": {
                    "gl2_remote_ip": "127.0.0.1",
                    "gl2_remote_port": 56498,
                    "gl2_source_node": "41283fec-36b4-4352-a859-7b3d79846b3c",
                    "gl2_source_input": "55f15092bee8e2841898eb53"
                },
                "id": "b7b08150-57a0-11e5-b2a2-d6b4cd83d1d5",
                "stream_ids": [
                    "55f1509dbee8e2841898eb64"
                ],
                "source": "127.0.0.1",
                "timestamp": "2015-09-10T09:45:49.284Z"
            },
            {
                "index": "graylog2_7",
                "message": "ERROR: This is an example error message",
                "fields": {
                    "gl2_remote_ip": "127.0.0.1",
                    "gl2_remote_port": 56481,
                    "gl2_source_node": "41283fec-36b4-4352-a859-7b3d79846b3c",
                    "gl2_source_input": "55f15092bee8e2841898eb53"
                },
                "id": "afd71342-57a0-11e5-b2a2-d6b4cd83d1d5",
                "stream_ids": [
                    "55f1509dbee8e2841898eb64"
                ],
                "source": "127.0.0.1",
                "timestamp": "2015-09-10T09:45:36.116Z"
            }
        ]
    },
    "stream": {
        "creator_user_id": "admin",
        "outputs": [],
        "matching_type": "AND",
        "description": "test stream",
        "created_at": "2015-09-10T09:42:53.833Z",
        "disabled": false,
        "rules": [
            {
                "field": "gl2_source_input",
                "stream_id": "55f1509dbee8e2841898eb64",
                "id": "55f150b5bee8e2841898eb7f",
                "type": 1,
                "inverted": false,
                "value": "55f15092bee8e2841898eb53"
            }
        ],
        "alert_conditions": [
            {
                "creator_user_id": "admin",
                "created_at": "2015-09-10T09:44:10.552Z",
                "id": "5e7a9c8d-9bb1-47b6-b8db-4a3a83a25e0c",
                "type": "message_count",
                "parameters": {
                    "grace": 1,
                    "threshold": 1,
                    "threshold_type": "more",
                    "backlog": 5,
                    "time": 1
                }
            }
        ],
        "id": "55f1509dbee8e2841898eb64",
        "title": "test",
        "content_pack": null
    }
}

Legacy Script alert notification

The Script Alert Notification lets you configure a script that will be executed when the alert is triggered.

Important

Script Alert Notification is an Enterprise Integrations plugin feature and thus requires an Enterprise license.

../../_images/alerts_script_notification.png

These are the supported configuration options.

Script Path

The path to where the script is located. Must me within the permitted script path (which is customizable).

Script Timeout

The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.

Script Arguments

String of parameters in which the delimiters are either a space-delimited or a new-line. The following argument variables may be used:

Stream

The stream this alert belongs to.

  • stream_id ID of the stream

  • stream_name title of the stream

  • stream_description stream description

  • stream_url a string that contains the URL to the view the relevant messages for the alert. Make sure to set the HTTP URL configuration parameter, as there is no default.

Alert

The check result object for this stream.

  • alert_description text that describes the check result

  • alert_triggered_at date when this condition was triggered

Condition

The available conditions to request are

  • condition_id ID of the condition

  • condition_description description of the condition

  • condition_title title of the condition

  • condition_type type of condition

  • condition_grace grace period for the condition

  • condition_repeat_notification repeat notification of the script

Send Alert Data Through STDIN

Sends JSON alert data through standard in. You can use a JSON parser in your script.

{
  "stream_id": "000000000000000000000001",
  "stream_name": "All messages",
  "stream_description": "Stream containing all messages",
  "stream_url": "http://localhost:8080///streams/000000000000000000000001/messages?rangetype=absolute&from=2019-01-25T20:57:50.793Z&to=2019-01-25T21:02:50.793Z&q=*",
  "alert_description": "Stream received messages matching <has_field:\"true\"> (Current grace time: 0 minutes)",
  "alert_triggered_at": "2019-01-25T21:02:50.793Z",
  "condition_id": "ea9fcdff-2037-44f9-801e-099bf4bb3dbd",
  "condition_description": "field: has_field, value: true, grace: 0, repeat notifications: false",
  "condition_title": "has_field",
  "condition_type": "field_content_value",
  "condition_grace": 0,
  "condition_parameters": {
    "backlog": 10,
    "repeat_notifications": false,
    "field": "has_field",
    "query": "*",
    "grace": 0,
    "value": "true"
  },
  "condition_repeat_notifications": false,
  "message_backlog": [
    {
      "has_field": "true",
      "gl2_remote_ip": "127.0.0.1",
      "gl2_remote_port": 56246,
      "streams": [
        "000000000000000000000001"
      ],
      "gl2_source_node": "e065896b-8a9a-4f45-83f2-e740525ed035",
      "_id": "92839500-20e4-11e9-8175-0637e3f7ecfc",
      "source": "example.org",
      "message": "Hello there",
      "gl2_source_input": "5c2e99687a90e30a3512f766",
      "facility": "test",
      "timestamp": "2019-01-25T21:02:49.423Z"
    },
    {
      "has_field": "true",
      "gl2_remote_ip": "127.0.0.1",
      "gl2_remote_port": 56245,
      "streams": [
        "000000000000000000000001"
      ],
      "gl2_source_node": "e065896b-8a9a-4f45-83f2-e740525ed035",
      "_id": "928087c0-20e4-11e9-8175-0637e3f7ecfc",
      "source": "example.org",
      "message": "Hello there",
      "gl2_source_input": "5c2e99687a90e30a3512f766",
      "facility": "test",
      "timestamp": "2019-01-25T21:02:49.403Z"
    }
  ],
  "message_backlog_size": 5
}

Script Alert Notification success is determined by its exit value; success equals zero. Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Here is a sample Python script that shows all of the supported Script Alert Notification functionality (argument parsing, STDIN JSON parsing, STDOUT, exit values, and returning an exit value).:

#!/usr/bin/env python3
import json
import sys
import time


# Function that prints text to standard error
def print_stderr(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# Main function
if __name__ == "__main__":

    # Print out all input arguments.
    sys.stdout.write("All Arguments Passed In: " + ' '.join(sys.argv[1:]) + "\n")
    sys.stdout.write("Stream Name: " + sys.argv[2] + "\n")
    sys.stdout.write("Stream Description: " + sys.argv[3] + "\n")
    sys.stdout.write("Alert Triggered At: " + sys.argv[6] + "\n")

    # Turn stdin.readlines() array into a string
    std_in_string = ''.join(sys.stdin.readlines())

    # Load JSON
    alert_object = json.loads(std_in_string)

    # Extract some values from the JSON.
    sys.stdout.write("Values from JSON: \n")
    sys.stdout.write("Stream ID: " + alert_object["stream_id"] + "\n")
    sys.stdout.write("Stream Name: " + alert_object["stream_name"] + "\n")
    sys.stdout.write("Alert Triggered At: " + alert_object["alert_triggered_at"] + "\n")

    # Extract Message Backlog field from JSON.
    sys.stdout.write("\n\nFields:\n")
    for message in alert_object["message_backlog"]:
        for field in message.keys():
            print("Field: " + field)
            print("Value: " + str(message[field]))

    # Write to stderr if desired
    # print_stderr("Test return through standard error")

    # Return an exit value. Zero is success, non-zero indicates failure.
    exit(0)

Event Summary

When all of the components have been defined the Event Summary will be displayed to the user. At this time, the user may select a previous point in the Workflow to change a parameter. The user may also cancel out of the workflow, select done. The Event may be viewed under Alerts>Event Definitions.

All events stream

The All events stream can be used to view all previous Events that have been triggered. The Event is recorded with the fields defined in the Custom Fields portion of the Event.

../../_images/alerts_all_events_stream.png