Alerts
  • 04 Apr 2022
  • 12 Minutes to read
  • Dark
    Light

Alerts

  • Dark
    Light

Alerts are created using Event Definitions that consist of Conditions. When a given condition is met it will be stored as an Event and can be used to trigger a notification. If your system has an enterprise license, then Events may be combined to create Correlations.

Graylog ships with default alert conditions and alert notifications , and both can be extended with Plugins.

Alerts & Events

As of Graylog 3.1.0, the Alerts page has changed to reflect a new method of generating Alerts. An Alert is triggered when a defined Event is detected.An Event is a condition that matches a log to a time period or aggregation. The Event may be used to group similar fields, change field content,or create new field content for use with Alerting and Correlation (an enterprise feature.)

If no Events have been defined, the Alerts & Events page will display the “Get Started!” button as shown below.

alertsstartingpagenoevents.png

Defining an Event

When you click on the “Get Started!” button you will be presented with a set of dialogues that allow you to setTitle, Description, and Priority. You may click back on the selection bar to step backward in the definitionprocess at any time.

alertseventdetails.png

Priority

The Priority of an Event is a classification for user purpose. The priority of an eventwill be displayed as a thermometer icon in the over view and will be written into the notification.

Filter

By combining a Filter and an Aggregation you can specifically describe the criteria of an Event.Define a Filter by using Search Query in the same syntax as the Search page. Select a Stream inwhich the message can found. Define the window of time that the Filter will search backward to match messages.The search will be executed at the given interval. If the Filter matches an Event can be created.However, it may be useful to augment the filtered data with an aggregation
alertsfilterdefinition.png
If the defined Filter matches messages currently within the Graylog Server, they will be displayedin the Filter Preview panel on the right.

Filter with dynamic lists (Enterprise feature)

Dynamic lists allow you to define a Filter where some of the search arguments are parameterized. Everytime an event defintion is being checked, these parameters are replaced with the result of a configured Look up table.For example, you maintain a list of former employees in Active Directory or an HR system and want an alert if anyone on the list tries to log in.You can define a filter like Login from username:$former_employee$, where the parameter $former_employee$is backed by a look up table, that returnsa current list of your former employees.

Aggregation

An Aggregation can run a mathematical operation on either a numeric field value or the raw count ofmessages generated that match the Filter. Also, Aggregation can now group matches by a selected fieldbefore making the comparison. For instance, if the field username is defined, then it is possible toalert on five successive failed logins by a particular username.This use case is shown below.

alertsaggregationexample.png

Fields

Creating Custom Fields allows the Event to populate data from the original log into the Graylog Events index. This prevents the operator from having to run subsequent searches to getvital information. This can also be used to limit the amount of data sent toa Notification target. The Event will be recorded to the “All Events” streamand will contain the Custom Field, as well as the result of the Aggregation that triggeredthe Event.
alertscustomFielddisplay.png

Notifications

After defining the Events that are needed to trigger an Alert it is possible to attach a Notifcation.By attaching a Notification to an Event or group of Events we can determine how and when informationwill flow out from Graylog. Notifications can be created by selecting the Notifications button underthe Alerts tab, or by defining them in the Event workflow.

In this section we explain what the default alert notifications included in Graylog do, and how to configure them. Alert notifications are meant to be extensible through Plugins, you can find more types in the Graylog Marketplace or even create your own.

Data available to notifications

Graylog makes the following data available when it runs a notification.

Different notification types will expose the data differently, the details are listed with the description of the specific notifications below.

Event Definition Metadata Information about the event definition that created the alert.

  • event_definition_id(String) - The database ID of the event definition
  • event_definition_type(String) - The internal name of the event definition type (aggregation-v1or correlation-v1)
  • event_definition_title(String) - The title set in the UI
  • event_definition_description(String) - The description set in the UI
  • job_definition_id(String) - The internal job definition ID associated with a scheduled event definition
  • job_trigger_id(String) - The internal ID associated with the current execution of the job

Event Data

  • eventThe event as it is stored in Graylog
    • id(String) - The message ID of the stored event.
    • event_definition_id(String) - Same as event_definition_idin the metadata section.
    • event_definition_type(String) - Same as event_definition_typein the metadata section.
    • origin_context(String) - URN of the message or event creating this event (either eventor message). Can be empty.
    • timestamp(DateTime) - The timestamp this event is describing, can be set to the underlying event or message (see origin_context).
    • timestamp_processing(DateTime) - The timestamp this event has been created by Graylog.
    • timerange_start(DateTime) - The start of the window of data Graylog used to create this event. Can be empty.
    • timerange_end(DateTime) - The end of the window of data Graylog used to create this event. Can be empty.
    • streams- (Strings) - The list of stream IDs the event is stored in.
    • source_streams(Strings) - The list of stream IDs the event pulled data from.
    • alert(bool) - Whether this event is considered to be an alert. Always truefor event definitions that have notifications.
    • message(String) - A human-friendly message describing this event.
    • source(String) - The host name of the Graylog server that created this event.
    • key_tuple(Strings) - The list of values making up the event’s key.
    • key(String) - The event’s key as a single string.
    • priority(long) - The event’s priority value.
    • fields(Map<String, String>) - The custom fields attached to the event.

Backlog

  • backlog(List of Message summaries) - The list of messages or events which lead to this alert being generated
    • id(String) - The message ID.
    • index(String) - The name of the index the message is stored in. Use together with idto uniquely identify a message in Graylog.
    • source(String) - The ```source ` field of the message.
    • message(String) - The messagefield of the message.
    • timestamp(DateTime) - The timestampfield of the message.
    • stream_ids(Strings) - The stream IDs of the message.
    • fields(Map<String, Object>) - The remaining fields of the message, can be iterated over

Email alert notification

The email alert notification can be used to send an email to the configured alert receivers when the conditions are triggered.

Make sure to check the email-related configuration settings in the Graylog configuration file.

Three configuration options are available for the alert notification to customize the email that will be sent. The email body and email subject are JMTE templates. JMTE is a minimal template engine that supports variables, loops and conditions. See the JMTE documentation for a language reference.

All of the data described above is available in the JMTE templates.
The default body template shows some advanced examples of accessing the information listed above:

--- [Event Definition] ---------------------------
Title:       ${event_definition_title}Description: ${event_definition_description}
Type:        ${event_definition_type}
--- [Event] --------------------------------------
Timestamp:            ${event.timestamp}
Message:              ${event.message}
Source:               ${event.source}
Key:                  ${event.key}
Priority:             ${event.priority}
Alert:                ${event.alert}
Timestamp Processing: ${event.timestamp}
Timerange Start:      ${event.timerange_start}
Timerange End:        ${event.timerange_end}
Fields:
${foreach event.fields field}${field.key}: ${field.value}
${end}
${if backlog}
--- [Backlog] ------------------------------------
Last messages accounting for this alert:
${foreach backlog message}
${message}
${end}
${end}

alertsemailnotification.png

HTTP alert notification

The HTTP alert notification lets you configure an endpoint that will be called when the alert is triggered.

Graylog will send a POST request to the notification URL including information about the alert. The body of the request is the JSON encoded data described above.

Here is an example of the payload included in a notification:

{
  "event_definition_id": "this-is-a-test-notification",
  "event_definition_type": "test-dummy-v1",
  "event_definition_title": "Event Definition Test Title",
  "event_definition_description": "Event Definition Test Description",
  "job_definition_id": "<unknown>",
  "job_trigger_id": "<unknown>",
  "event": {
    "id": "NotificationTestId",
    "event_definition_type": "notification-test-v1",
    "event_definition_id": "EventDefinitionTestId",
    "origin_context": "urn:graylog:message:es:testIndex_42:b5e53442-12bb-4374-90ed-0deadbeefbaz",
    "timestamp": "2020-05-20T11:35:11.117Z",
    "timestamp_processing": "2020-05-20T11:35:11.117Z",
    "timerange_start": null,
    "timerange_end": null,
    "streams": [
      "000000000000000000000002"
    ],
    "source_streams": [],
    "message": "Notification test message triggered from user <admin>",
    "source": "000000000000000000000001",
    "key_tuple": [
      "testkey"
    ],
    "key": "testkey",
    "priority": 2,
    "alert": true,
    "fields": {
      "field1": "value1",
      "field2": "value2"
    }
  },
  "backlog": []
}

alertshttpnotification.png

PagerDuty alert notification

The PagerDuty alert notification allows you create new incidents in PagerDuty in response to Events in your Graylog server.

Note

PagerDuty alert notifications are intended as a replacement for the PagerDutyintegration previously available from Graylog Labs. If you are using the Graylog Labs PagerDuty integration, you should migrate to the officiallysupported PagerDuty alert notifications at your earliest convenience.

Important

In order to use PagerDuty alert notifications, you will need an integration routing key. If you do not already have one, you will need to create a new integration in PagerDuty:PagerDuty Documentation

These are the supported configuration options:

Routing Key
Your PagerDuty integration routing key.

Use Custom Incident Key
If enabled, an incident key will be generated using the provided Incident Key Prefix. This will prevent PagerDuty from creating multipleincidents for a single Event. If not checked, no incident key willbe generated and each event notification will create a new incident in PagerDuty.

Incident Key Prefix
If Custom Incident Keyis enabled, this will be used as the prefix for theincident key.

Client Name
The name of the Graylog system that triggered the PagerDuty incident.

Client URL
The URL for your Graylog web interface. If provided, this will be used to constructlinks which will be embedded in your PagerDuty incident.

alertspagerdutynotification.png

Slack alert notification

The Slack alert notification allows you to send notifications to your slack workspace in response to events in your Graylog server.

Note

Slack alert notifications are intended as a replacement for the Slackintegration previously available from Graylog Labs. If youare using the Graylog Labs Slack plugin, you should migrate to the officiallysupported Slack alert notifications at your earliest convenience.

alertsslackpluginnotification.png
These are the supported configuration options:

Webhook URL
The unique URL used to send messages to your Slack instance.

Channel
A channel to send message to.

Configuration Color
Highlight the custom message with this color.

Custom Message
The message that will be sent to Slack. The data described above can be used in this template.

Message Backlog Limit (optional)
Limit the number of backlog messages sent as part of the Slack notification. If set to 0, no limit will be enforced.

User Name (optional)
User name of the sender in Slack.

Icon URL
Image to use as the icon for this message.

Icon Emoji
Emoji to use as the icon for this message (overrides Icon URL).

Script alert notification [Enterprise]


The Script Alert Notification lets you configure a script that will be executed when the alert is triggered.

Important

Script Alert Notification is an Enterprise Integrations plugin feature and thus requires an Enterprise license.

alertsscriptnotification.png
These are the supported configuration options.

Script Path
The path to where the script is located. Must me within the permitted script path (which is customizable).

Script Timeout
The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.

Script Arguments
Space-delimited string of parameters. Any of the data described above can be used.

Send Alert Data Through STDIN
Sends the JSON encoded data described above through standard in. You can use a JSON parser in your script.

Script Alert Notification success is determined by its exit value; success equals zero.Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Here is a sample Python script that shows all of the supported Script Alert Notificationfunctionality (argument parsing, STDIN JSON parsing, STDOUT, exit values, and returning an exit value).:

#!/usr/bin/env python3
import json
import sys

# Function that prints text to standard error
def print_stderr(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

# Main function
if __name__ == "__main__":

    # Print out all input arguments.
    sys.stdout.write("All Arguments Passed In: " + ' '.join(sys.argv[1:]) + "\n")

    # Turn stdin.readlines() array into a string
    std_in_string = ''.join(sys.stdin.readlines())

    # Load JSON
    event_data = json.loads(std_in_string)

    # Extract some values from the JSON.
    sys.stdout.write("Values from JSON: \n")
    sys.stdout.write("Event Definition ID: " + event_data["event_definition_id"] + "\n")
    sys.stdout.write("Event Definition Title: " + event_data["event_definition_title"] + "\n")
    sys.stdout.write("Event Timestamp: " + event_data["event"]["timestamp"] + "\n")

    # Extract Message Backlog field from JSON.
    sys.stdout.write("\nBacklog:\n")
    for message in event_data["backlog"]:
        for field in message.keys():
            sys.stdout.write("Field: " + field + "\t")
            sys.stdout.write("Value: " + str(message[field]) + "\n")

    # Write to stderr if desired
    # print_stderr("Test return through standard error")

    # Return an exit value. Zero is success, non-zero indicates failure.
    exit(0) 

Legacy Script alert callback

The Legacy Script Alert Callback lets you configure a script that will be executed when the alert is triggered.

Warning

Legacy alarm callbacks are being deprecated. If you are currently using the Legacy Script Alert Callback, you should migrate to Script Alert Notification as soon as possible.

Important

Legacy Script Alert Callback is an Enterprise Integrations plugin feature and thus requires an Enterprise license.

alertslegacyscriptcallback.png
These are the supported configuration options.

Script Path
The path to where the script is located. Must me within the permitted script path (which is customizable).

Script Timeout
The maximum time (in milliseconds) the script will be allowed to execute before being forcefully terminated.

Script Arguments
String of parameters in which the delimiters are either a space-delimited or a new-line. The following argument variables may be used:

Stream
The stream this alert belongs to.

  • stream_idID of the stream
  • stream_nametitle of the stream
  • stream_descriptionstream description
  • stream_urla string that contains the URL to the view the relevant messages for the alert. Make sure to set the HTTP URL configuration parameter, as there is no default.

Alert

The check result object for this stream.

  • alert_descriptiontext that describes the check result
  • alert_triggered_atdate when this condition was triggered

Condition

The available conditions to request are

  • condition_idID of the condition
  • condition_descriptiondescription of the condition
  • condition_titletitle of the condition
  • condition_typetype of condition
  • condition_gracegrace period for the condition
  • condition_repeat_notificationrepeat notification of the script

Send Alert Data Through STDIN

Sends JSON alert data through standard in. You can use a JSON parser in your script.

Legacy Script Alert Callback success is determined by its exit value; success equals zero. Any non-zero exit value will cause it to fail. Returning any error text through STDERR will also cause the alarm callback to fail.

Event Summary

When all of the components have been defined the Event Summary will be displayed to the user. At this time, the user may select a previous point in the Workflow to change a parameter. The user may also cancel out of the workflow, select done. The Event may be viewed under Alerts > Event Definitions.


Was this article helpful?